Exchange 2010 virtualization storage gotchas

There’s a lot of momentum for Exchange virtualization. At Trace3, we do a lot of work with VMware, so the majority of the customers I work with already have VMware deployed strategically into their production operation model. As a result, we see a lot of Exchange 2010 under VMware. With Exchange 2010 SP1 and lots of customer feedback, the Exchange product team has really stepped up to provide better support for virtual environments as well as more detailed guidance on planning for and deploying Exchange 2007 and 2010 in virtualization.

Last week, I was talking with a co-worker about Exchange’s design requirements in a virtual environment. I casually mentioned the “no file-level storage protocols” restriction for the underlying storage and suddenly, the conversation turned a bit more serious. Many people who deploy VMware create large data stores on their SAN and share them to the ESX cluster via the NFS protocol. There are a lot of advantages to doing it this way, and it’s a very flexible and relatively easy way to deploy VMs. However, it’s not supported for Exchange VMs.

The Heck You Say?

“But Devin,” I can hear some of you say, “what do you mean it’s not supported to run Exchange VMs on NFS-mounted data stores? I deploy all of my virtual machines using VMDKs on NFS-mounted data stores. I have my Exchange servers there. It all works.”

It probably does work. Whether or not it works, though, it’s not a supported configuration, and one thing Masters are trained to hate with a passion is letting people deploy Exchange in a way that gives them no safety net. It is an essential tool in your toolkit to have the benefit of Microsoft product support to walk you through the times when you get into a strange or deep problem.

Let’s take a look at Microsoft’s actual support statements. For Exchange 2010, Microsoft has the following to say in http://technet.microsoft.com/en-us/library/aa996719.aspx under virtualization (emphasis added):

The storage used by the Exchange guest machine for storage of Exchange data (for example, mailbox databases or Hub transport queues) can be virtual storage of a fixed size (for example, fixed virtual hard disks (VHDs) in a Hyper-V environment), SCSI pass-through storage, or Internet SCSI (iSCSI) storage. Pass-through storage is storage that’s configured at the host level and dedicated to one guest machine. All storage used by an Exchange guest machine for storage of Exchange data must be block-level storage because Exchange 2010 doesn’t support the use of network attached storage (NAS) volumes. Also, NAS storage that’s presented to the guest as block-level storage via the hypervisor isn’t supported.

Exchange 2007 has pretty much the same restrictions as shown in the http://technet.microsoft.com/en-us/library/bb738146(EXCHG.80).aspx TechNet topic. What about Exchange 2003? Well, that’s trickier; Exchange 2003 was never officially supported under any virtualization environment other than Microsoft Virtual Server 2005 R2.

The gist of the message is this: it is not supported by Microsoft for Exchange virtual machines to use disk volumes that are on file-level storage such as NFS or CIFS/SMB, if those disk volumes hold Exchange data. I realize this is a huge statement, so let me unpack this a bit. I’m going to assume a VMware environment here, but these statements are equally true for Hyper-V or any other hypervisor supported under the Microsoft SVVP.

While the rest of the discussion will focus on VMware and NFS, all of the points made are equally valid for SMB/CIFS and other virtualization system. (From a performance standpoint, I would not personally want to use SMB for backing virtual data stores; NFS, in my experience, is much better optimized for the kind of large-scale operations that virtualization clusters require. I know Microsoft is making great strides in improving the performance of SMB, but I don’t know if it’s there yet.

It’s Just Microsoft, Right?

So is there any way to design around this? Could I, in theory, deploy Exchange this way and still get support from my virtualization vendor? A lot of people I talk to point to a whitepaper that VMware published in 2009 that showed the relative performance of Exchange 2007 over iSCSI, FC, and NFS. They use this paper as “proof” that Exchange over NFS is supported.

Not so much, at least not with VMware. The original restriction may come from the Exchange product group (other Microsoft workloads are supported in this configuration), but the other vendors certainly know the limitation and honor it in their guidance. Look at VMware’s Exchange 2010 best practices at http://www.vmware.com/files/pdf/Exchange_2010_on_VMware_-_Best_Practices_Guide.pdf on page 13:

It is important to note that there are several different shared-storage options available to ESX (iSCSI, Fibre Channel, NAS, etc.); however, Microsoft does not currently support NFS for the Mailbox Server role (clustered or standalone). For Mailbox servers that belong to a Database Availability Group, only Fibre Channel is currently supported; iSCSI can be used for standalone mailbox servers. To see the most recent list of compatibilities please consult the latest VMware Compatibility Guides.

According to this document, VMware is even slightly more restrictive! If you’re going to use RDMs (this section is talking about RDMs, so don’t take the iSCSI/FC statement as a limit on guest-level volume mounts), VMware is saying that you can’t use iSCSI RDMs, only FC RDMs.

Now, I believe – and there is good evidence to support me – that this guidance as written is actually slightly wrong:

  • The HT queue database is also an ESE database and is subject to the same limitations; this is pretty clear on a thorough read-through of the Exchange 2010 requirements in TechNet. Many people leave the HT queue database on the same volume they install Exchange to, which means that volume also cannot be presented via NFS. If you follow best practices, you move this queue database to a separate volume (which should be an RDM or guest-mounted iSCSI/FC LUN).
  • NetApp, one of the big storage vendors that supports the NFS-mounted VMware data store configuration, only supports Exchange databases mounted via FC/iSCSI LUNs using SnapManager for Exchange (SME) as shown in NetApp TR-3845. Additionally, in the join NetApp-VMware-Cisco performance whitepaper on virtualizing Microsoft workloads, the only configuration tested for Exchange 2010 is FC LUNs (TR-3785).
  • It is my understanding that the product group’s definition of Exchange files doesn’t just extend to ESE files and transaction logs, but to all of the Exchange binaries and associated files. I have not yet been able to find a published source to document this interpretation, but I am working on it.
  • I am not aware of any Microsoft-related restriction about iSCSI + DAG. This VMware Exchange 2010 best practices document (published in 2010) is the only source I’ve seen mention this restriction, and in fact, the latest VMware Microsoft clustering support matrix (published in June 2011) lists no such restriction. Microsoft’s guidelines seem to imply that block storage is block storage is block storage when it comes to “SCSI pass-through storage”). I have queries in to nail this one down because I’ve been asking in various communities for well over a year with no clear resolution other than, “That’s the way VMware is doing it.”

Okay, So Now What?

When I’m designing layouts for customers who are used to deploying Windows VMs via NFS-mounted VMDKs, I have a couple of options. My preferred option, if they’re also using RDMs, is to just have them provision one more RDM for the system drive and avoid NFS entirely for Exchange servers. That way, if my customer does have to call Microsoft support, we don’t have to worry about the issue at all.

However, that’s not always possible. My customer may have strict VM provisioning processes in place, have limited non-NFS storage to provision, or have some other reason why they need to use NFS-based VMDKs. In this case, I have found the following base layout to work well:

Volume Type Notes
C: VMDK or RDM Can be on any type of supported data store. Should be sized to include static page file of size PhysicalRAM + 10 MB.
E: RDM or guest iSCSI/FC iSCSI/FC    All Exchange binaries installed here. Move IIS files here (scripts out on Internet to do this for you). Create an E:\Exchdata directory and use NTFS mount points to mount each of the data volumes the guest will mount.
Data volumes RDM or guest iSCSI/FC Any volume holding mailbox/PF database EDB or logs, or HT queue EDB or logs. Should mount these separately, NTFS mount points recommended. Format these NTFS volumes with 64K block size, not default.

Note that we have several implicit best practices in use here:

  • Static page file, properly sized for a 64-bit operating system with a large amount of physical RAM. Doing this ensures that you have enough virtual memory for the Exchange memory profile AND that you can write a kernel memory crash dump to disk in the event of a blue screen. (If the page file is not sized properly, or is not on C:, the full dump cannot be written to disk.)
  • Exchange binaries not installed on the system drive. This makes restores much easier. Since Exchange uses IIS heavily, I recommend moving the IIS data files (the inetpub and children folders) off of the system drive and onto the Exchange volume. This helps reduce the rate of change on the system drive and offers other benefits such as making it easier to properly configure anti-virus exclusions.
  • The use of NTFS mount points (which mount the volume to a directory) instead of separate drive letters. For large DAGs, you can easily have a large number of volumes per MB role, making the use of drive letters a limitation on scalability. NTFS mount points work just like Unix mount points and work terribly well – they’ve been supported since Exchange 2003 and recommended since the late Exchange 2003 era for larger clusters. In Exchange 2007 and 2010 continuous replication environments (CCR, SCR, DAG), all copies must have the same pathnames.
  • Using NTFS 64K block allocations for any volumes that hold ESE databases. While not technically necessary for log partitions, doing so does not hurt performance.

So Why Is This Even A Problem?

This is the money question, isn’t it? Windows itself is supported under this configuration. Even SQL Server is. Why not Exchange?

At heart, it comes down to this: the Exchange ESE database engine is a very finely-tuned piece of software, honed for over 15 years. During that time, with only one exception (the Windows Storage Server 2003 Feature Pack 1, which allowed storage solutions running WSS 2003 + FP1 to host Exchange database files over NAS protocols), Exchange has never supported putting Exchange database files over file-level storage. I’m not enough of an expert on ESE to whip up a true detailed answer, but here is what I understand about it.

Unlike SQL Server, ESE is not a general purpose database engine. SQL is optimized to run relational databases of all types. The Exchange flavor of ESE is optimized for just one type of data: Exchange. As a result, ESE has far more intimate knowledge about the data than any SQL Server instance can. ESE provides a lot of performance boosts for I/O hungry Exchange databases and it can do so precisely because it can make certain assumptions. One of those assumptions is that it’s talking to block-level storage.

When a host process commits writes to storage, there’s a very real difference in the semantics of the write operation between block-level protocols and file-level protocols. Exchange, in particular, depends dramatically on precise control over block-level writes – which file protocols like NFS and SMB can mask. The cases under which this can cause data corruption for Exchange are admittedly corner cases, but they do exist and they can cause impressive damage.

Cleaning Up

What should we do about it if we have an Exchange deployment that is in violation of these support guidelines?

Ideally, we fix it. Microsoft’s support stance is very clear on this point, and in the unlikely event that data loss occurs in this configuration, Microsoft support is going to point at the virtualization/storage vendors and say, “Get them to fix it.” I am not personally aware of any cases of a configuration like this causing data loss or corruption, but I am not the Exchange Product Group – they get access to an amazing amount of data.

At the very least, you need to understand and document that you are in an unsupported configuration so that you can make appropriate plans to get into support as you roll out new servers or upgrade to future versions of Exchange. This is where getting a good Exchange consultant to do an Exchange health check can help you get what you need and provide the support you need with your management – we will document this in black and white and help provide the outside validation you might need to get things put right.

One request for the commenters: if all you’re going to do is say, “Well we run this way and have no problems,” don’t bother. I know and stipulate that there are many environments out there running in violating of this support boundary that have not (yet) run into issues. I’ve never said it won’t work. There are a lot of things we can do, but that doesn’t mean we should do them. At the same time, at the end of the day – if you know the issues and potential risks, you have to make the design decision that’s right for your organization. Just make sure it’s an informed (and documented, and signed-off!) decision.

10 Comments

  1. Scott McDonnell

    Devin,

    this is a great article! I currently work with Netapp storage and like the customers you describe, I have provisioned only NFS datastores to the ESX cluster. So far so good, but now our Organization wants to deploy Exchange 2010.

    The FAS2020 bundle we have came licensed with Snapmanager for Exchange. I have used it in the past on a physical Ex2003 environment, so I knew that we would have to present the storage to the Exch VM via iscsi targets. The disk based snapshots are great but I have only 1 Filer (No SnapMirror or DR site) so I need to find some method of backing up the entire Exchange Org , as well as any of the other VMs to external disk.

    This lead me to research Veeam’s Backup and Recovery product. Have you had much hands on with it?

    I have demoed the product and even spoken with the engineers. It all sounds great looks like it can backup everything we need to any target. So I asked the engineer about backing up Exchange and possibly combining this product and Netapp SME and they said no.
    In order for Veeam B&R to work, it can not have any in guest iscsi. All disks must be VMDKS. At first I thought this was no problem, it will make everything even simpler, all NFS, on pane of glass for all backups and restores of VMs and application data..perfect.. or maybe not.

    Then I saw the Microsoft documents and eventual your blog, NFS not “supported”. So now I am back to square one.

    I am wondering( more like hoping ), if this will be lifted in a future release or Service Pack? It seems there is a greater push or perhaps acceptance for NFS as a viable solution for Virtual storage. Even Vmware’s new VSA is NFS only, if I’m not mistaken?

    I’m still unsure as to the direction I want to take, but if we do say choose the Veeam solution, then I think I might take your advice and get something written down and signed off!

    Thanks for the great article, I am now more conflicted then ever ;)

  2. Jon

    I have a Exchange 2010 SP2 environment on VMWare / NetApp. trace3 is our VAR. Anyways, I have 4 CAS/HT servers with one vmdk each. I can’t fathom creating an iSCSI LUN for my CAS inetpub dirs, and my HT’s queues. That being said, I am in no hurry to prove microsoft right w/ our MB servers, and the DB/LOG volumes are guest initiated iSCSI based LUNs.

    What I need to do right now is look at ways to migrate my Exchange from one aggregate to another. I know I should use snapmirror, but I can’t help but think about setting up a new MB server, with large NFS based vmdks, seed it, and have it host the active copies, while I offline and create new volumes for the existing servers on a new aggregate and seed them


  3. Jon: call your rep at Trace3 and ask them to set up a call with me or one of the other Exchange architects. We’ll be happy to work with one of our storage gurus to make sure you’re moving those mailbox replicas the right way.


  4. Scott, sorry for not replying to this sooner! I would be very unwilling to violate a known support boundary that potentially affects all of my Exchange data. Think, for a second, about what happens to clients when an NFS mount hangs underneath them. Now think about the effect that could have on your databases. All of them. At the same time.

    If I remember correct, the Veeam solution is making snapshots of the virtual machine (hence the restriction about VMDKs). This is *also* not supported by Microsoft for Exchange, regardless of hypervisor flavor. See this TechNet article for more information, specifically:

    Some hypervisors include features for taking snapshots of virtual machines. Virtual machine snapshots capture the state of a virtual machine while it is running. This feature enables you to take multiple snapshots of a virtual machine and then revert the virtual machine to any of the previous states by applying a snapshot to the virtual machine. However, virtual machine snapshots are not application-aware, and using them can have unintended and unexpected consequences for a server application that maintains state data, such as Exchange Server. As a result, making virtual machine snapshots of an Exchange guest virtual machine is not supported.

    SME is a fine product, and once you have the snapshots on your filer you can use NDMP to back them up to another target.

  5. Hussain

    Hello,
    I’m in the process of storage migration from EMC AX4 to IBM DS3512.
    Currently in my Exchange 2007 Enterprise holds 1800 users I have configured all the Disks as Virtual RDMs with separate Storage Groups and Databases, each from vRDMs under RAID 5.

    Now, when the storage migration comes into play, I’m using Storage vMotion to Move Exchange from EMC AX4 to DS3500, When I do the same for the exchange, it automatically will convert the vRDM to VMDK disks.

    Here is comes to make the decision to stick to vRDM or to go ahead and move to VMDK. with vRDM a bit of issue with veeam backup it works, but very slow and time to time gives VSS Error.

    My only options that also it depends if I want to stick to the RDM for my Exchange 2007 or to get rid of vRDM and go ahead with VMDK, If I’m planning to stick to the vRDM, then I have to create all the disks similar to the EMC and present them to the host where the Exchange VM is hosted, then I have to create a new Storage Groups along with their Databases and start moving mailboxes from Databases to another Databases on the new IBM Storage.

    If I want to get rid of the vRDM and go back to VMDK I just need one BIG LUN Raid-5 to old the entire Exchange VM along with it’s disk. But what is the way to go? DS3512 will use 4 into 1 TB Nearline SAS Disk 7200 RPM, will user RAID1+0 for the Databases and Logs, but again what vRDM or VMDKs?

    If VMDK, how I would design the datastores that will hold the Logs and DBs?

    What’s your thoughts?
    Thanks,
    Hussain


  6. My thoughts are very simple: is it block-level storage all the way down? How are you storing those VMDKs? If those VMDKs are on a volume that is being presented by NFS, it’s not supported. See http://technet.microsoft.com/en-us/library/cc794548(v=EXCHG.80).aspx for the Exchange 2007 virtualization guidelines. The key part is this:

    • The storage that is used by the Exchange Server guest machine for the storage of Exchange data (for example, mailbox databases or hub transport queues) can be virtual storage of a fixed size (for example, fixed virtual hard drives [VHD] in a Hyper-V environment), SCSI pass-through storage, or Internet SCSI (iSCSI) storage. Pass-through storage is storage that is configured at the host level and that is dedicated to one guest machine. All storage that is used by an Exchange guest machine for the storage of Exchange data must be block-level storage. Exchange 2010 does not support using network attached storage (NAS) volumes. NAS storage that is presented to the guest as block-level storage by using the hypervisor is not supported. Pass-through volumes must be presented as block-level storage to the hardware virtualization software. This is because Exchange 2010 does not support using network attached storage (NAS) volumes. The following virtual disk requirements apply to volumes that are used to store Exchange data.
    • Virtual disks that dynamically expand are not supported by Exchange.
    • Virtual disks that use differencing or delta mechanisms (such as Hyper-V’s differencing VHDs or snapshots) are not supported.

    So you can use VMDKs as long as you’re not storing them on NFS, which means you need to use VMFS on a FC or iSCSI LUN. A lot of people ignore this and don’t have problems for a while, but when they do they are not supported. There’s a lot of “advice” out there that says you can badger Microsoft to support you with Exchange over NFS — this is bad advice. Ask those people to pony up references of customers that have actually gotten *Microsoft support* for this configuration.

    You mention that you can convert the entire VM to a big RAID-5 LUN. I would be very, very cautious about doing this. First, look at your VMFS and VMware volume size limitations. Second, RAID-5 has horrible random write performance, and writes are critical to Exchange. Whatever configuration you use, make sure you validate it with the Exchange sizer and Jetstress!

  7. Hussain

    Hello Devin,

    I highly appreciate your response.. I’m in iSCSI LUNs not NFS. IBM DS3512 is a bad product when it comes to a heavy intensive application such as Exchange/SQL or Oracle DB. We made a mistake when we purchase it with 1 TB NearLine-SAS Disk 7200 rpm.. The chassis itself is having a problem and now almost 3 month with IBM Support to get it rectified or get the box replaced.

    Since the EMC AX4 warranty and support is going to be finished soon this year, my focus on the Storage has gone a bit wide in selecting the best iSCSI SAN Storage..

    Any recommendation for a SAN Storage that would be the best fit for Exchange 2007?

    Thanks,


  8. Hussain, I’ll tackle this one in email. I’ll note, however, that my current employer made a name for itself as a NetApp reseller, and I picked them with that partially in mind.


  9. Hi Devin,

    I stumbled upon this writeup accidentally, I’m sorry for stirring it up again.
    We also run Exchange 2007 with .vdmk files on a NFS datastore in VMware (provided by a FAS2040 with all the fancy stuff turned on) and your post obviously got me worried.

    Can the remedy be as simple as :
    - create an iSCSI target on the NetApp
    - create a datastore on it
    - do a storage migration of the entire virtual machine to an iSCSI datastore ?

    If snapshotting is prohibited, what about stuff like deduplication and compression?

    Sincerely
    Guy


  10. That should be sufficient, yes. Make sure the iSCSI LUN is formatted as VMFS.

    Dedupe and compression should work. You may not get as much benefit from them as you think but they aren’t actively harmful as far as I know.

Add to the legend