Busting the Exchange Trusted Subsystem Myth

It’s amazing what kind of disruption leaving your job, looking for a new job, and starting to get settled in to a new job can have on your routines. Like blogging. Who knew?

At any rate, I’m back with some cool Exchange blogging. I’ve been getting a chance to dive into a “All-Devin, All-Exchange, All The Time” groove and it’s been a lot of fun, some of the details of which I hope to be able to share with you in upcoming months. In the process, I’ve been building a brand new Exchange 2010 lab environment and ran smack into a myth that seems to be making the rounds among people who are deploying Exchange 2010. This myth gives bum advice for those of you who are deploying an Exchange 2010 DAG and not using an Exchange 2010 Hub Transport as your File Share Witness (FSW). I call it the Exchange Trusted Subsystem Myth, and the first hint of it I see seems to be on this blog post. However, that same advice seems to have gotten around the net, as evidenced by this almost word-for-word copy or this posting that links to the first one. Like many myths, this one is pernicious not because it’s completely wrong, but because it works even though it’s wrong.

If you follow the Exchange product group’s deployment assumptions, you’ll never run into the circumstance this myth addresses; the FSW is placed on an Exchange 2010 HT role in the organization. Although you can specify the FSW location (server and directory) or let Exchange pick a server and directory or you, the FSW share isn’t created during the configuration of the DAG (as documented by fellow Exchange MVP Elan Shudnow and the “Witness Server Requirements” section of the Planning for High Availability and Site Resilience TechNet topic). Since it’s being created on an Exchange server as the second member of the DAG is joined, Exchange has all the permissions it needs on the system to create the share. If you elect to put the share on a non-Exchange server, then Exchange doesn’t have permissions to do it. Hence the myth:

  1. Add the FSW server’s machine account to the Exchange Trusted Subsystem group.
  2. Add the Exchange Trusted Subsystem group to the FSW server’s local Administrators group.

The sad part is, only the second action is necessary. True, doing the above will make the FSW work, but it will also open a much wider hole in your security than you need or want. Let me show you from my shiny new lab! In this configuration, I have three Exchange systems: EX10MB01, EX10MB02, and EX10MB03. All three systems have the Mailbox, Client Access, and Hub Transport roles. Because of this, I want to put the FSW on a separate machine. I could have used a generic member server, but I specifically wanted to debunk the myth, so I picked my DC EX10DC01 with malice aforethought.

  • In Figure 1, I show adding the Exchange Trusted Subsystem group to the Builtin/Administrators group on EX10DC01. If this weren’t a domain controller, I could add it to the local Administrators group instead, but DCs require tinkering. [1]

ExTrSubSys-DC-AdminsGroup
Figure 1: Membership of the Builtin/Administrators group on EX10DC01

  • In Figure 2, I show the membership of the Builtin/Administrators group on EX10DC01. No funny business up my sleeve!

ExTrSubSys-Members
Figure 2: Membership of the Exchange Trusted Subsystem group

  • I now create the DAG object, specifying EX10DC01 as my FSW server and the C:\EX10DAG01 directory so we can see if it ever gets created (and when).
  • In Figure 3, I show the root of the C:\ drive on EX10DC01 after adding the second Exchange 2010 server to the DAG. Now, the directory and share are created, without requiring the server’s machine account to be added to the Exchange Trusted Subsystem group.

ExTrSubSys-FSWCreated
Figure 3: The FSW created on EX10DC01

I suspect that this bad advice came about through a combination of circumstances, including an improper understanding of Exchange caching of Active Directory information and when the FSW is actually created. However it came about, though, it needs to be stopped, because any administrator that configures their Exchange organization is opening a big fat hole in the Exchange security model.

So, why is adding the machine account to the Exchange Trusted Subsystem group a security hole? The answer lies in Exchange 2010’s shift to Role Based Access Control (RBAC). In previous versions of Exchange, you delegated permissions directly to Active Directory and Exchange objects, allowing users to perform actions directly from their security context. If they had the appropriate permissions, their actions succeeded.

In Exchange 2010 RBAC, this model goes away; you now delegate permissions by telling RBAC what options given groups, policies, or users can perform, then assigning group memberships or policies as needed. When the EMS cmdlets run, they do so as the local machine account; since the local machine is an Exchange 2010 server, this account has been added to the Exchange Trusted Subsystem group. This group has been delegated the appropriate access entries in Active Directory and Exchange databases objects, as described in the Understanding Split Permissions TechNet topic. For a comprehensive overview of RBAC and how all the pieces fit together, read the Understanding Role Based Access Control TechNet topic.

By improperly adding a non-Exchange server to this group, you’re now giving that server account the ability to read and change any Exchange-related object or property in Active Directory or Exchange databases. Obviously, this is a hole, especially given the relative ease with which one local administrator can get a command line prompt running as one of the local system accounts.

So please, do us all a favor: if you ever hear or see someone passing around this myth, please, link them here.

ExTrSubSys-Busted
Busted!

[1] Yes, it is also granting much broader permissions than necessary to make a DC the FSW node. Now the Exchange Trusted Subsystem group is a member of the Domain Admins group. This is probably not what you want to do, so really, don’t do this outside of a demo lab.

28 Comments



  1. Absolutely correct – I need to update that post. That “myth” occurred becase during that Beta/RC timeframe there was no documentation released about what the necessay permissions were so it was purely me poking around to see what happened. Definitely went looking on Technet and most of the DAG articles were empty placeholders back then, so doin the correct thing was essentially impossible. . Good post though – this should clear it up.



  2. Hey Devin, thanks for the link. I’m a bit surprised to see those errors on those two blogs about adding the computer account to the Exchange Trusted Subsystem group when indeed, that is not needed. I posted on those 2 blogs hoping they’ll change their guidance.


  3. Tom — I figured that’s a large part of what it was. Between lack of documentation and the incremental process of making one change, seeing if it worked, making another change, seeing if it worked, and then caching finally updating so that the change you make that should have been enough not working yet…

    Elan — dude, you keep rocking the house.




  4. Great article. Your text in the bullet above figure 2 is a little confusing.


  5. Chris Johnston

    Well I would have hoped it would be blinding obvious not to set a DC as a FSW in your example regardless whether you did step 1 and/or two. So perhaps using a DC in your example isn’t the best idea. However if you have a requirement to use a DC (as we do) how can a DC be used without laxing security? I don’t have a lab handy to check however beyond creating the FSW folder are there any additional security requirements for the FSW? If not surely manually creating the FSW folder and adding Exchange Trusted Subsystem to the permissions on the folder would be sufficient?


  6. Chris,

    As I said in the post:

    “I could have used a generic member server, but I specifically wanted to debunk the myth, so I picked my DC EX10DC01 with malice aforethought.”

    Regarding using a DC: you really don’t have any other member server that’s more suitable? If I were your consultant, I would invest a few minutes of time trying to dig into the reasoning that led to this requirement. I’m not saying your wrong — I can think of a couple of scenarios that would lead to that requirement — but in each of those scenarios, I can also think of alternatives.

    If I really truly needed to put the FSW on a DC, check out this blog post that talks about the permissions needed for manual FSW creation:

    http://runebelune.spaces.live.com/blog/cns!214999A8EBE4FCB4!188.entry

    Make sure you keep an eye on that directory/file share on a regular basis, though, so that an unexpected permission change doesn’t wipe it out just when you need it.



  7. Steen Kirkby

    This sounds very resonable. I too can not see any reason why the FSW computer account should be included in the “Exchange Trusted Subsystem” group. It’s quite logical that our Exchange servers need permissions on the FSW computer in order to create the folder, share it and set permissions hence “Exchange Trusted Subsystem” should be in the Administrators group on the FSW computer.
    There is no logical reason why our FSW computer should need permissions to our Exchange Servers (which it will get if we put the FSW computer account in the “Exchange Trusted Subsystem” group)
    However when I try this with the Exchange 2010 SP1 EMC after only adding the “Exchange Trusted Subsystem” group to the local administrators group on my FSW computer I keep getting this error when creating the DAG:

    Warning:
    The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server fsw.contoso.com.

    Exchange Management Shell command completed:
    New-DatabaseAvailabilityGroup -Name ‘DAG02′ -WitnessServer ‘fsw.contoso.com’ -WitnessDirectory ‘C:\DAG01FSW’

    The strange thing is that despite what the error says the Exchange Trusted Subsystem really IS in the local Administrators group on fsw.contoso.com (it’s the exact only thing that I did)

    What’s even more strange is that if use the “myth” trick and add my FSW computer account to the “Exchange Trusted Subsystem” group then away goes the error…

    Are you sure you didn’t at some point have your FSW computer account in the “Exchange Trusted Subsystem” group and that membership might have been cached at startup and still effective for a while after you removed it again?

    Sorry for questioning it. I agree that theoretically it makes no sense to put it in the “Exchange Trusted Subsystem” group but I just can’t get it to work without – or maybe it’s just the EMC (perhaps the SP1 version) that’s keeping the myth alive by displaying an error that actually should be ignored?


  8. I’m positive on the build-out for that lab.

    I’ll have to test this for SP1 when I get a chance. I hope nothing changed…

  9. Steen Kirkby

    Hi again – I just got my DAG up and running. I had some problems adding a second DAG member due to Symantec Endpoint Protection (now removed). While troubleshooting I created a brand new member server just for the FSW and only added “Exchange Trusted Subsystem” to the local Administrators group on it.

    The DAG (and my brand new dedicated FSW) seems to work fine despite the error in EMC while creating the DAG so it’s probably just the EMC that gives an incorrect error. It’s bit strange that it disappears when adding the FSW computer account to the “Exchange Trusted Subsystem” group – looks like a bug in the SP1 EMC. Maybe they still have some remains of the myth alive @ Microsoft ;)


  10. Laura

    What about Alternate FSW implemented in 2010 SP1? If I wanted to put a AFSW share on a DC but I did not want to add Exchange Trusted Subsystem to the Domain Admins group, but I did give Full Control to Exchange Trusted Subsystem group and also Owner, would there be any issues? Thanks


  11. Laura,

    The Alternate FSW functionality doesn’t change the basic scenario here, which is that placing a FSW on a DC requires you to either open up more permissions than needed (for automatic creation) or manual pre-creation.

    If it were me, I’d manually create the alternate FSW if I absolutely had to have it on a DC. That way it’s all ready to go if I have to switch over to it and I don’t have to muck around with permissions.

  12. Simon

    How about placing the FSW on a Netapp filer? Would that work with the same workaround?


  13. Devin,

    As per Steen’s comment I have found that the old “myth” trick is required to remove the FSW error. What is more, is that not doing this also means the FSW is not usable as a cluster vote (and this subsequently causes the cluster to fail in the event of a minority node failure :( )

    I presume something changed in SP1?



  14. Actually, the reason you get the error message is not for the reason you think. If you just put the ETS group in the local Administrators group and then go on with adding DAG members, you will get the error message, but you will also see the DAG file share be created and the subsequent files and folders within it. Look for a new post on this soon.




Add to the legend