Another solution for Autodiscover 401 woes in #MSExchange

Earlier tonight, I was helping a customer troubleshoot why users in their mixed Exchange 2013/2007 organization were getting 401 errors when trying to use Autodiscover to set up profiles. Well, more accurately, the Remote Connectivity Analyzer was getting a 401, and users were getting repeating authentication prompts. However, when we tested internally against the Autodiscover endpoints everything worked fine, and manual testing externally against the Autodiscover endpoint also worked.

So why did our manual tests work when the automated tests and Outlook didn’t?

Well, some will tell you it’s because of bad NTFS permissions on the virtual directory, while others will say it’s because of the loopback check being disabled. And in your case, that might in fact be the cause…but it wasn’t in mine.

In my case, the clue was in the Outlook authentication prompt (users and domains have been changed to protect the innocent):

image

 

I’m attempting to authenticate with the user’s UPN, and it’s failing…hey.

Re-run the Exchange Remote Connectivity analyzer, this time with the Domain\Username syntax, and suddenly I pass the Autodiscover test. Time to go view the user account – and sure enough, the account’s UPN is not set to the primary SMTP address.

Moral of the story: check your UPNs.

Upgrade Windows 2003 crypto in #MSExchange migrations

Just had this bite me at one of my customers. Situation: Exchange Server 2007 on Windows Server 2003 R2, upgrading to Exchange Server 2013 on Windows Server 2012. We ordered a new SAN certificate from GoDaddy (requesting it from Exchange 2013) and installed it on the Exchange 2013 servers with no problems. When we installed it on the Exchange 2007 servers, however, the certificates would import but the new certificates (and its chain) all showed the dreaded red X.

Looking at the certificate, we saw the following error message:

image

 

If you look more closely at the certificates in GoDaddy’s G2 root chain, you’ll see it’s signed both in SHA1 and SHA2-256. And the latter is the problem for Windows Server 2003 – it has an older cryptography library that doesn’t handle the newer cypher algorithms.

The solution: Install KB968730 on your Windows Server 2003 machines, reboot, and re-check your certificate. Now you should see the “This certificate is OK” message we all love.

Load Balancing ADFS on Windows 2012 R2

Greetings, everyone! I ran across this issue recently with a customer’s Exchange Server 2007 to Office 365 migration and wanted to pass along the lessons learned.

The Plan

It all started so innocently: the customer was going to deploy two Exchange Server 2013 hybrid servers into their existing Exchange Server 2007 organization for a Hybrid organization using directory synchronization and SSO with ADFS. They’ve been investing a lot of work into upgrading their infrastructure and have been upgrading systems to newer versions of Windows, including some spiffy new Windows Server 2012 Hyper-V servers. We decided that we’d deploy all of the new servers on Windows Server 2012 R2, the better to future-proof them. We were also going to use Windows NLB for the ADFS and ADFS proxy servers instead of using their existing F5 BIG-IP load balancer, as the network team is in the middle of their own projects.

The Problem

There were actually two problems. The first, of course, was the combination of Hyper-V and Windows NLB. Unicast was obviously no good, multicast has its issues, and because we needed to get the servers up and running as fast as possible we didn’t have time to explore using IGMP with Multicast. Time to turn to the F5. The BIG-IP platform is pretty complex and full of features, but F5 is usually good about documentation. Sure enough, the F5 ADFS 2.0 deployment guide (Deploying F5 with Microsoft Active Directory Federation Services) got us most of the way there. If we had been deploying ADFS  2.0 on Server 2012 and the ADFS proxy role, I’d have been home free.

In Windows 2012 R2 ADFS, you don’t have the ADFS proxy role any more – you use the Web Application Proxy (WAP) role service component of the Remote Access role. However, that’s not the only change. If you follow this guide with Windows Server 2012 R2, your ADFS and WAP pools will fail their health checks (F5 calls them monitors) and the virtual server will not be brought online because the F5 will mistakenly believe that your pool servers are down. OOPS!

The Resolution

So what’s different and how do we fix it?

ADFS on Windows Server 2012 R2 is still mostly ADFS 2.0, but some things have been changed – out with the ADFS proxy role, in with the WAP role service. That’s the most obvious change, but the real sticker here is under the hood in the guts of the Windows Server 2012 R2 HTTP server. In Windows Server 2012 R2, IIS and the Web server engine has a new architecture that supports the SNI extension to TLS. SNI is insanely cool. The connecting machine tells it what host name it’s trying to connect to as part of the HTTPS session setup so that one IP address can be used host multiple HTTPS sites with different certificates, just like HTTP 1.1 added the Hosts: header to HTTP.

But the fact that Windows 2012 R2 uses SNI gets in the way of the HTTPS health checks that the F5 ADFS 2.0 deployment guide has you configure. We were able to work around it by replacing the HTTPS health checks with TCP Half Open checks, which connect to the pool servers on the target TCP port and wait for the ACK. If they receive it, the server is marked up.

For long-term use, the HTTPS health checks are better; they allow you to configure the health check to probe a specific URL and get a specific response back before it declares a server in the pool is healthy. This is better than ICMP or TCP checks which only check for ping responses or TCP port responses. It’s totally possible for a machine to be up on the network and IIS answering connections but something is misconfigured with WAP or ADFS so it’s not actually a viable service. Good health checks save debugging time.

The Real Fix

As far as I know there’s no easy, supported way to turn SNI off, nor would I really want to; it’s a great standard that really needs to be widely deployed and supported because it will help servers conserve IP addresses and allow them to deploy multiple HTTPS sites on fewer IP/port combinations while using multiple certificates instead of big heavy SAN certificates. Ultimately, load balancer vendors and clients need to get SNI-aware fixes out for their gear.

If you’re an F5 user, the right way is to read and follow this F5 DevCentral blog post Big-IP and ADFS Part 5 – “Working with ADFS 3.0 and SNI” to configure your BIG-IP device with a new SNI-aware monitor; you’re going to want it for all of the Windows Server 2012 R2 Web servers you deploy over the next several years. This process is a little convoluted – you have to upload a script to the F5 and pass in custom parameters, which just seems really wrong (but is a true measure of just how powerful and beastly these machines really are) – but at the end of the day, you have a properly configured monitor that not only supports SNI connections to the correct hostname, but uses the specific URI to ensure that the ADFS federation XML is returned by your servers.

An SNI-aware F5 monitor (from DevCentral)

What do you do if you don’t have an F5 load balancer and your vendor doesn’t support F5? Remember when I said that there’s no way to turn SNI off? That’s not totally true. You can go mess with the SNI configuration and change the SSL bindings in a way that seems to mimic the old behavior. You run the risk of really messing things up, though. What you can do is follow the process in this TechNet blog post How to support non-SNI capable Clients with Web Application Proxy and AD FS 2012 R2.

 

Postscript

As a side note, almost everyone seems to be calling the ADFS flavor on Windows Server 2012 R2 “ADFS 3.0.” Everyone, that is, except for Microsoft. It’s not a 3.0; as I understand it the biggest differences have to do with the underlying server architecture, not the ADFS functionality on top of it per se. So don’t call it that, but recognize most other people will. It’s just AD FS 2012 R2.

Why Virtualization Still Isn’t Mature

As a long-time former advocate for Exchange virtualization (and virtualization in general), it makes me glad to see other pros pointing out the same conclusions I reached a while ago about the merits of Exchange virtualization. In general, it’s not a matter of whether you can solve the technological problems; I’ve spent years proving for customer after customer that you can. Tony does a great job of talking about the specific mismatch between Exchange and virtualization. I agree with everything he said, but I’m going to go one further and say that part of the problem is that virtualization is still an immature technology.

Now when I say that, you have to understand: I believe that virtualization is more than just the technology you use to run virtual machines. It includes the entire stack. And obviously, lots of people agree with me, because the core of private cloud technology is creating an entire stack of technology to wrap around your virtualization solution, such as Microsoft System Center or OpenStack. These solutions include software defined networking, operating system configuration, dynamic resource management, policy-driven allocation, and more. There are APIs, automation technologies, de facto standards, and interoperability technologies. The goal is to reduce or remove the amount of human effort required to deploy virtual solutions by bringing every piece of the virtualization pie under central control. Configure policies and templates and let automation use those to guide the creation and configuration of your specific instances, so that everything is consistent.

But there’s a missing piece – a huge one – one that I’ve been saying for years. And that’s the application layer. When you come right down to it, the Exchange community gets into brawls with the virtualization community (and the networking community, and the storage community, but let’s stay focused on one brawl at a time please) because there are two different and incompatible principles at play:

  • Exchange is trying to be as aware of your data as possible and take every measure to keep it safe, secure, and available by making specific assumption about how the system is deployed and configured.
  • Your virtualization product is trying to treat all applications (including Exchange) as if they are completely unaware of the virtualization stack and provide features and functionality whether they were designed for it or not.

The various stack solutions are using the right approach, but I believe they are doing it in the wrong direction; they work great in the second scenario, but they create exceptions and oddities for Exchange and other programs like Exchange that fit the first scenario. So what’s missing? How do I think virtualization stacks need to fix this problem?

Create a standard by which Exchange and other applications can describe what capabilities they offer and define the dependencies and requirements for those capabilities that must in turn be provided by the stack. Only by doing this can policy-driven private cloud solutions close that gap and make policies extend across the entire stack, continuing to reduce the change for human error.

With a standard like this, virtualizing Exchange would become a lot easier. As an example, consider VM to host affinity. Instead of admins having to remember to manually configure Exchange virtual DAG members to not be on the same host, Exchange itself would report  this requirement to the virtualization solution. DAG Mailbox servers would never be on the same host, and the FSW wouldn’t be on the same host as any of the Mailbox servers. And when host outages resulted in the loss of redundant hosts, the virtualization solution could throw an event caught by the monitoring system that explained the problem before you got into a situation where this constraint was broken. But don’t stop there. This same standard could be applied to network configuration, allowing Exchange and other applications to have load balancing automatically provisioned by the private cloud solution.  Or imagine deploying Exchange mailbox servers into a VMware environment that’s currently using NFS. The minute the Mailbox role is deployed, the automation carves off the appropriate disk blocks and presents them as iSCSI to the new VM (either directly or through the hypervisor as an RDM, based on the policy) so that the storage meets Exchange’s requirements.

Imagine the arguments that could solve. Instead of creating problems, applications and virtualization/private cloud stacks would be working together — a very model of maturity.

The iPhone wars, concluded

This happened not too long after I posted my last iPhone update, but I forgot to blog it until now.

I made the decision to get rid of the iPhone. There were a few things I liked about it, but overall, I found the user experience for core behavior and integration was just nowhere near the level of excellence provided by Windows Phone. Yes, I could have probably solved the problems I found by purchasing additional apps – I noticed that for the most part, the better apps are not the free ones – but it wouldn’t have solved the larger problems of each piece being just a piece, not part of a larger hole.

So, I ditched it and replaced the necessary functionality with a 4G access point. I still have the tethering when necessary but now it’s not driving down my phone battery, I only have one device to handle – one that I like – and I still don’t need to pass out my personal cell number, by simply giving my customers the option to call my main Lync number and forward the call to my cell.

So it was interesting, but ultimately…iPhones aren’t for me.

Let go of Windows XP, Office 2003, and Exchange 2003

The day has come. s the end of an era, one that many people do not want to let go. I can understand that.

I drove my last car, a Ford Focus 2000, until it died in the summer of 2010. I loved that car, and we seriously considered replacing the engine (which would have been a considerable chunk of money we didn’t have) so we could keep it. In the end, though, we had to take a long hard look between finances and our family requirements, and we moved on to a new vehicle. It was the requirements portion that was the key. It was certainly cheaper to fix the immediate problem – the blown engine – and we had friends who could do it for us professionally but inexpensively.

However, our kids were getting older. The four-door mini-sedan model wasn’t roomy enough for us and all of our stuff if we wanted to take a longer road trip like we’d been talking about. If we wanted to get a new sofa, we had to ask a friend with a truck. It would be nice, we thought, to have some additional carrying capacity for friends, family, groceries, and the occasional find from Craigslist. We’d been limiting our activities to those that were compatible with our car. With the new vehicle, we found we had far greater options.

On the road again
On the road again

 

Two years ago we took the entire family on a 2-week road trip across the United States, camping along the way. Last summer, we took our family down to Crater Lake, the California Redwoods, and the Oregon Coast. We’ve been to the Olympic Rain Forest. I’ve hauled Scouts and their gear home from Jamboree shakedowns. We’ve moved. We’ve hauled furniture. In short, we’ve found that our forced upgrade, although being more expensive in the long run, also gave us far more opportunity in the long run.

I know many of you like Windows XP. For some crazy reason, I know there are still quite a few of you out there who love Office 2003 and refused to let it go. I even still run across Exchange 2003 on a regular basis. I know that there is a certain mindset that says, “We paid for it, it’s not going to wear out, so we’re just going to keep using it.” Consider, if you will, the following points:

  • Software doesn’t wear out, per se, but it does age out. You have probably already seen this in action. It’s not limited to software – new cars get features the old cars don’t. However, when a part for an old car breaks down, it’s a relatively simple matter for a company to make replacement parts (either by reverse-engineering the original, or licensing it from the original car-maker). In the software world, there is a significant amount of work revolved in back-porting code from the new version and running it backwards several versions. There’s programming time, there’s testing time, and there’s support time. 10 years is more than just about any other software company out there (get any paid Linux support company to give you 10-year support for one up-front price). Microsoft is not trying to scam more money out of you. They want you to move on and stay relatively current with the rest of the world.
  • You are a safety hazard for others. There has been plenty written about the dangers of running XP past the end of life. There are some really good guides on how to mitigate the dangers. But make no mistake – you’re only mitigating them. And in a networked office or home, your risk is exposing other people to danger as well. Don’t be surprised in a couple of months, after one or two well-publicized large-scale malware breakouts targeting these ancient editions, that your business partners, ISP, and other vendors take strong steps to protect their networks by shutting down your access. When people won’t vaccinate and get sick, quarantine is a reasonable and natural response. If you don’t want to be the attack vector or the weakest link, get off your moral high ground and upgrade your systems.
  • This is why you can’t have nice things. Dude, you’re still running Windows XP. The best you have to look forward to is Internet Explorer 8, unless you download Firefox, Chrome, or some other browser. And even those guys are only going to put up with jumping through the hoops required to make XP work for so long. News flash: few software companies like supporting their applications on an operating system (or application platform) that itself is unsupported. You’re not going to find better anti-virus software for that ancient Exchange 2003 server. You’re going to be lucky to continue getting updates. And Office 2003 plug-ins and files? Over the next couple of years, you’re going to enjoy more and more cases of files that don’t work as planned with your old suite. Don’t even think about trying to install new software and applications on that old boat. You’ve picked your iceberg.

Look, I realize there are reasons why you’ve chosen to stay put. They make sense. They make financial sense. But Microsoft is not going to relent, and this choice is not going to go away, and it’s not going to get cheaper. Right now you still have a small window of time when you will have tools to help you get your data to a newer system. That opportunity is going away faster than you think. It will probably, if past experience serves, cost you more to upgrade at this time next year than it does now.

So do the right thing. Get moving. If you need help, you know where to find us. Don’t think about all the things the new stuff does that you don’t need; think about all the ways you could be making your life easier.

The enemy’s gate is down: lessons in #Lync

Sometimes what you need is a change in perspective.

I started my IT career as a technician: desktops and peripherals, printers, and the parts of networks not involving the actual building and deployment of servers. I quickly moved into the systems and network administration role. After 9/11 and a 16-month gap in my employment status, I met these guys and moved my career into a radically different trajectory – one that would take me to places I’d never dreamed of. From there, I moved into traditional consulting.

There is a different mindset between systems administration (operation) and consulting (architecture): the latter guy designs and builds the system, while the former guy keeps it running. Think of it like building a house. The contracting team are the experts at what current code is, how to get a crew going and keep them busy, how to navigate the permit process, and all the other things you need when designing and building a house. The people who buy the house and live there, though, don’t need that same body of knowledge. They may be able to do basic repairs and maintenance, but for remodels they may need to get some expert help. However, they’re also the people who have to live day in and day out with the compromises the architect and builders made. Those particular design decisions may be played out over tens of houses, with neither the designer nor the builder aware that it’s ultimately a poor choice and that a different set of decisions would have been better.

I personally find it helpful to have feet in both worlds. One of the drawbacks I’d had in working at Trace3 is that I was moving steadily away from my roots in systems administration. With Cohesive Logic, I’m getting to step somewhat back in the systems role. What I’m remembering is that there is a certain mindset good systems administrators have: when faced with a problem, they will work to synthesize a solution, even if it means going off the beaten path. The shift from “working within the limitations” to “creatively working around the limitations” is a mental reorientation much like that described in Ender’s Game: in a zero-G battle arena, the title characters realizes that carrying his outside orientation into battle was a liability. By re-visualizing the enemy’s gate as being “down”, Ender changed the entire axis of the conflict in ways both subtle and profound – and turned his misfit team into an unstoppable army.

enemys-gate-is-down

Case in point: I wanted to get my OCS/Lync Tanjay devices working our Lync Server 2013 deployment. This involved getting the firmware upgraded, which ended up being quite a challenge. In the end, I managed to do something pretty cool – get a Tanjay device running 1.0.x firmware to upgrade to 4.0.x in one jump against a native Lync Server 2013 deployment – something many Lync people said wasn’t possible.

Here’s how I did it.

All it took was a mental adjustment. Falling is effortless – so aim yourself to fall toward success.