Another solution for Autodiscover 401 woes in #MSExchange

Earlier tonight, I was helping a customer troubleshoot why users in their mixed Exchange 2013/2007 organization were getting 401 errors when trying to use Autodiscover to set up profiles. Well, more accurately, the Remote Connectivity Analyzer was getting a 401, and users were getting repeating authentication prompts. However, when we tested internally against the Autodiscover endpoints everything worked fine, and manual testing externally against the Autodiscover endpoint also worked.

So why did our manual tests work when the automated tests and Outlook didn’t?

Well, some will tell you it’s because of bad NTFS permissions on the virtual directory, while others will say it’s because of the loopback check being disabled. And in your case, that might in fact be the cause…but it wasn’t in mine.

In my case, the clue was in the Outlook authentication prompt (users and domains have been changed to protect the innocent):

image

 

I’m attempting to authenticate with the user’s UPN, and it’s failing…hey.

Re-run the Exchange Remote Connectivity analyzer, this time with the Domain\Username syntax, and suddenly I pass the Autodiscover test. Time to go view the user account – and sure enough, the account’s UPN is not set to the primary SMTP address.

Moral of the story: check your UPNs.

Upgrade Windows 2003 crypto in #MSExchange migrations

Just had this bite me at one of my customers. Situation: Exchange Server 2007 on Windows Server 2003 R2, upgrading to Exchange Server 2013 on Windows Server 2012. We ordered a new SAN certificate from GoDaddy (requesting it from Exchange 2013) and installed it on the Exchange 2013 servers with no problems. When we installed it on the Exchange 2007 servers, however, the certificates would import but the new certificates (and its chain) all showed the dreaded red X.

Looking at the certificate, we saw the following error message:

image

 

If you look more closely at the certificates in GoDaddy’s G2 root chain, you’ll see it’s signed both in SHA1 and SHA2-256. And the latter is the problem for Windows Server 2003 – it has an older cryptography library that doesn’t handle the newer cypher algorithms.

The solution: Install KB968730 on your Windows Server 2003 machines, reboot, and re-check your certificate. Now you should see the “This certificate is OK” message we all love.

Load Balancing ADFS on Windows 2012 R2

Greetings, everyone! I ran across this issue recently with a customer’s Exchange Server 2007 to Office 365 migration and wanted to pass along the lessons learned.

The Plan

It all started so innocently: the customer was going to deploy two Exchange Server 2013 hybrid servers into their existing Exchange Server 2007 organization for a Hybrid organization using directory synchronization and SSO with ADFS. They’ve been investing a lot of work into upgrading their infrastructure and have been upgrading systems to newer versions of Windows, including some spiffy new Windows Server 2012 Hyper-V servers. We decided that we’d deploy all of the new servers on Windows Server 2012 R2, the better to future-proof them. We were also going to use Windows NLB for the ADFS and ADFS proxy servers instead of using their existing F5 BIG-IP load balancer, as the network team is in the middle of their own projects.

The Problem

There were actually two problems. The first, of course, was the combination of Hyper-V and Windows NLB. Unicast was obviously no good, multicast has its issues, and because we needed to get the servers up and running as fast as possible we didn’t have time to explore using IGMP with Multicast. Time to turn to the F5. The BIG-IP platform is pretty complex and full of features, but F5 is usually good about documentation. Sure enough, the F5 ADFS 2.0 deployment guide (Deploying F5 with Microsoft Active Directory Federation Services) got us most of the way there. If we had been deploying ADFS  2.0 on Server 2012 and the ADFS proxy role, I’d have been home free.

In Windows 2012 R2 ADFS, you don’t have the ADFS proxy role any more – you use the Web Application Proxy (WAP) role service component of the Remote Access role. However, that’s not the only change. If you follow this guide with Windows Server 2012 R2, your ADFS and WAP pools will fail their health checks (F5 calls them monitors) and the virtual server will not be brought online because the F5 will mistakenly believe that your pool servers are down. OOPS!

The Resolution

So what’s different and how do we fix it?

ADFS on Windows Server 2012 R2 is still mostly ADFS 2.0, but some things have been changed – out with the ADFS proxy role, in with the WAP role service. That’s the most obvious change, but the real sticker here is under the hood in the guts of the Windows Server 2012 R2 HTTP server. In Windows Server 2012 R2, IIS and the Web server engine has a new architecture that supports the SNI extension to TLS. SNI is insanely cool. The connecting machine tells it what host name it’s trying to connect to as part of the HTTPS session setup so that one IP address can be used host multiple HTTPS sites with different certificates, just like HTTP 1.1 added the Hosts: header to HTTP.

But the fact that Windows 2012 R2 uses SNI gets in the way of the HTTPS health checks that the F5 ADFS 2.0 deployment guide has you configure. We were able to work around it by replacing the HTTPS health checks with TCP Half Open checks, which connect to the pool servers on the target TCP port and wait for the ACK. If they receive it, the server is marked up.

For long-term use, the HTTPS health checks are better; they allow you to configure the health check to probe a specific URL and get a specific response back before it declares a server in the pool is healthy. This is better than ICMP or TCP checks which only check for ping responses or TCP port responses. It’s totally possible for a machine to be up on the network and IIS answering connections but something is misconfigured with WAP or ADFS so it’s not actually a viable service. Good health checks save debugging time.

The Real Fix

As far as I know there’s no easy, supported way to turn SNI off, nor would I really want to; it’s a great standard that really needs to be widely deployed and supported because it will help servers conserve IP addresses and allow them to deploy multiple HTTPS sites on fewer IP/port combinations while using multiple certificates instead of big heavy SAN certificates. Ultimately, load balancer vendors and clients need to get SNI-aware fixes out for their gear.

If you’re an F5 user, the right way is to read and follow this F5 DevCentral blog post Big-IP and ADFS Part 5 – “Working with ADFS 3.0 and SNI” to configure your BIG-IP device with a new SNI-aware monitor; you’re going to want it for all of the Windows Server 2012 R2 Web servers you deploy over the next several years. This process is a little convoluted – you have to upload a script to the F5 and pass in custom parameters, which just seems really wrong (but is a true measure of just how powerful and beastly these machines really are) – but at the end of the day, you have a properly configured monitor that not only supports SNI connections to the correct hostname, but uses the specific URI to ensure that the ADFS federation XML is returned by your servers.

An SNI-aware F5 monitor (from DevCentral)

What do you do if you don’t have an F5 load balancer and your vendor doesn’t support F5? Remember when I said that there’s no way to turn SNI off? That’s not totally true. You can go mess with the SNI configuration and change the SSL bindings in a way that seems to mimic the old behavior. You run the risk of really messing things up, though. What you can do is follow the process in this TechNet blog post How to support non-SNI capable Clients with Web Application Proxy and AD FS 2012 R2.

 

Postscript

As a side note, almost everyone seems to be calling the ADFS flavor on Windows Server 2012 R2 “ADFS 3.0.” Everyone, that is, except for Microsoft. It’s not a 3.0; as I understand it the biggest differences have to do with the underlying server architecture, not the ADFS functionality on top of it per se. So don’t call it that, but recognize most other people will. It’s just AD FS 2012 R2.

The iPhone wars, concluded

This happened not too long after I posted my last iPhone update, but I forgot to blog it until now.

I made the decision to get rid of the iPhone. There were a few things I liked about it, but overall, I found the user experience for core behavior and integration was just nowhere near the level of excellence provided by Windows Phone. Yes, I could have probably solved the problems I found by purchasing additional apps – I noticed that for the most part, the better apps are not the free ones – but it wouldn’t have solved the larger problems of each piece being just a piece, not part of a larger hole.

So, I ditched it and replaced the necessary functionality with a 4G access point. I still have the tethering when necessary but now it’s not driving down my phone battery, I only have one device to handle – one that I like – and I still don’t need to pass out my personal cell number, by simply giving my customers the option to call my main Lync number and forward the call to my cell.

So it was interesting, but ultimately…iPhones aren’t for me.

Let go of Windows XP, Office 2003, and Exchange 2003

The day has come. s the end of an era, one that many people do not want to let go. I can understand that.

I drove my last car, a Ford Focus 2000, until it died in the summer of 2010. I loved that car, and we seriously considered replacing the engine (which would have been a considerable chunk of money we didn’t have) so we could keep it. In the end, though, we had to take a long hard look between finances and our family requirements, and we moved on to a new vehicle. It was the requirements portion that was the key. It was certainly cheaper to fix the immediate problem – the blown engine – and we had friends who could do it for us professionally but inexpensively.

However, our kids were getting older. The four-door mini-sedan model wasn’t roomy enough for us and all of our stuff if we wanted to take a longer road trip like we’d been talking about. If we wanted to get a new sofa, we had to ask a friend with a truck. It would be nice, we thought, to have some additional carrying capacity for friends, family, groceries, and the occasional find from Craigslist. We’d been limiting our activities to those that were compatible with our car. With the new vehicle, we found we had far greater options.

On the road again
On the road again

 

Two years ago we took the entire family on a 2-week road trip across the United States, camping along the way. Last summer, we took our family down to Crater Lake, the California Redwoods, and the Oregon Coast. We’ve been to the Olympic Rain Forest. I’ve hauled Scouts and their gear home from Jamboree shakedowns. We’ve moved. We’ve hauled furniture. In short, we’ve found that our forced upgrade, although being more expensive in the long run, also gave us far more opportunity in the long run.

I know many of you like Windows XP. For some crazy reason, I know there are still quite a few of you out there who love Office 2003 and refused to let it go. I even still run across Exchange 2003 on a regular basis. I know that there is a certain mindset that says, “We paid for it, it’s not going to wear out, so we’re just going to keep using it.” Consider, if you will, the following points:

  • Software doesn’t wear out, per se, but it does age out. You have probably already seen this in action. It’s not limited to software – new cars get features the old cars don’t. However, when a part for an old car breaks down, it’s a relatively simple matter for a company to make replacement parts (either by reverse-engineering the original, or licensing it from the original car-maker). In the software world, there is a significant amount of work revolved in back-porting code from the new version and running it backwards several versions. There’s programming time, there’s testing time, and there’s support time. 10 years is more than just about any other software company out there (get any paid Linux support company to give you 10-year support for one up-front price). Microsoft is not trying to scam more money out of you. They want you to move on and stay relatively current with the rest of the world.
  • You are a safety hazard for others. There has been plenty written about the dangers of running XP past the end of life. There are some really good guides on how to mitigate the dangers. But make no mistake – you’re only mitigating them. And in a networked office or home, your risk is exposing other people to danger as well. Don’t be surprised in a couple of months, after one or two well-publicized large-scale malware breakouts targeting these ancient editions, that your business partners, ISP, and other vendors take strong steps to protect their networks by shutting down your access. When people won’t vaccinate and get sick, quarantine is a reasonable and natural response. If you don’t want to be the attack vector or the weakest link, get off your moral high ground and upgrade your systems.
  • This is why you can’t have nice things. Dude, you’re still running Windows XP. The best you have to look forward to is Internet Explorer 8, unless you download Firefox, Chrome, or some other browser. And even those guys are only going to put up with jumping through the hoops required to make XP work for so long. News flash: few software companies like supporting their applications on an operating system (or application platform) that itself is unsupported. You’re not going to find better anti-virus software for that ancient Exchange 2003 server. You’re going to be lucky to continue getting updates. And Office 2003 plug-ins and files? Over the next couple of years, you’re going to enjoy more and more cases of files that don’t work as planned with your old suite. Don’t even think about trying to install new software and applications on that old boat. You’ve picked your iceberg.

Look, I realize there are reasons why you’ve chosen to stay put. They make sense. They make financial sense. But Microsoft is not going to relent, and this choice is not going to go away, and it’s not going to get cheaper. Right now you still have a small window of time when you will have tools to help you get your data to a newer system. That opportunity is going away faster than you think. It will probably, if past experience serves, cost you more to upgrade at this time next year than it does now.

So do the right thing. Get moving. If you need help, you know where to find us. Don’t think about all the things the new stuff does that you don’t need; think about all the ways you could be making your life easier.

The enemy’s gate is down: lessons in #Lync

Sometimes what you need is a change in perspective.

I started my IT career as a technician: desktops and peripherals, printers, and the parts of networks not involving the actual building and deployment of servers. I quickly moved into the systems and network administration role. After 9/11 and a 16-month gap in my employment status, I met these guys and moved my career into a radically different trajectory – one that would take me to places I’d never dreamed of. From there, I moved into traditional consulting.

There is a different mindset between systems administration (operation) and consulting (architecture): the latter guy designs and builds the system, while the former guy keeps it running. Think of it like building a house. The contracting team are the experts at what current code is, how to get a crew going and keep them busy, how to navigate the permit process, and all the other things you need when designing and building a house. The people who buy the house and live there, though, don’t need that same body of knowledge. They may be able to do basic repairs and maintenance, but for remodels they may need to get some expert help. However, they’re also the people who have to live day in and day out with the compromises the architect and builders made. Those particular design decisions may be played out over tens of houses, with neither the designer nor the builder aware that it’s ultimately a poor choice and that a different set of decisions would have been better.

I personally find it helpful to have feet in both worlds. One of the drawbacks I’d had in working at Trace3 is that I was moving steadily away from my roots in systems administration. With Cohesive Logic, I’m getting to step somewhat back in the systems role. What I’m remembering is that there is a certain mindset good systems administrators have: when faced with a problem, they will work to synthesize a solution, even if it means going off the beaten path. The shift from “working within the limitations” to “creatively working around the limitations” is a mental reorientation much like that described in Ender’s Game: in a zero-G battle arena, the title characters realizes that carrying his outside orientation into battle was a liability. By re-visualizing the enemy’s gate as being “down”, Ender changed the entire axis of the conflict in ways both subtle and profound – and turned his misfit team into an unstoppable army.

enemys-gate-is-down

Case in point: I wanted to get my OCS/Lync Tanjay devices working our Lync Server 2013 deployment. This involved getting the firmware upgraded, which ended up being quite a challenge. In the end, I managed to do something pretty cool – get a Tanjay device running 1.0.x firmware to upgrade to 4.0.x in one jump against a native Lync Server 2013 deployment – something many Lync people said wasn’t possible.

Here’s how I did it.

All it took was a mental adjustment. Falling is effortless – so aim yourself to fall toward success.

The iPhone Wars, Day 121

120 days later and I figured it was time for an update on the war.

First: I still hate this thing.

Somewhere along the way with one of the iOS updates, the battery life started going to crap, even when I’m barely using the device. When I use it as a personal hotspot, I can practically watch the battery meter race to zero.

I’ve nailed down what it is about the email client that I don’t like, and it’s the same thing I don’t like about many of the apps: the user interfaces are inconsistent and cramped. Navigating my way through a breadcrumb trail that is up near (but not quite) up at the top just feels clunky. This is where contrast with Windows Phone really, really hurts the iPhone in my experience; the Metro (I know, we’re not supposed to call it that anymore, but they can bite me) user interface principles are clean and clear. Trying to figure out simple tasks like how to get the iPhone to actually resync is more complex than necessary. Having the “new message” icon down on the bottom when the navigation is up top is stupid.

The one thing that impresses me consistently is even though the screen is small, the on-screen keyboard is really good at figuring out which letter I am trying to hit. On my Windows Phone I mistype things all the time. This rarely happens on the iPhone. Even though the on-screen keys are much smaller, the iPhone typing precision is much higher. Microsoft, take note – I’m tired of what feels like pressing on one key only to have another key grab the focus.

Even the few custom apps I do use on this iPhone fail to impress. Thanks to a lack of consistent design language, learning one doesn’t help me with the rest, and I have discovered that iPhone developers are just as bad as Windows Phone developers when it comes to inexplicable gaps in functionality.

I guess no one knows how to write good mobile software yet.

The iPhone Wars, Day 1

Part of the fun of settling into a new job is the new tools. In this trade, that’s the laptop and the cell phone. Now, I already have a perfectly good laptop and cell phone, so I probably could have just gone on using those, but where so much of what I do is from home, I find it important to have a clear break between personal business and work. Having separate devices helps me define that line.

My current cell phone is a Nokia Lumia 1020 (Windows Phone 8), which I definitely enjoy. I haven’t had a good chance to take the camera for a full spin, but I’m looking forward to it. I’ve had a lot of PDAs and smart phones in my time: Palm Pilot, Handspring Visor, Windows Mobile, BlackBerry, Windows Phone 7, even an Android. The one I’ve never had, though, is an iPhone.

And it’s not that I hate Apple. My favorite past laptop was my MacBook Pro (Apple has ruined me for any other touchpad). Granted, I’m that weird bastard who loaded Vista SP1 into Boot Camp and hardly ever booted back into Mac OS X again, but ever since then I’ve usually had a spare Apple computer around the house, if only for Exchange interop testing. OS X is a good operating system, but it’s not my favorite, so my main device is always a Windows machine. My current favorite is my Surface Pro.

In all of that, though, I’ve never had an iOS device. Never an iPhone, never an iPad. Yesterday, that all changed.

I needed a business smart phone that runs a specific application, one that hasn’t yet been ported to Windows Phone. I’ve long been an advocate that “apps matter first; pick your OS and platform after you know what apps you need.” Here was my opportunity not to be a shining hypocrite! After discussion with Jeremy, I finally settled on a iPhone 5, as Android was going to be less suitable for reasons too boring to go into.

So now I have an iPhone, and I have just one question for you iPhone-lovers of the world: You really like this thing? Honest to goodness, no one is putting a gun to your head?

I can’t stand this bloody thing! First, it’s too damn small! I mean, yes, I like my smart phones somewhat large, but I have big hands and I have pockets. The iPhone 5 is a slim, flat little black carbon slab with no heft. I’ve taken to calling it the CSD – the Carbon Suppository of Death. Now, if it were just the form factor, I could get used to it, but there’s so much more that I can’t stand:

  • I didn’t realize how much I love the Windows Phone customizable menu until I wasn’t using it. I forget who once called the iPhone (and Android) menu “Program Manager Reborn” but it’s totally apt. Plus, all the chrome (even in iOS 7) just feels cluttered and junky now.
  • Speaking of cluttered, Apple sometimes takes the minimalist thing too far. One button is not enough. This, I think, Windows Phone nails perfectly. Android’s four buttons feel extraneous, but Apple’s “let there be one” approach feels like dogma that won’t bow to practicality.
  • The last time I used an iPod, it was still black & white. I can’t stand iTunes as a music manager, and I don’t like the device-side interface – so I won’t be putting any music on the CSD. No advantage there.
  • Likewise, you think I’m going to dink around with the camera on the CSD when I have the glorious Lumia camera to use? Get real, human.
  • The on-screen keyboard sucks. Part of this is because the device is so much smaller, but part of it is that Apple doesn’t seem to understand little touches that improve usability. On Windows and Android, when you touch the shift key, the case of the letters on the keys changes correspondingly; Apple is all, “LOL…NOPE!”
  • Even the Mail client irritates me, even though I haven’t managed to put my finger on exactly why yet.

So is there anything I like about the device? Sure! I’m not a total curmudgeon:

  • Build quality looks impressive. If the CSD wasn’t as flimsy as a communion wafer, I would be blown away by the feel of the device. It’s got good clean lines and understated elegance, like that suit from the expensive Saville Row tailors.
  • Power usage. The CSD goes through battery very slowly. Now part of that is because I’m not using it, but Apple has had time to optimize their game, and they do it very well indeed.
  • The simple little physical switch to put the CSD into silent mode. This is exactly the kind of physical control EVERY smart phone should have, just like every laptop should have a physical switch to disable the radios (not just a hotkey combination).

This is where I’m at, with a fistful of suck. Even an Android phone would be better than this. I’ve got no-one to blame but myself, and it’s not going to get any better. So look forward to more of these posts from time to time as I find yet another aspect of the CSD that drives me crazy.

“But Devin,” I hear some of you Apple-pandering do-gooders say, “You’re just not used to it yet. Give it time. You’ll grow to love it.”

CHALLENGE ACCEPTED.

Meet the New Corporate Overlords @CohesiveLogic

Just a brief announcement (you’ll be hearing more about it later) to let everyone know that I’ve found new employment with Cohesive Logic as a Principal Consultant. Jeremy and the rest are good people and I’m happy to be hanging my hat under their shingle. We’ve got some exciting stuff coming down the pipe, and while I’ll still be focusing on Exchange, I’ll have the opportunity to broaden my skill set.

Back on the market

I sent out a brief tweet about this Friday and have received a number of queries, so I figured I should expand on this publicly.

No, I am no longer with Trace3. No, this was not my decision — I was happy with my position and work and was excited about what was happening there. At the same time, this was not a complete shock. I’m not at liberty to go into it (and even if I were, I don’t think I would anyway) but all living organisms (including vibrant corporations) change, and while many of those changes are good for the organism as a whole, they aren’t always so great for individual cells.

I have no hard feelings. I had a fantastic time at Trace3 and have learned a lot. I wish everyone there all the success in the world and am reasonably confident they’ll grab it. At the same time, there were some aspects of my fit at Trace3 that could have been improved on. Always being remote with no local co-workers, for one — that was a definite downer.

I’m feeling confident in my ability to find my next job. I have some exciting opportunities under way. In the meantime, though, if you have a lead or opportunity, let me know — and yes, that does include the potential for 1099 independent consulting work.

Beating Verisign certificate woes in Exchange

I’ve seen this problem in several customers over the last two years, and now I’m seeing signs of it in other places. I want to document what I found so that you can avoid the pain we had to go through.

The Problem: Verisign certificates cause Exchange publishing problems

So here’s the scenario: you’re deploying Exchange 2010 (or some other version, this is not a version-dependent issue with Exchange) and you’re using a Verisign certificate to publish your client access servers. You may be using a load balancer with SSL offload or pass-through, a reverse proxy like TMG 2010, some combination of the above, or you may even be publishing your CAS roles directly. However you publish Exchange, though, you’re running into a multitude of problems:

  • You can’t completely pass ExRCA’s validation checks. You get an error something like:  The certificate is not trusted on any version of Windows Phone device. Root = CN=VeriSign Class 3 Public Primary Certification Authority – G5, OU=”(c) 2006 VeriSign, Inc. – For authorized use only”, OU=VeriSign Trust Network, O=”VeriSign, Inc.”, C=US
  • You have random certificate validation errors across a multitude of clients, typically mobile clients such as smartphones and tablets. However, some desktop clients and browsers may show issues as well.
  • When you view the validation chain for your site certificate on multiple devices, they are not consistent.

These can be very hard problems to diagnose and fix; the first time I ran across it, I had to get additional high-level Trace3 engineers on the call along with the customer and a Microsoft support representative to help figure out what the problem was and how to fix it.

The Diagnosis: Cross-chained certificates with an invalid root

So what’s causing this difficult problem? It’s your basic case of a cross-chained certificate with an invalid root certificate. “Oh, sure,” I hear you saying now. “That clears it right up then.” The cause sounds esoteric, but it’s actually not hard to understand when you remember how certificates work: through a chain of validation. Your Exchange server certificate is just one link in an entire chain. Each link is represented by an X.509v3 digital certificate that is the footprint of the underlying server it represents.

At the base of this chain (aka the root) is the root certificate authority (CA) server. This digital certificate is unique from others because it’s self-signed – no other CA server has signed this server’s certificate. Now, you can use a root CA server to issue certificates to customers, but that’s actually a bad idea to do for a lot of reasons. So instead, you have one or more intermediate CA servers added into the chain, and if you have multiple layers, then the outermost layer are the CA servers that process customer requests. So a typical commercially generated certificate has a validation chain of 3-4 layers: the root CA, one or two intermediate CAs, and your server certificate.

Remember how I said there were reasons to not use root CAs to generate customer certificates? You can probably read up on the security rationales behind this design, but some of the practical reasons include:

  • The ability to offer different classes of service, signed by separate root servers. Instead of having to maintain separate farms of intermediate servers, you can have one pool of intermediate servers that issue certificates for different tiers of service.
  • The ability to retire root and intermediate CA servers without invalidating all of the certificates issued through that root chain, if the intermediate CA servers cross-chain from multiple roots. That is, the first layer intermediate CA servers’ certificates are signed by multiple root CA servers, and the second layer intermediate CA servers’ certificates are signed by multiple intermediate CA servers from the first layer.

So, cross-chaining is a valid practice that helps provide redundancy for certificate authorities and helps protect your investment in certificates. Imagine what a pain it would be if one of your intermediate CAs got revoked and nuked all of your certificates. I’m not terribly fond of having to redeploy certificates for my whole infrastructure without warning.

However, sometimes cross-chained certificates can cause problems, especially when they interact with another feature of the X.509v3 specification: the Authority Information Access (AIA) certificate extension. Imagine a situation where a client (such as a web browser trying to connect to OWA), presented with an X.509v3 certificate for an Exchange server, cannot validate the certificate chain because it doesn’t have the upstream intermediate CA certificate.

If the Exchange server certificate has the AIA extension, the client has the information it needs to try to retrieve the missing intermediate CA certificate – either retrieving it from the HTTPS server, or by contacting a URI from the CA to download it directly. This only works for intermediate CA certificates; you can’t retrieve the root CA certificate this way. So, if you are missing the entire certificate chain, AIA won’t allow you to validate it, but as long as you have the signing root CA certificate, you can fill in any missing intermediate CA certificates this way.

Here’s the catch: some client devices can only request missing certificates from the HTTPS server. This doesn’t sound so bad…but what happens if the server’s certificate is cross-chained, and the certificate chain on the server goes to a root certificate that the device doesn’t have…even if it does have another valid root to another possible chain? What happens is certificate validation failure, on a certificate that tested as validated when you installed it on the Exchange server.

I want to note here that I’ve only personally seen this problem with Verisign certificates, but it’s a potential problem for any certificate authority.

The Fix: Disable the invalid root

We know the problem and we know why it happens. Now it’s time to fix it by disabling the invalid root.

Step #1 is find the root. Fire up the Certificates MMC snap-in, find your Exchange server certificate, and view the certificate chain properties. This is what the incorrect chain has looked like on the servers I’ve seen it on:

image

The invalid root CA server circled in red

That’s a not very helpful friendly name on that certificate, so let’s take a look at the detailed properties:

image

Meet “VeriSign Class 3 Public Primary Certification Authority – G5”

Step #2 is also performed in the Certificates MMC snap-in. Navigate to the Third-Party Root Certification Authorities node and find your certificate. Match the attributes above to the certificate below:

image

Root CA certificate hide and seek

Right-click the certificate and select Properties (don’t just open the certificate) to get the following dialog, where you will want to select the option to disable the certificate for all purposes:

image

C’mon…you know you want to

Go back to the server certificate and view the validation chain again. This time, you should see the sweet, sweet sign of victory (if not, close down the MMC and open it up again):

image

Working on the chain gang

It’s a relatively easy process…so where do you need to do it? Great question!

The process I outlined obviously is for Windows servers, so you would think that you can fix this just on the the Exchange CAS roles in your Internet-facing sites. However, you may have additional work to do depending on how you’re publishing Exchange:

  • If you’re using a hardware load balancer with the SSL certificate loaded, you may not have the ability to disable the invalid root CA certificate on the load balancer. You may simply need to remove the invalid chain, re-export the correct chain from your Exchange server, and reinstall the valid root and intermediate CA certificates.
  • If you’re publishing through ISA/TMG, perform the same process on the ISA/TMG servers. You may also want to re-export the correct chain from your Exchange server onto your reverse proxy servers to ensure they have all the intermediate CA certificates loaded locally.

The general rule is that the outermost server device needs to have the valid, complete certificate chain loaded locally to ensure AIA does its job for the various client devices.

Let me know if this helps you out.

Exchange 2010 virtualization storage gotchas

There’s a lot of momentum for Exchange virtualization. At Trace3, we do a lot of work with VMware, so the majority of the customers I work with already have VMware deployed strategically into their production operation model. As a result, we see a lot of Exchange 2010 under VMware. With Exchange 2010 SP1 and lots of customer feedback, the Exchange product team has really stepped up to provide better support for virtual environments as well as more detailed guidance on planning for and deploying Exchange 2007 and 2010 in virtualization.

Last week, I was talking with a co-worker about Exchange’s design requirements in a virtual environment. I casually mentioned the “no file-level storage protocols” restriction for the underlying storage and suddenly, the conversation turned a bit more serious. Many people who deploy VMware create large data stores on their SAN and share them to the ESX cluster via the NFS protocol. There are a lot of advantages to doing it this way, and it’s a very flexible and relatively easy way to deploy VMs. However, it’s not supported for Exchange VMs.

The Heck You Say?

“But Devin,” I can hear some of you say, “what do you mean it’s not supported to run Exchange VMs on NFS-mounted data stores? I deploy all of my virtual machines using VMDKs on NFS-mounted data stores. I have my Exchange servers there. It all works.”

It probably does work. Whether or not it works, though, it’s not a supported configuration, and one thing Masters are trained to hate with a passion is letting people deploy Exchange in a way that gives them no safety net. It is an essential tool in your toolkit to have the benefit of Microsoft product support to walk you through the times when you get into a strange or deep problem.

Let’s take a look at Microsoft’s actual support statements. For Exchange 2010, Microsoft has the following to say in http://technet.microsoft.com/en-us/library/aa996719.aspx under virtualization (emphasis added):

The storage used by the Exchange guest machine for storage of Exchange data (for example, mailbox databases or Hub transport queues) can be virtual storage of a fixed size (for example, fixed virtual hard disks (VHDs) in a Hyper-V environment), SCSI pass-through storage, or Internet SCSI (iSCSI) storage. Pass-through storage is storage that’s configured at the host level and dedicated to one guest machine. All storage used by an Exchange guest machine for storage of Exchange data must be block-level storage because Exchange 2010 doesn’t support the use of network attached storage (NAS) volumes. Also, NAS storage that’s presented to the guest as block-level storage via the hypervisor isn’t supported.

Exchange 2007 has pretty much the same restrictions as shown in the http://technet.microsoft.com/en-us/library/bb738146(EXCHG.80).aspx TechNet topic. What about Exchange 2003? Well, that’s trickier; Exchange 2003 was never officially supported under any virtualization environment other than Microsoft Virtual Server 2005 R2.

The gist of the message is this: it is not supported by Microsoft for Exchange virtual machines to use disk volumes that are on file-level storage such as NFS or CIFS/SMB, if those disk volumes hold Exchange data. I realize this is a huge statement, so let me unpack this a bit. I’m going to assume a VMware environment here, but these statements are equally true for Hyper-V or any other hypervisor supported under the Microsoft SVVP.

While the rest of the discussion will focus on VMware and NFS, all of the points made are equally valid for SMB/CIFS and other virtualization system. (From a performance standpoint, I would not personally want to use SMB for backing virtual data stores; NFS, in my experience, is much better optimized for the kind of large-scale operations that virtualization clusters require. I know Microsoft is making great strides in improving the performance of SMB, but I don’t know if it’s there yet.

It’s Just Microsoft, Right?

So is there any way to design around this? Could I, in theory, deploy Exchange this way and still get support from my virtualization vendor? A lot of people I talk to point to a whitepaper that VMware published in 2009 that showed the relative performance of Exchange 2007 over iSCSI, FC, and NFS. They use this paper as “proof” that Exchange over NFS is supported.

Not so much, at least not with VMware. The original restriction may come from the Exchange product group (other Microsoft workloads are supported in this configuration), but the other vendors certainly know the limitation and honor it in their guidance. Look at VMware’s Exchange 2010 best practices at http://www.vmware.com/files/pdf/Exchange_2010_on_VMware_-_Best_Practices_Guide.pdf on page 13:

It is important to note that there are several different shared-storage options available to ESX (iSCSI, Fibre Channel, NAS, etc.); however, Microsoft does not currently support NFS for the Mailbox Server role (clustered or standalone). For Mailbox servers that belong to a Database Availability Group, only Fibre Channel is currently supported; iSCSI can be used for standalone mailbox servers. To see the most recent list of compatibilities please consult the latest VMware Compatibility Guides.

According to this document, VMware is even slightly more restrictive! If you’re going to use RDMs (this section is talking about RDMs, so don’t take the iSCSI/FC statement as a limit on guest-level volume mounts), VMware is saying that you can’t use iSCSI RDMs, only FC RDMs.

Now, I believe – and there is good evidence to support me – that this guidance as written is actually slightly wrong:

  • The HT queue database is also an ESE database and is subject to the same limitations; this is pretty clear on a thorough read-through of the Exchange 2010 requirements in TechNet. Many people leave the HT queue database on the same volume they install Exchange to, which means that volume also cannot be presented via NFS. If you follow best practices, you move this queue database to a separate volume (which should be an RDM or guest-mounted iSCSI/FC LUN).
  • NetApp, one of the big storage vendors that supports the NFS-mounted VMware data store configuration, only supports Exchange databases mounted via FC/iSCSI LUNs using SnapManager for Exchange (SME) as shown in NetApp TR-3845. Additionally, in the join NetApp-VMware-Cisco performance whitepaper on virtualizing Microsoft workloads, the only configuration tested for Exchange 2010 is FC LUNs (TR-3785).
  • It is my understanding that the product group’s definition of Exchange files doesn’t just extend to ESE files and transaction logs, but to all of the Exchange binaries and associated files. I have not yet been able to find a published source to document this interpretation, but I am working on it.
  • I am not aware of any Microsoft-related restriction about iSCSI + DAG. This VMware Exchange 2010 best practices document (published in 2010) is the only source I’ve seen mention this restriction, and in fact, the latest VMware Microsoft clustering support matrix (published in June 2011) lists no such restriction. Microsoft’s guidelines seem to imply that block storage is block storage is block storage when it comes to “SCSI pass-through storage”). I have queries in to nail this one down because I’ve been asking in various communities for well over a year with no clear resolution other than, “That’s the way VMware is doing it.”

Okay, So Now What?

When I’m designing layouts for customers who are used to deploying Windows VMs via NFS-mounted VMDKs, I have a couple of options. My preferred option, if they’re also using RDMs, is to just have them provision one more RDM for the system drive and avoid NFS entirely for Exchange servers. That way, if my customer does have to call Microsoft support, we don’t have to worry about the issue at all.

However, that’s not always possible. My customer may have strict VM provisioning processes in place, have limited non-NFS storage to provision, or have some other reason why they need to use NFS-based VMDKs. In this case, I have found the following base layout to work well:

Volume Type Notes
C: VMDK or RDM Can be on any type of supported data store. Should be sized to include static page file of size PhysicalRAM + 10 MB.
E: RDM or guest iSCSI/FC iSCSI/FC    All Exchange binaries installed here. Move IIS files here (scripts out on Internet to do this for you). Create an E:\Exchdata directory and use NTFS mount points to mount each of the data volumes the guest will mount.
Data volumes RDM or guest iSCSI/FC Any volume holding mailbox/PF database EDB or logs, or HT queue EDB or logs. Should mount these separately, NTFS mount points recommended. Format these NTFS volumes with 64K block size, not default.

Note that we have several implicit best practices in use here:

  • Static page file, properly sized for a 64-bit operating system with a large amount of physical RAM. Doing this ensures that you have enough virtual memory for the Exchange memory profile AND that you can write a kernel memory crash dump to disk in the event of a blue screen. (If the page file is not sized properly, or is not on C:, the full dump cannot be written to disk.)
  • Exchange binaries not installed on the system drive. This makes restores much easier. Since Exchange uses IIS heavily, I recommend moving the IIS data files (the inetpub and children folders) off of the system drive and onto the Exchange volume. This helps reduce the rate of change on the system drive and offers other benefits such as making it easier to properly configure anti-virus exclusions.
  • The use of NTFS mount points (which mount the volume to a directory) instead of separate drive letters. For large DAGs, you can easily have a large number of volumes per MB role, making the use of drive letters a limitation on scalability. NTFS mount points work just like Unix mount points and work terribly well – they’ve been supported since Exchange 2003 and recommended since the late Exchange 2003 era for larger clusters. In Exchange 2007 and 2010 continuous replication environments (CCR, SCR, DAG), all copies must have the same pathnames.
  • Using NTFS 64K block allocations for any volumes that hold ESE databases. While not technically necessary for log partitions, doing so does not hurt performance.

So Why Is This Even A Problem?

This is the money question, isn’t it? Windows itself is supported under this configuration. Even SQL Server is. Why not Exchange?

At heart, it comes down to this: the Exchange ESE database engine is a very finely-tuned piece of software, honed for over 15 years. During that time, with only one exception (the Windows Storage Server 2003 Feature Pack 1, which allowed storage solutions running WSS 2003 + FP1 to host Exchange database files over NAS protocols), Exchange has never supported putting Exchange database files over file-level storage. I’m not enough of an expert on ESE to whip up a true detailed answer, but here is what I understand about it.

Unlike SQL Server, ESE is not a general purpose database engine. SQL is optimized to run relational databases of all types. The Exchange flavor of ESE is optimized for just one type of data: Exchange. As a result, ESE has far more intimate knowledge about the data than any SQL Server instance can. ESE provides a lot of performance boosts for I/O hungry Exchange databases and it can do so precisely because it can make certain assumptions. One of those assumptions is that it’s talking to block-level storage.

When a host process commits writes to storage, there’s a very real difference in the semantics of the write operation between block-level protocols and file-level protocols. Exchange, in particular, depends dramatically on precise control over block-level writes – which file protocols like NFS and SMB can mask. The cases under which this can cause data corruption for Exchange are admittedly corner cases, but they do exist and they can cause impressive damage.

Cleaning Up

What should we do about it if we have an Exchange deployment that is in violation of these support guidelines?

Ideally, we fix it. Microsoft’s support stance is very clear on this point, and in the unlikely event that data loss occurs in this configuration, Microsoft support is going to point at the virtualization/storage vendors and say, “Get them to fix it.” I am not personally aware of any cases of a configuration like this causing data loss or corruption, but I am not the Exchange Product Group – they get access to an amazing amount of data.

At the very least, you need to understand and document that you are in an unsupported configuration so that you can make appropriate plans to get into support as you roll out new servers or upgrade to future versions of Exchange. This is where getting a good Exchange consultant to do an Exchange health check can help you get what you need and provide the support you need with your management – we will document this in black and white and help provide the outside validation you might need to get things put right.

One request for the commenters: if all you’re going to do is say, “Well we run this way and have no problems,” don’t bother. I know and stipulate that there are many environments out there running in violating of this support boundary that have not (yet) run into issues. I’ve never said it won’t work. There are a lot of things we can do, but that doesn’t mean we should do them. At the same time, at the end of the day – if you know the issues and potential risks, you have to make the design decision that’s right for your organization. Just make sure it’s an informed (and documented, and signed-off!) decision.

Devin’s Load Balancer for Exchange 2010

Overview

One of the biggest differences I’m seeing when deploying Exchange 2010 compared to previous versions is that for just about all of my customers, load balancing is becoming a critical part of the process. In Exchange 2003 FE/BE, load balancing was a luxury unheard of for all but the largest organizations with the deepest pockets. Only a handful of outfits offered load balancing products, and they were expensive. For Exchange 2007 and the dedicated CAS role, it started becoming more common.

For Exchange 2003 and 2007, you could get all the same benefits of load balancing (as far as Exchange was concerned) by deploying an ISA server or ISA server cluster using Windows Network Load Balancing (WNLB). ISA included the concept of a “web farm” so it would round-robin incoming HTTP connections to your available FE servers (and Exchange 2007 CAS servers). Generally, your internal clients would directly talk to their mailbox servers, so this worked well. Hardware load balancers were typically used as a replacement for publishing with an ISA reverse proxy (and more rarely to load balance the ISA array instead of WNLB). Load balancers could perform SSL offloading, pre-authentication, and many of the same tasks people were formerly using ISA for. Some small shops deployed WNLB for Exchange 2003 FEs and Exchange 2007 CAS roles.

In Exchange 2010, everything changes. Outlook RPC connections now go to the CAS servers in the site, not the MB server that hosts the active copy of the database. Mailbox databases now have an affiliation with either a specific CAS server or a site-specific RPC client access array, which you can see using the –RpcClientAccessServer parameter of the Get-MailboxDatabase cmdlet. If you have two or more servers, I recommend you set up the RPC client access array as part of the initial deployment and get some sort of load balancer in place.

Load Balancing Options

At Trace3, we’re an F5 reseller, and F5 is one of the few load balancer companies out there that has really made an effort to understand and optimize Exchange 2010 deployments. However, I’m not on the sales side; I have customers using a variety of load balancing solutions for their Exchange deployments. At the end of the day, we want the customer to do what’s right for them. For some customers, that’s an F5. Others require a different solution. In those cases, we have to get creative – sometimes they don’t have budget, sometimes the networking team has their own plans, and on some rare occasions, the plans we made going in turned out not to be a good fit after all and now we have to come up with something on the fly.

If you’re not in a position to use a high-end hardware load balancer like an F5 BIG-IP or a Cisco ACE solution, and can’t look at some of the lower-cost (and correspondingly lower-feature) solutions that are now on the market, there are few alternatives:

  • WNLB. To be honest, I have attempted to use this in several environments now and even when I spent time going over the pros and cons, it failed to meet expectations. If you’re virtualizing Exchange (like many of my customers) and are trying to avoid single points of failure, WNLB is so clearly not the way to go. I no longer recommend this to my customers.
  • DNS round robin. This method at least has the advantage of in theory driving traffic to all of the CAS instances. However, in practice it gets in the way of quickly resolving problems when they come up. It’s better than nothing, but not by much.
  • DAG cluster IP. Some clever people came up with this option for instances where you are deploying multi-role servers with MB+HT+CAS on all servers and configuring them in a DAG. DAG = cluster, these smart people think, and clusters have a cluster IP address. Why can’t we just use that as the IP address of the RPC client access array? Sure enough, this works, but it’s not tested or supported by Microsoft and it isn’t a perfect solution. It’s not load balancing at all; the server holding the cluster IP address gets all the CAS traffic. Server sizing is important!

The fact of the matter is, there are no great alternatives if you’re not going to use hardware load balancing. You’re going to have to compromise something.

Introducing Devin’s Load Balancer

For many of my customers, we end up looking something like this:

  • The CAS/HT roles are co-located on one set of servers, while MB (and the DAG) is on another. This rules out the DAG cluster IP option.
  • They don’t want users to complain excessively when something goes wrong with one of the CAS/HT servers. This rules out DNS round robin.
  • They don’t have the budget for a hardware solution yet, or one is already in the works but not ready because of schedule. They need a temporary, low-impact solution. This effectively rules out WNLB.

I’ve come up with a quick and dirty fix I call Devin’s Load Balancer or, as I commonly call it, the DLB. It looks like this:

  1. Pick one CAS server that can handle all the traffic for the site. This is our target server.
  2. Pick an IP address for the RPC client access array for the site. Create the DNS A record for the RPC client access array FQDN, pointing to the IP address.
  3. Create the RPC client access array in EMS, setting the name, FQDN, and site.
  4. On the main network interface of the target server, add the IP address. If this IP address is on the same subnet as the main IP address, there is no need to create a secondary interface! Just add it as a secondary IP address/subnet mask.
  5. Make sure the appropriate mailbox databases are associated with the RPC client access array.
  6. Optionally, point the internal HTTP load balance array DNS A record to this IP address as well (or publish this IP address using ISA).

You may have noticed that this sends all traffic to the target server; it doesn’t really load balance. DLB also stands for Doesn’t Load Balance!

This configuration, despite its flaws, gives me what I believe are several important benefits:

  • It’s extremely easy to switchover/failover. If something happens to my target server, I simply add the RPC client access array IP address as a secondary IP address to my next CAS instance. There are no DNS cache entries to wait to expire. There are are no switch configurations to modify. There are no DNS records I have to update. If this is a planned switchover, client get disrupted but can immediately connect. I can make the update as soon as I get warning that something happened and my clients can reconnect without any further action on their part.
  • It isolates what I do with the other CAS instances. Windows and Exchange no longer have any clue they’re in a load balanced pseudo-configuration. With WNLB, if I make any changes to the LB cluster (like add or remove a member), all connections to the cluster IP addresses are dropped!
  • It makes it very easy to upgrade to a true load balancing solution. I set the true solution up in parallel with an alternate, temporary IP address. I use local HOSTS file entries on my test machines while I’m getting everything tested and validated. And then I simply take the RPC client access array IP address off the target server and put it on the load balancer. Existing connections are dropped, but new ones immediately connect with no timeouts – and now we’re really load balancing.

Note that you do not need the CAS SSL certificate to contain the FQDN of the RPC client access array as a SAN entry. RPC doesn’t use SSL for encryption (it’s not based on HTTP).

Even in a deployment where the customer is putting all roles into single-server configuration, if there’s any thought at all that they might want to expand to an HA configuration in the future, I now am in the habit of configuring this. The RPC client access array is now configured and somewhat isolated from the CAS configuration, so now my future upgrades are easier and less disruptive.

Moving to Exchange Server 2010 Service Pack 1

Microsoft recently announced that Service Pack 1 (SP1) for Exchange Server 2010 had been released to web, prompting an immediate upgrade rush for all of us Exchange professionals. Most of us maintain at least one home/personal lab environment, the better to pre-break things before setting foot on a customer site. Before you go charging out to do this for production (especially if you’re one of my customers, or don’t want to run the risk of suddenly becoming one of my customers), take a few minutes to learn about some of the current issues with SP1.

Easy Installation and Upgrade Slipstreaming

One thing that I love about Exchange service packs is that from Exchange 2007 on, they’re full installations in their own right. Ready to deploy a brand new Exchange 2010 SP1 server? Just run setup from the SP1 binaries – no more fiddling around with the original binaries, then applying your service packs. Of course, the Update Rollups now take the place of that, but there’s a mechanism to slipstream them into the installer (and here is the Exchange 2007 version of this article).

Note: If you do make use of the slipstream capabilities, remember that Update Rollups are both version-dependent (tied to the particular RTM/SP release level) and are cumulative. SP1 UR4 is not the same thing as RTM UR4! However, RTM UR4 will include RTM UR3, RTM UR2, and RTM UR1…just as SP1 UR4 will contain SP1 UR3, SP1 UR2, and SP1 UR1.

The articles I linked to say not to slipstream the Update Rollups with a service pack, and I’ve heard some confusion about what this means. It’s simple: you can use the Updates folder mechanism to slipstream the Update Rollups when you are performing a clean install. You cannot use the slipstream mechanism when you are applying a service pack to an existing Exchange installation. In the latter situation, apply the service pack, then the latest Update Rollup.

It’s too early for any Update Rollups for Exchange 2010 SP1 to exist at the time of writing, but if there were (for the sake of illustration, let’s say that SP1 UR X just came out), consider these two scenarios:

  • You have an existing Exchange 2010 RTM UR4 environment and want to upgrade directly to SP1 UR1. You would do this in two steps on each machine: run the SP1 installer, then run the latest SP1 UR X installer.
  • You now want to add a new Exchange 2010 server into your environment and want it to be at the same patch level. You could perform the installation in a single step from the SP1 binaries by making sure the latest SP1 UR X installer was in the Updates folder.

If these scenarios seem overly complicated, just remember back to the Exchange 2003 days…and before.

Third Party Applications

This might surprise you, but in all of the current Exchange 2010 projects I’m working on, I’ve not even raised the question of upgrading to SP1 yet. Why would I not do that? Simple – all of these environments have dependencies on third-party software that is not yet certified for Exchange 2010 SP1. In some cases, the software has barely just been certified for Exchange 2010 RTM! If the customer brings it up, I always encourage them to start examining SP1 in the lab, but for most production environments, supportability is a key requirement.

Make sure you’re not going to break any applications you care about before you go applying service packs! Exchange service packs always make changes – some easy to see, some harder to spot. You may need to upgrade your third-party applications, or you may simply need to make configuration changes ahead of time – but if you blindly apply service packs, you’ll find these things out the hard way. If you have a critical issue or lack of functionality that the Exchange 2010 SP1 will address, get it tested in your lab and make sure things will work.

Key applications I encourage my customers to test include:

Applications that use SMTP submission are typically pretty safe, and there are other applications that you might be okay living without if something does break. Figure out what you can live with, test them (or wait for certifications), and go from there.

Complications and Gotchas

Unfortunately, not every service pack goes smoothly. For Exchange 2010 SP1, one of the big gotchas that early adopters are giving strong feedback about is the number of hotfixes you must download and apply to Windows and the .NET Framework before applying SP1 (a variable number, depending on which base OS your Exchange 2010 server is running).

Having to install hotfixes wouldn’t be that bad if the installer told you, “Hey, click here and here and here to download and install the missing hotfixes.” Exchange has historically not done that (citing boundaries between Microsoft product groups) even though other Microsoft applications don’t seem to be quite as hobbled. However, this instance of (lack of) integration is particularly egregious because of two factors.

Factor #1: hotfix naming conventions. Back in the days of Windows 2000, you knew whether a hotfix was meant for your system, because whether you were running Workstation or Server, it was Windows 2000. Windows XP and Windows 2003 broke that naming link between desktop and server operating systems, often confusingly so once 64-bit versions of each were introduced (32-bit XP and 32-bit 2003 had their own patch versions, but 64-bit XP applied 64-bit 2003 hotfixes).

Then we got a few more twists to deal with. For example, did you know that Windows Vista and Windows Server 2008 are the same codebase under the hood? Or that Windows 7 and Windows Server 2008 R2, likewise, are BFFs? It’s true. Likewise, the logic behind the naming of Windows Server 2003 R2 and Windows Server 2008 R2 were very different; Windows Server 2003 R2 was basically Windows Server 2003 with a SP and few additional components, while Windows Server 2008 R2 has some substantially different code under the hood than Windows Server 2008 with SP. (I would guess that Windows Server 2008 R2 got the R2 moniker to capitalize on Windows 2008’s success, while Windows 7 got a new name to differentiate itself from the perceived train wreck that Vista had become, but that’s speculation on my part.)

At any rate, figuring out which hotfixes you need – and which versions of those hotfixes – is less than easy. Just remember that you’re always downloading the 64-bit patch, and that Windows 2008=Vista while Windows 2008 R2=Windows 7 and you should be fine.

Factor #2: hotfix release channels. None of these hotfixes show up under Windows Update. There’s no easy installer or tool to run that gets them for you. In fact, at least two of the hotfixes must be obtained directly from Microsoft Customer Support Services. All of these hotfixes include scary legal boilerplate about not being fully regression tested and thereby not supported unless you were directly told to install them by CSS. This has caused quite a bit of angst out in the Exchange community, enough so that various people are collecting the various hotfixes and making them available off their own websites in one easy package to download[1].

I know that these people mean well and are trying to save others from a frustrating experience, but in this case, the help offered is a bad idea. That same hotfix boilerplate means that everyone who downloads those hotfixes agree not to redistribute those hotfixes. There’s no exception for good intentions. If you think this is bogus, let me give you two things to think about:

  • You need to be able to verify that your hotfixes are legitimate and haven’t been tampered with. Do you really want to trust production mission-critical systems to hotfixes you scrounged from some random Exchange pro you only know through blog postings? Even if the pro is trustworthy, is their web site? Quite frankly, I trust Microsoft’s web security team to prevent, detect, and mitigate hotfix-affecting intrusions far more quickly and efficiently than some random Exchange professional’s web host. I’m not disparaging any of my colleagues out there, but let’s face it – we have a lot more things to stay focused on. Few of us (if any) have the time and resources the Microsoft security guys do.
  • Hotfixes in bundles grow stale. When you link to a KB article or Microsoft Download offering to get a hotfix, you’re getting the most recent version of that hotfix. Yes, hotfixes may be updated behind the scenes as issues are uncovered and testing results come in. In the case of the direct-from-CSS hotfixes, you can get them for free through a relatively simple process. As part of that process, Microsoft collects your contact info so they can alert you if any issues later come up with the hotfix that may affect you. Downloading a stale hotfix from a random bundle increases the chances of getting an old hotfix version that may cause issues in your environment, costing you a support incident. How many of these people are going to update their bundles as new hotfix versions become available? How quickly will they do it – and how will you know?

The Exchange product team has gotten an overwhelming amount of feedback on this issue, and they’ve responded on their blog. Not only do they give you a handy table rounding up links to get the hotfixes, they also collect a number of other potential gotchas and advice to learn from from before beginning your SP1 deployment. Go check it out, then start deploying SP1 in your lab.

Good luck, and have fun! SP1 includes some killer new functionality, so take a look and enjoy!

[1] If you’re about to deploy a number of servers in a short period of time, of course you should cache these downloaded hotfixes for your team’s own use. Just make sure that that you check back occasionally for updated versions of the hotfixes. The rule of thumb I’d use is about a week – if I’m hitting my own hotfix cache and it’s older than a week, it’s worth a couple of minutes to make sure it’s still current.

Manually creating a DAG FSW for Exchange 2010

I just had a comment from Chris on my Busting the Exchange Trusted Subsystem Myth post that boiled down to asking what you do when you have to create the FSW for an Exchange 2010 DAG manually?

In order for this to be true, you have to have the following conditions:

  1. You have no other Exchange 2010 servers in the AD site. This implies that at least one of your DAG nodes is multi-role — remember that you need to have a CAS role and an HT role in the same site as your MB roles, preferably two or more of each for redundancy and load. If you have another Exchange 2010 server, then it’s already got the correct permissions — let Exchange manage the FSW automatically.
  2. If the site in question is part of a DAG that stretches sites, there are more DAG nodes in this site than in the second site. If you’re trying to place the FSW in the site with fewer members, you’re asking for trouble[1].
  3. You have no other Windows 2003 or 2008 servers in the site that you consider suitable for Exchange’s automatic FSW provisioning[2]. By this, I mean you’re not willing to the the Exchange Trusted Subsystem security group to the server’s local Administrators group so that Exchange can create, manage, and repair the FSW on its own. If your only other server in the site is a DC, I can understand not wanting to add the group to the Domain Admins group.

If that’s the case, and you’re dead set on doing it this way, you will have to manually create the FSW yourself. A FSW consists of two pieces: the directory, and the file share. The process for doing this is not documented anywhere on TechNet that I could find with a quick search, but happily, one Rune Bakkens blogs the following process:

To pre-create the FSW share you need the following:
– Create a folder etc. D:\FilesWitness\DAGNAME
– Give the owner permission to Exchange Trusted Subsystem
– Give the Exchange Trusted Subsystem Full Control (NTFS)
– Share the folder with the following DAGNAME.FQDN (If you try a different share name,
it won’t work. This is somehow required)
– Give the DAGNAME$ computeraccount Full Control (Share)

When you’ve done this, you can run the set-databaseavailabilitygroup -witnessserver CLUSTERSERVER – witnessdirectory D:\Filewitness\DAGNAME

You’ll get the following warning message:

WARNING: Specified witness server Cluster.fqdn is not an Exchange server, or part of the Exchange Servers security group.
WARNING: Insufficient permission to access file shares on witness server Cluster.fqdn. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the set-databaseavailabilitygroup cmdlet to try the operation again. Error: Access is denied

This is expected, since the cmdlet tries to create the folder and share, but don’t have the permissions to do this.

When this is done, the FSW should be configured correct. To verify this, the following files should be created:

- VerifyShareWriteAccess
– Witness

Just for the record, I have not tested this process yet. However, I’ve had to do some recent FSW troubleshooting lately and this matches with what I’ve seen for naming conventions and permissions, so I’m fairly confident this should get you most of the way there. Thank you, Rune!

Don’t worry, I haven’t forgotten the next installment of my Exchange 2010 storage series. It’s coming, honest!

[1] Consider the following two-site DAG scenarios:

  • If there’s an odd number of MB nodes, Exchange won’t use the FSW.
  • An even number (n) of nodes in each site. The FSW is necessary for there to even be a quorum (you have 2n+1 nodes so a simple majority is n+1). If you lose the FSW and one other node — no matter where that node is — you’ll lose quorum. If you lose the link between sites, you lose quorum no matter where the FSW is.
  • A number (n) nodes in site A, with at least one fewer nodes (m) in site B. If n+m is odd, you have an odd number of nodes — our first case. Even if m is only 1 fewer than n, putting the FSW in site B is meaningless — if you lose site A, B will never have quorum (in this case, m+1 = n, and n is only half — one less than quorum).

I am confident in this case that if I’ve stuffed up the math here, someone will come along to correct me. I’m pretty sure I’m right, though, and now I’ll have to write up another post to show why. Yay for you!

[2] You do have at least one other Windows server in that site, though, right — like your DC? Exchange doesn’t like not having a DC in the local site — and that DC should also be a GC.

The Disk’s The Thing! Exchange 2010 Storage Essays, part 2

Greetings, readers! When I first posted From Whence Redundancy? (part 1 of this series of essays on Exchange 2010 storage) I’d intended to follow up with other posts a bit faster than I have been. So much for intentions; let us carry on.

In part 1, I began the process of talking about how I think the new Exchange 2010 storage options will play out in live Exchange deployments over the next several years. The first essay in this series discussed what is I believe the fundamental question at the heart an Exchange 2010 storage design: at what level will you ensure the redundancy of your Exchange mailbox databases? The traditional approach has used RAID at the disk level, but Exchange 2010 DAGs allow you to deploy mailbox databases in JBOD configurations. While I firmly believe that’s the central question, answering it requires us to dig under the hood of storage.

With Exchange 2010, Microsoft specifically designed Exchange mailbox servers to be capable of using the lowest common denominator of server storage: a directly attached storage (DAS) array of 7200 RPM SATA disks in a Just a Box of Disks (JBOD) configuration (what I call DJS). Understanding why they’ve made this shift requires us to understand more about the disk drive technology. In this essay, part 2 of this series, let’s talk about disk technology and find out how Fibre Channel (FC), Serially Attached SCSI (SAS), and Serial Advanced Technology Attachment (SATA) disk drives are the same – and more importantly, what slight differences they have and what that means for your Exchange systems.

Exchange Storage SATA vs SAS

So here’s the first dirty little secret: for the most part, all disks are the same. Regardless of what type of bus they use, what form factor they are, what capacity they are, and what speed they rotate at, all modern disks use the same construction and principles:

  • They all have one or more thin rotating platters coated with magnetic media; the exact number varies by form factor and capacity. Platters look like mini CD-ROM disks, but unlike CDs, platters are typically double-sided. Platters have a rotational speed measured in revolutions per minute (RPMs).
  • Each side of a platter has an associated read-write head. These heads are on a single-track arm that moves in toward the hub of the platter or out towards the rim. The heads do not touch the platter, but float very close to the surface. It takes a measurable fraction of a second for the head to relocate from one position to another; this is called its seek time.
  • The circle described by the head’s position on the platter is called a track. In a multi-platter disk, the heads move in synchronization (there’s no independent tracking per platter or side). As a result, each head is on the same track at the same time, describing a cylinder.
  • Each drive unit has embedded electronics that implement the bus protocol, control the rotational speed of the platters, and translate I/O requests into the appropriate commands to the heads. Even though there are different flavors, they all perform the same basic functions.

If you would like a more in-depth primer on how disks work, I recommend starting with this article. I’ll wait for you.

Good? Great! So that’s how all drives are the same. It’s time to dig into the differences. They’re relatively small, but small differences have a way of piling up. Take a look at Table 1 which summarizes the differences between various FC, SATA, and SAS disks, compared with legacy PATA 133 (commonly but mistakenly referred to as IDE) and SCSI Ultra 320 disks:

Table 1: Disk parameter differences by disk bus type

Type Max wire bandwidth(Mbit/s) Max data transfer(MB/s)
PATA 133 1,064 133.5
SCSI Ultra 320 2,560 320
SATA-I 1,500 150
SATA-II 3,000 300
SATA 6 Gb/s 6,000 600
SAS 150 1,500 150
SAS 300 3,000 300
FC (copper) 4,000 400
FC (optic) 10,520 2,000

 

As of this writing, the most common drive types you’ll see for servers are SATA-II, SAS 300, and FC over copper. Note that while SCSI Ultra 320 drives in theory have a maximum data transfer higher than either SATA-II or SAS 300, in reality that bandwidth is shared among all the devices connected to the SCSI bus; both SATA and SAS have a one-to-one connection between disk and controller, removing contention. Also remember that SATA is only a half-duplex protocol, while SAS is a full-duplex protocol. SAS and FC disks use the full SCSI command set to allow better performance when multiple I/O requests are queued for the drive, whereas SATA uses the ATA command set. Both SAS and SATA implement tagged queuing, although they use two different standards (each of which has its pros and cons).

The second big difference is the average access time of the drive, which is the sum of multiple factors:

  • The average seek time of the heads. The actuator motors that move the heads from track to track are largely the same from drive to drive and thus the time contributed to the drive’s average seek time by just the head movements is roughly the same from drive to drive. What varies is the length of the head move; is it moving to a neighboring track, or is it moving across the entire surface? We can average out small track changes with large track changes to come up with idealized numbers.
  • The average latency of the platter. How fast the platters are spinning determines how quickly a given sector containing the data to be read (or where new data will be written) will move into position under the head once it’s in the proper track. This is a simple calculation based on the RPM of the platter and the observed average drive latency. We can assume that a given sector will move into position, on average, in no more than half a rotation. This gives us 30 seconds out of each minute of rotation, or 30,000 ms, into which we can divide the drive’s actual rotation.
  • The overhead caused by the various electronics and queuing mechanisms of the drive electronics, including any power saving measures such as reducing the spin rate of the drive platters. Although electricity is pretty fast and on-board electronics are relatively small circuits, there may be other factors (depending on the drive type) that may introduce delays into the process of fulfilling the I/O request received from the host server.

What has the biggest impact is how fast the platter is spinning, as shown in Table 2:

Table 2: Average latency caused by rotation speed

Platter RPM Average latency in ms
7,200 4.17
10,000 3
12,000 2.5
15,000 2

 

(As an exercise, do the same math on the disk speeds for the average laptop drives. This helps explain why laptop drives are so much slower than even low-end 7,200 RPM SATA desktop drives.)

Rather than painfully take you through the result of all of these tables and calculations step by step, I’m simply going to refer you to work that’s already been done. Once we know the various averages and performance metrics, we can figure out how many I/O operations per second (IOPS) a given drive can sustain on average, according to the type, RPMs, and nature of the I/O (sequential or random). Since Microsoft has already done that work for us as part of the Exchange 2010 Mailbox Role Calculator (version 6.3 as of this writing, I’m going to simply use the values there. Let’s take a look at how all this plays out in Table 3 by selecting some representative values.

Table 3: Drive IOPS by type and RPM

Size Type RPM Average Random IOPS
3.5” SATA 5,400 50
2.5” SATA 5,400 55
3.5” SAS 5,400 52.5
3.5” SAS 5,900 52.5
3.5” SATA 7,200 55
2.5” SATA 7,200 60
3.5” SAS 7,200 57.5
2.5” SAS 7,200 62.5
3.5” FC/SCSI/SAS 10,000 130
2.5” SAS 10,000 165
3.5” FC/SCSI/SAS 15,000 180
2.5” SAS 15,000 230

 

There are three things to note about Table 3.

  1. These numbers come from Microsoft’s Exchange 2010 Mailbox Sizing Calculator and are validated across vendors through extensive testing in an Exchange environment. While there may be minor variances between drive model and manufacturers and these number may seem pessimistic according to calculated IOPS number published for individual drives, these are good figures to use in the real world. Using calculated IOPS numbers can lead both to a range of figures, depending on the specific drive model and manufacturer, as well as to overestimating the amount of IOPS the drive will actually provide to Exchange.
  2. For the most part, SAS and FC are indistinguishable from the IOPs point of view. Regardless of the difference between the electrical interfaces, the drive mechanisms and I/O behaviors are comparable.
  3. Sequential IOPS are not listed; they will be quite a bit higher than the random IOPS (that same 7,200RPM SATA drive can provide 300+ IOPS for sequential operations). The reason is simple; although a lot of Exchange 2010 I/O has been converted from random to sequential, there’s still some random I/O going on. That’s going to be the limiting factor.

The IOPS listed are per-drive IOPS. When you’re measuring your drive system, remember that the various RAID configurations have their own IOPS overhead factor that will consume a certain number

There are of course some other factors that we need to consider, such as form factor and storage capacity. We can address these according to some generalizations:

  • Since SAS and FC tend to have the same performance characteristics, the storage enclosure tends to differentiate between which technology is used. SAS enclosures can often be used for SATA drives as well, giving more flexibility to the operator. SAN vendors are increasingly offering SAS/SATA disk shelves for their systems because paying the FC toll can be a deal-breaker for new storage systems.
  • SATA disks tend to have a larger storage capacity than SAS or FC disks. There are reasons for this, but the easiest one to understand is that SAS, being traditionally a consumer technology, has a lower duty cycle and therefore lower quality control specifications that must be met.
  • SATA disks tend to be offered with lower RPMs than SAS and FC disks. Again, we can acknowledge that quality control plays a part here – the faster a platter spins, the more stringently the drive components need to meet their specifications for a longer period of time.
  • 2.5” drives tend to have lower capacity than their 3.5” counterparts. This makes sense – they have smaller platters (and may have fewer platters in the drive).
  • 2.5” drives tend to use less power and generate less heat than equivalent 3.5” drives. This too makes sense – the smaller platters have less mass, requiring less energy to sustain rotation.
  • 2.5” drives tend to permit a higher drive density in a given storage chassis while using only fractionally more power. Again, this makes sense based on the previous two points; I can physically fit more drives into a given space, sometimes dramatically so.

Let’s look at an example. A Supermicro SC826 chassis holds 12 3.5” drives with a minimum of 800W power while the equivalent Supermicro SC216 chassis holds 24 2.5” drives with a minimum of 900W of power in the same 2Us of rack space. Doubling the number of drives makes up for the capacity difference between the 2.5” and 3.5” drives, provides twice as many spindles and allows a greater aggregate IOPS for the array, and only requires 12.5% more power.

The careful reader has noted that I’ve had very little to say about capacity in this essay, other than the observation above that SATA drives tend to have larger capacities, and that 3.5” drives tend to be larger than 2.5” drives. From what I’ve seen in the field, the majority of shops are just now looking at 2.5” drive shelves, so it’s safe to assume 3.5” is the norm. As a result, the 3.5” 7,200 RPM SATA drive represents the lowest common denominator for server storage, and that’s why the Exchange product team chose that drive as the performance bar for DJS configurations.

Exchange has been limited by performance (IOPS) requirements for most of its lifetime; by going after DJS, the product team has been able to take advantage of the fact that the capacity of these drives is the first to grow. This is why I think that Microsoft is betting that you’re going to want to simplify your deployment, aim for big, cheap, slow disks, and let Exchange DAGs do the work of replicating your data.

Now that we’ve talked about RAID vs. JBOD and SATA vs. SAS/FC, we’ll need to examine the final topic: SAN vs. DAS. Look for that discussion in Part 3, which will be forthcoming.

More Exchange blogging with Trace3!

I just wanted to drop a quick note to let you all know that I’ll be cross-posting all of my Exchange related material both here and at the Trace3 blog. The Trace3 blog is a multi-author blog, so you’ll get not only all my Exchange-related content, but you’ll get a variety of other interesting discussions from a number of my co-workers.

To kick it off, I’ve updated my From Whence Redundancy? Exchange 2010 Storage Essays, Part 1 post with some new material on database reseed times and reposted it there in its entirety. Don’t worry, I’ve also updated it here.

What Exchange 2010 on Windows Datacenter Means

Exchange Server has historically come in two flavors for many versions – Standard Edition and Enterprise Edition. The main difference this license change made for you was the maximum number of supported mailbox databases as shown in Table 1:

Version Standard Edition Enterprise Edition
Exchange 2003 1 (75GB max) 20
Exchange 2007 5 50
Exchange 2010 5 100

Table 1: Maximum databases per Exchange editions

However, the Exchange Server edition is not directly tied to the Windows Server edition:

  • For Exchange 2003 failover cluster mailbox servers, Exchange 2007 SCC/CCR environments [1], and  Exchange 2010 DAG environments, you need Windows Server Enterprise Edition in order to get the MSCS cluster component framework.
  • For Exchange 2003 servers running purely as bridgeheads or front-end servers, or Exchange 2007/2010 HT, CAS, ET, and UM servers, you only need Windows Server Standard Edition.

I’ve seen some discussion around the fact that Exchange 2010 will install on Windows Server 2008 Datacenter Edition and Windows Server 2008 R2 Datacenter Edition, even though it’s not supported there and is not listed in the Operating System requirements section of the TechNet documentation.

HOWEVER…if we look at the Prerequisites for Exchange 2010 Server section of the Exchange Server 2010 Licensing site, we now see that Datacenter edition is, in fact listed as shown in Figure 1:

Exchange 2010 server license comparison

Figure 1: Exchange 2010 server license comparison

This is pretty cool, and the appropriate TechNet documentation is in the process of being updated to reflect this. What this means is that you can deploy Exchange 2010 on Windows Server Datacenter Edition; the differences between editions of Windows Server 2008 R2 are found here.[2] If you take a quick scan through the various feature comparison charts in the sidebar, you might wonder why anyone would want to install Exchange 2010 on Windows Server Datacenter Edition; it’s more costly and seems to provide the same benefits. However, take a look at the technical specifications comparison; this is, I believe, the meat of the matter:

  • Both editions give you a maximum of 2 TB – more than you can realistically throw at Exchange 2010.
  • Enterprise Edition gives you support for a maximum eight (8) x64 CPU sockets, while Datacenter Edition gives you sixty-four (64). With quad-core CPUs, this means a total of 32 cores under Enterprise vs. 256 cores under Datacenter.
  • With the appropriate hardware, you can hot-add memory in Enterprise Edition. However, you can’t perform a hot-replace, nor can you hot-add or hot-replace CPUs under Enterprise. With Datacenter, you can hot-add and hot-remove both memory and CPUs.

These seem to be compelling in many scenarios at first glance, unless you’re familiar with the recommended maximum configurations for Exchange 2010 server sizing. IIRC, the maximum CPUs that are recommended for most Exchange 2010 server configurations (including multirole servers) would be 24 cores – which fits into the 8 socket limitation of Enterprise Edition while using quad core CPUs.

With both Intel and AMD now offering hexa-core (6 core) CPUs, you can move up to 48 cores in Enterprise Edition. This is more than enough for any practical deployment of Exchange Server 2010 I can think of at this time, unless future service packs drastically change the CPU performance factors. Both Enterprise and Datacenter give you a ceiling of 2TB of RAM, which is far greater than required by even the most aggressively gigantic mailbox load I’d want to place on a single server. I’m having a difficult time seeing how anyone could realistically build out an Exchange 2010 server that goes beyond the performance and scalability limits of Enterprise Edition in any meaningful way.

In fact, I can think of only three reasons someone would want to run Exchange 2010 on Windows Server Datacenter Edition:

  • You have spare Datacenter Edition licenses, aren’t going to use them, and don’t want to buy more Enterprise Edition licenses. This must be a tough place to be in, but it can happen under certain scenarios.
  • You have a very high server availability requirements and require the hot-add/hot-replace capabilities. This will get costly – the server hardware that supports this isn’t cheap – but if you need it, you need it.
  • You’re already running a big beefy box with Datacenter and virtualization[3]. The box has spare capacity, so you want to make use of it.

The first two make sense. The last one, though, I’d be somewhat leery of doing. Seriously, think about this – I’m spending money on monstrous hardware with awesome fault tolerance capabilities, I’ve forked over for an OS license[4] that gives me the right to unlimited virtual machines, and now I’m going to clutter up my disaster recovery operations by mixing Exchange and other applications (including virtualization) in the same host OS instance? That may be great for a lab environment, but I’d have a long conversation with any customer who wanted to do this under production. Seriously, just spin up a new VM, use Windows Server Enterprise Edition, and go to town. The loss of hardware configuration flexibility I get from going virtual is less than I gain by compartmentalizing my Exchange server to its own machine, along with the ability to move that virtual machine to any virtualization host I have.

So, there you have it: Exchange 2010 can now be run on Windows Server Datacenter Edition, which means yay! for options. But in the end, I don’t expect this to make a difference for any of the deployments I’m like to be working on. This is a great move for a small handful of customers who really need this.

[1] MSCS is not required for Exchange 2007 SCR, although manual target activation can be easier in some scenarios if your target is configured as a single passive node cluster.

[2] From what I can tell, the same specs seem to be valid for Windows Server 2008, with the caveat that Windows Server 2008 R2 doesn’t offer a 32-bit version so the chart doesn’t give that information. However, since Exchange 2010 is x64 only, this is a moot point.

[3] This is often an attractive option, since you can hosted an unlimited number of Windows Server virtual machines without having to buy further Windows Server licenses for them.

[4] Remember that Datacenter is not licensed at a flat cost per server like Enterprise is; it’s licensed per socket. The beefier the machine you run it on, the more you pay.

From Whence Redundancy? Exchange 2010 Storage Essays, part 1

Updated 4/13 with improved reseed time data provided by item #4 in the Top 10 Exchange Storage Myths blog post from the Exchange team.

Over the next couple of months, I’d like to slowly sketch out some of the thoughts and impressions that I’ve been gathering about Exchange 2010 storage over the last year or so and combine them with the specific insights that I’m gaining at my new job. In this inaugural post, I want to tackle what I have come to view as the fundamental question that will drive the heart of your Exchange 2010 storage strategy: will you use a RAID configuration or will you use a JBOD configuration?

In the interests of full disclosure, the company I work for now is a strong NetApp reseller, so of course my work environment is conducive to designing Exchange in ways that make it easy to sell the strengths of NetApp kit. However, part of the reason I picked this job is precisely because I agree with how they address Exchange storage and how I think the Exchange storage paradigm is going to shake out in the next 3-5 years as more people start deploying Exchange 2010.

In Exchange 2010, Microsoft re-designed the Exchange storage system to target what we can now consider to be the lowest common denominator of server storage: a directly attached storage (DAS) array of 7200 RPM SATA disks in a Just a Box of Disks (JBOD) configuration. This DAS/JBOD/SATA (what I will now call DJS) configuration has been an unworkable configuration for Exchange for almost its entire lifetime:

  • The DAS piece certainly worked for the initial versions of Exchange; that’s what almost all storage was back then. Big centralized SANs weren’t part of the commodity IT server world, reserved instead for the mainframe world. Server administrators managed server storage. The question was what kind of bus you used to attach the array to the server. However, as Exchange moved to clustering, it required some sort of shared storage. While a shared SCSI bus was possible, it not only felt like a hack, but also didn’t scale well beyond two nodes.
  • SATA, of course, wasn’t around back in 1996; you had either IDE or SCSI. SCSI was the serious server administrator’s choice, providing better I/O performance for server applications, as well as faster bus speeds. SATA, and its big brother SAS, both are derived from the lessons that years of SCSI deployments have provided. Even for Exchange 2007, though, SATA’s poor random I/O performance made it unsuitable for Exchange storage. You had to use either SAS or FC drives.
  • RAID has been a requirement for Exchange deployments, historically, for two reasons: to combine enough drive spindles together for acceptable I/O performance (back when disks were smaller than mailbox databases), and to ensure basic data redundancy. Redundancy was especially important once Exchange began supporting shared storage clustering and required both aggregate I/O performance only achievable with expensive disks and interfaces as well as the reduced chance of a storage failure being a single point of failure.

If you look at the marketing material for Exchange 2010, you would certainly be forgiven for thinking that DJS is the only smart way to deploy Exchange 2010, with SAN, RAID, and non-SATA systems supported only for those companies caught in the mire of legacy deployments. However, this isn’t at all true. There are a growing number of Exchange experts (and not just those of us who either work for storage vendors or resell their products) who think that while DJS is certainly an interesting option, it’s not one that’s a good match for every customer.

In order to understand why DJS is truly possible in Exchange 2010, and more importantly begin to understand where DJS configurations are a good fit and what underlying conditions and assumptions you need to meet in order to get the most value from DJS, we need to separate these three dimensions and discuss them separately.

JBOD vs RAID

While I will go into more detail on all three dimensions at later date, I want to focus on the JBOD vs.. RAID question now. If you need some summaries, then check out fellow Exchange MVP (and NetApp consultant) John Fullbright’s post on the economics of DAS vs. SAN as well as Microsoft’s Matt Gossage and his TechEd 2009 session on Exchange 2010 storage. Although there are good arguments for diving into drive technology or storage connection debates, I’ve come to believe that the central philosophy question you must answer in your Exchange 2010 design is at what level you will keep your data redundant. Until Exchange 2007, you had only one option: keeping your data redundant at the disk controller level. Using RAID technologies, you had two copies of your data[1]. Because you had a second copy of the data, shared storage clustering solutions could be used to provide availability for the mailbox service.

With Exchange 2007’s continuous replication features, you could add in data redundancy at the application level and avoid the dependency of shared storage; CCR creates two copies, and SCR can be used to create one or more additional copies off-site. However, given the realities of Exchange storage, for all but the smallest deployments, you had to use RAID to provide the required number of disk spindles for performance. With CCR, this really meant you were creating four copies; with SCR, you were creating an additional two copies for each target replica you created.

This is where Exchange 2010 throws a wrench into the works. By virtue of a re-architected storage engine, it’s possible under specific circumstances to design a mailbox database that will fit on a single drive while still providing acceptable performance. The reworked continuous replication options, now simplified into the DAG functionality, create additional copies on the application level. If you hit that sweet spot of the 1:1 database to disk ratio, then you only have a single copy of the data per replica and can get an n-1 level of redundancy, where n is the number of replicas you have. This is clearly far more efficient for disk usage…or is it? The full answer is complex, the simple answer is, “In some cases.”

In order to get the 1:1 database to disk ratio, you have to follow several guidelines:

  1. Have at least three replicas of the database in the DAG, regardless of which sites they are in. Doing so allows you to place both the EDB and transaction log files on the same physical drive, rather than separating them as you did in previous versions of Exchange.
  2. Ensure that you have at least two replicas per site. The reason for this is that unlike Exchange 2007, you can reseed a failed replica from another passive copy. This allows you to avoid reseeding over your WAN, which is something you do not want to do.
  3. Size your mailbox databases to include no more users than will fit in the drive’s performance envelope. Although Exchange 2010 converts many of the random I/O patterns to sequential, giving better performance, not all has been converted, so you still have to plan against the random I/O specs.
  4. Ensure that write transactions can get written successfully to disk. Use a battery-backed caching controller for your storage array to ensure the best possible performance from the disks. Use write caching for the physical disks, which means ensuring each server hosting a replica has a UPS.

At this point, you probably have disk capacity to spare, which is why Exchange 2010 allows the creation of archive mailboxes in the same mailbox database. All of the user’s data is kept at the same level of redundancy, and the archived data – which is less frequently accessed than the mainline data – is stored without additional significant disk or I/O penalty. This all seems to indicate that JBOD is the way to go, yes? Two copies in the main site, two off-site DR copies, and I’m using cheaper storage with larger mailboxes and only four copies of my data instead of the minimum of six I’d have with CCR+SCR (or the equivalent DAG setup) on RAID configurations.

Not so fast. Microsoft’s claims around DJS configurations usually talk about the up-front capital expenditures. There’s more to a solid design than just the up-front storage price tag, and even if the DJS solution does provide savings in your situation, that is only the start. You also need to think about the lifetime of your storage and all the operational costs. For instance, what happens when one of those 1:1 drives fails?

Well, if you bought a really cheap DAS array, your first indication will be when Exchange starts throwing errors and the active copy moves to one of the other replicas. (You are monitoring your Exchange servers, right?) More expensive DAS arrays usually directly let you know that a disk failed. Either way, you have to replace the disk. Again, with a cheap white-box array, you’re on your own to buy replacement disks, while a good DAS vendor will provide replacements within the warranty/maintenance period. Once the disk is replaced, you have to re-establish the database replica. This brings us to the wonderful manual process known as database reseeding, which is not only a manual task, but can take quite a significant amount of time – especially if you made use of archival mailboxes and stuffed that DJS configuration full of data. Let’s take a closer look at what this means to you.

[Begin 4/13 update]

There’s a dearth of hard information out there about what types of reseed throughputs we can achieve in the real world, and my initial version of this post where I assumed 20GB/hour as an “educated guess” earned me a bit of ribbing in some quarters. In my initial example, I said that if we can reseed 20GB of data per hour (from a local passive copy to avoid the I/O hit to the active copy), that’s 10 hours for a 200GB database, 30 hours for a 600GB database, or 60 hours –two and a half days! – for a 1.2 TB database[2].

According to the Top 10 Exchange Storage Myths post on the Exchange team blog, 20GB/hour is way too low; in their internal deployments, they’re seeing between 35-70GB per hour. How would these speeds affect reseed times in my examples above? Well, let’s look at Table 1:

Table 1: Example Exchange 2010 Mailbox Database reseed times

Database Size Reseed Throughput Reseed Time
200GB 20GB/hr 10 hours
200GB 35GB/hr 7 hours
200GB 50GB/hr 4 hours
200GB 70GB/hr 3 hours
600GB 20GB/hr 30 hours
600GB 35GB/hr 18 hours
600GB 50GB per hour 12 hours
600GB 70GB per hour 9 hours
1.2TB 20GB/hr 60 hours
1.2TB 35GB/hr 35 hours
1.2TB 50GB/hr 24 hours
1.2TB 70GB/hr 18 hours

As you can see, reseed time can be a key variable in a DJS design. In some cases, depending on your business needs, these times could make or break whether this is a good design. I’ve done some talking around and found out that reseed times in the field are all over the charts. I had several people talk to me at the MVP Summit and ask me under what conditions I’d seen 20GB/hour, as that was too high. Astrid McClean and Matt Gossage of Microsoft had a great discussion with me and obviously felt that 20GB/hour is way too low.

Since then, I’ve received a lot of feedback and like I said, it’s all over the map. However, I’ve yet to hear anyone outside of Microsoft publicly state a reseed throughput higher than 20GB/hour. What this says to me is that getting the proper network design in place to support a good reseed rate hasn’t been a big point in deployments so far, and that in order to make a DJS design work, this may need to be an additional consideration.

If your replication network is designed to handle the amount of traffic required for normal DAG replication and doesn’t have sufficient throughput to handle reseed operations, you may be hurting yourself in the unlikely event of suffering multiple simultaneous replica failures on the same mailbox database.

This is a bigger concern for shops that have a small tolerance for any given drive failure. In most environments, one of the unspoken effects of a DJS DAG design is that you are trading number of replicas – and database-level failover – for replica rebuild time. If you’re reduced from four replicas down to three, or three down to two during the time it takes to detect the disk failure, replace the disk, and complete the reseed, you’ll probably be okay with that taking a longer period of time as long as you have sufficient replicas.

All during the reseed time, you have one fewer replica of that database to protect you. If your business processes and requirements don’t give you that amount of leeway, you either have to design smaller databases (and waste the disk capacity, which brings us right back to the good old bad days of Exchange 2000/2003 storage design) or use RAID.

[End 4/13 update]

Now, with a RAID solution, we don’t have that same problem. We still have a RAID volume rebuild penalty, but that’s happening inside the disk shelf at the controller, not across our network between Exchange servers. And with a well-designed RAID solution such as generic RAID 10 (1+0) or NetApp’s RAID-DP, you can actually survive the loss of more disks at the same time. Plus, a RAID solution gives me the flexibility to populate my databases with smaller or larger mailboxes as I need, and aggregate out the capacity and performance across my disks and databases. Sure, I don’t get that nice 1:1 disk to database ratio, but I have a lot more administrative flexibility and can survive disk loss without automatically having to begin the reseed dance.

Don’t get me wrong – I’m wildly enthusiastic that I as an Exchange architect have the option of designing to JBOD configurations. I like having choices, because that helps me make the right decisions to meet my customers’ needs. And that, in the end, is the point of a well-designed Exchange deployment – to meet your needs. Not the needs of Microsoft, and not the needs of your storage or server vendors. While I’m fairly confident that starting with a default NetApp storage solution is the right choice for many of the environments I’ll be facing, I also know how to ask the questions that lead me to consider DJS instead. There’s still a place for RAID at the Exchange storage table.

In further installments over the next few months, I’ll begin to address the SATA vs. SAS/FC and DAS vs. SAN arguments as well. I’ll then try to wrap it up with a practical and realistic set of design examples that pull all the pieces together.

[1] RAID-1 (mirroring) and RAID-10 (striping and mirroring) both create two physical copies of the data. RAID-5 does not, but it allows the loss of a single drive failure — effectively giving you a virtual second copy of the data.

[2] Curious why picked these database sizes?  200GB is the recommended maximum size for Exchange 2007 (due to backup limitations), and 600GB/1.2TB are the realistic recommended maximums you can get from 1TB and 2TB disks today in a DJS replica-per-disk deployment; you need to leave room for the content index, transaction logs, and free space.

Busting the Exchange Trusted Subsystem Myth

It’s amazing what kind of disruption leaving your job, looking for a new job, and starting to get settled in to a new job can have on your routines. Like blogging. Who knew?

At any rate, I’m back with some cool Exchange blogging. I’ve been getting a chance to dive into a “All-Devin, All-Exchange, All The Time” groove and it’s been a lot of fun, some of the details of which I hope to be able to share with you in upcoming months. In the process, I’ve been building a brand new Exchange 2010 lab environment and ran smack into a myth that seems to be making the rounds among people who are deploying Exchange 2010. This myth gives bum advice for those of you who are deploying an Exchange 2010 DAG and not using an Exchange 2010 Hub Transport as your File Share Witness (FSW). I call it the Exchange Trusted Subsystem Myth, and the first hint of it I see seems to be on this blog post. However, that same advice seems to have gotten around the net, as evidenced by this almost word-for-word copy or this posting that links to the first one. Like many myths, this one is pernicious not because it’s completely wrong, but because it works even though it’s wrong.

If you follow the Exchange product group’s deployment assumptions, you’ll never run into the circumstance this myth addresses; the FSW is placed on an Exchange 2010 HT role in the organization. Although you can specify the FSW location (server and directory) or let Exchange pick a server and directory or you, the FSW share isn’t created during the configuration of the DAG (as documented by fellow Exchange MVP Elan Shudnow and the “Witness Server Requirements” section of the Planning for High Availability and Site Resilience TechNet topic). Since it’s being created on an Exchange server as the second member of the DAG is joined, Exchange has all the permissions it needs on the system to create the share. If you elect to put the share on a non-Exchange server, then Exchange doesn’t have permissions to do it. Hence the myth:

  1. Add the FSW server’s machine account to the Exchange Trusted Subsystem group.
  2. Add the Exchange Trusted Subsystem group to the FSW server’s local Administrators group.

The sad part is, only the second action is necessary. True, doing the above will make the FSW work, but it will also open a much wider hole in your security than you need or want. Let me show you from my shiny new lab! In this configuration, I have three Exchange systems: EX10MB01, EX10MB02, and EX10MB03. All three systems have the Mailbox, Client Access, and Hub Transport roles. Because of this, I want to put the FSW on a separate machine. I could have used a generic member server, but I specifically wanted to debunk the myth, so I picked my DC EX10DC01 with malice aforethought.

  • In Figure 1, I show adding the Exchange Trusted Subsystem group to the Builtin/Administrators group on EX10DC01. If this weren’t a domain controller, I could add it to the local Administrators group instead, but DCs require tinkering. [1]

ExTrSubSys-DC-AdminsGroup
Figure 1: Membership of the Builtin/Administrators group on EX10DC01

  • In Figure 2, I show the membership of the Builtin/Administrators group on EX10DC01. No funny business up my sleeve!

ExTrSubSys-Members
Figure 2: Membership of the Exchange Trusted Subsystem group

  • I now create the DAG object, specifying EX10DC01 as my FSW server and the C:\EX10DAG01 directory so we can see if it ever gets created (and when).
  • In Figure 3, I show the root of the C:\ drive on EX10DC01 after adding the second Exchange 2010 server to the DAG. Now, the directory and share are created, without requiring the server’s machine account to be added to the Exchange Trusted Subsystem group.

ExTrSubSys-FSWCreated
Figure 3: The FSW created on EX10DC01

I suspect that this bad advice came about through a combination of circumstances, including an improper understanding of Exchange caching of Active Directory information and when the FSW is actually created. However it came about, though, it needs to be stopped, because any administrator that configures their Exchange organization is opening a big fat hole in the Exchange security model.

So, why is adding the machine account to the Exchange Trusted Subsystem group a security hole? The answer lies in Exchange 2010’s shift to Role Based Access Control (RBAC). In previous versions of Exchange, you delegated permissions directly to Active Directory and Exchange objects, allowing users to perform actions directly from their security context. If they had the appropriate permissions, their actions succeeded.

In Exchange 2010 RBAC, this model goes away; you now delegate permissions by telling RBAC what options given groups, policies, or users can perform, then assigning group memberships or policies as needed. When the EMS cmdlets run, they do so as the local machine account; since the local machine is an Exchange 2010 server, this account has been added to the Exchange Trusted Subsystem group. This group has been delegated the appropriate access entries in Active Directory and Exchange databases objects, as described in the Understanding Split Permissions TechNet topic. For a comprehensive overview of RBAC and how all the pieces fit together, read the Understanding Role Based Access Control TechNet topic.

By improperly adding a non-Exchange server to this group, you’re now giving that server account the ability to read and change any Exchange-related object or property in Active Directory or Exchange databases. Obviously, this is a hole, especially given the relative ease with which one local administrator can get a command line prompt running as one of the local system accounts.

So please, do us all a favor: if you ever hear or see someone passing around this myth, please, link them here.

ExTrSubSys-Busted
Busted!

[1] Yes, it is also granting much broader permissions than necessary to make a DC the FSW node. Now the Exchange Trusted Subsystem group is a member of the Domain Admins group. This is probably not what you want to do, so really, don’t do this outside of a demo lab.

The Great Exchange Blog Migration

Over the next few days, I’ll be adding a large number of posts (just over 250!!!) to the archives of this blog. For a number of congruent reasons, 3Sharp is closing down the Platform Services Group (which focused on Exchange, OCS, Windows Server, Windows Mobile, and DPM) and my last day will be this Friday, October 16 after over six and half years with them. With 3Sharp’s gracious permission and blessing, I’ll be duplicating all of the content I’ve posted on the 3Sharp blog server over to here. If you have a link or bookmark for my work blog or are following it via RSS, please take a moment to update your settings. Yes, that means there’s going to be more geeky technical Exchange stuff going forward, but hey, with a single blog to focus on, maybe I’ll be more prolific overall!

To head off some of the obvious questions:

  • This is not a horrible thing. 3Sharp and I are parting ways peacefully because it’s the right decision for all of us; they need to focus on SharePoint, and I’m so not a SharePoint person. They’ve done fantastic things for my career and I cherish my time with them, but part of being an adult is knowing when to move on. We’re all agreed that time has come.
  • I’m not quite sure where I’m going to end up yet. I’ve got a couple of irons in the fire and I have high hopes for them, but it’s not time to talk about them. I am going to have at least a week or two of time off, which is good; there are several projects at home in dire need of sustained attention (unburying my home office, for one; fixing a balky Exchange account for another).
  • I’m not going to be a complete shut-in. I’ve got a couple of appointments for the following week, including a Microsoft focus group and a presentation on PowerPoint for Treanna’s English class. I’m open to doing some short-term independent consulting or contracting work as well, so contact me if you know someone who needs some Exchange help.

Thank you to 3Sharp and the best damn co-workers I could ever hope to work with over the years – and a huge thank you to all of my readers, regardless of which blog you’ve been following. The last several years have been a wild ride, and I look forward to continuing the journey with many of you, even if I’m not sure yet where it will take me.

Am I hot or not?

Stupid website, but it gives me a chance to taunt my co-worker Kevin. This morning I got a puzzled e-mail from him, asking me why this picture of me in Sydney from February (yes, that’s Sydney, Australia; we were there for training for work) was the most-viewed picture in his online galleries (warning, probably not a worksafe gallery). I have no clue, but I think it’s damned funny.


Kevin’s a hard-core picture nerd; he’s got a wireless card for his digital cameras that will automatically use any nearby open WiFi connection to upload pictures to his Web gallery. This means that on a trip he’s usually got pictures uploaded before he gets back to his hotel, let alone before he gets home. That’s pretty cool, even if (like me) you aren’t inclined to take gigabytes of pictures everywhere you go.

Thoughts from the break part 1

So, I’m in Sydney for a training conference that I’m talking about in my work blog if you’re interested. There’s a lot of interesting small differences that have more of a mental impact to me than the big ones:



  • The exit signs inside buildings are green with white letters. I’m used to the opposite.

  • Didn’t find a single country radio station. Of course, this could be because the alarm clock/radio in my hotel room is cheap.

  • Speaking of hotel rooms, holy crap are they small! I’m having flashbacks to the really crappy hotel room in London from a couple of years back.

  • Did I mention that the hotel rooms have a distinct lack of ornamentation? Very small, very functional, but it feels like living in a cupboard.

  • Apartment buildings are painted interesting colors.

  • Window dimensions are subtly off.

  • They’ve got Taylor Swift’s Teardrops on My Guitar on the muzak system here at the conference center. I noticed an interesting lyrics change: the line “it’s just so funny” is “it’s so damn funny” here. Apparently, in the United Nanny States of America, the terrorist will win if a 17-yo girl says “damn” in a country song.

  • The magazine in my hotel room had Nicole Kidman on the cover, but I’ve not yet seen a single mention of Kylie Minogue.

  • Speaking of Nicole Kidman, she’s done two movies with Daniel Craig — The Golden Compass and Invasion, which I watched on the plane. Basically another cover of Invasion of the Body Snatchers, only they wimped out on the ending.
Whoops! Time to go, more later!

Heading for the underside of the world

In just a few more hours, I’ll be on my way to do something I’ve never done before — cross the equator. I’m just going to kayak down to the big black line floating on top of the ocean, nip across, spin around counter-clockwise, then nip back home.

No, seriously — I’m heading to Sydney, Australia with my cow-orker Kevin to a week-long training conference. I think I might be able to survive the flight. I leave Seattle on the evening of February 2 and land in Sydney on the morning of February 4. By my calculations, that’s nearly 40 hours of time — but due to the vagaries of the International Date Line, I’ll experience far less of that (Sydney’s 16 hours ahead of us). In effect, I will not exist for February 3. That’s right, folks, I’ll be completely non-existent for my 12th wedding anniversary. Don’t worry, though; I’ve been aware of this for long enough to have made (and executed) plans and all the proper observances have been made; Steph will not be shortchanged.

Other than survive my time in time-zone limbo and the grueling plane flight, I’m hoping to meet up with a longtime net.friend and fellow SJGames freelancer and maybe even get to do the Harbour bridge climb. So, if you don’t hear from me for a week, it’s all good — I’m having fun down in the land of the Southern Cross.

Whoa. I just realized, that if I see any stars, they’re going to be totally different.

Living with Asperger’s on the road and on the stage

Four 3-day trips in four weeks:



  • Apr 2-4, Orlando, to present 3 hours of sessions at Exchange Connections.

  • Apr 8-10, Denver, to be the main speaker at the Exchange 2007 Unified Messaging Roadshow.

  • Apr 18-20, Anaheim, to be the main speaker at the Exchange 2007 Unified Messaging Roadshow.

  • Apr 23-25, Dallas, to be the main speaker at the Exchange 2007 Unified Messaging Roadshow.

For those of you keeping score at home, yes, it’s why my blogging has been very sporadic of late. And I’m particularly annoyed about the timing of the current trip; turns out John Scalzi was doing a signing in Seattle yesterday, and that would have been something I would have gone to had my travel schedule not prevented it.


Needless to say, I’m a bit burnt out. Travel always screws me up to begin with; this series has been particularly hard, because the last time I did this kind of back-to-back travel was several jobs ago (not that I cared for it then, either). And I’m not done yet: I still have one more Exchange 2007 roadshow date in Phoenix May 14-16, although at least for that one I’m flying into Tucson a couple of days early and meeting up with my parents and sister for the weekend.


Travel screws up my sleep schedule, big time. I finally got a decent night of sleep last night — I’m not sure how — but my body is repaying me now big time with massive insomnia. Partly it’s being away from home in a strange place and bed, partly it’s that the hotel beds are never quite right no matter how comfortable they may be (and since the roadshow has been putting me up in decent hotels, bed quality is actually pretty good). It screws up my eating, especially when I’m speaking — I hardly ever can eat lunch on a day I’m speaking, because I’m just too damn busy/nervous; even if I did find the time to eat and could find something suitable in whatever catered options we have at the events, I’d probably just throw it back up. It doesn’t help that I’ve made some recent big changes to my normal routine, and I’m trying to keep those changes in place and going even while traveling.


Plus, since I’m outside of my routines, I’m an insecure nervous wreck. I’ll take 15 minutes to lay out my clothes and various articles for the next day, then re-check them six times before I go to bed. I’m very precise about how I unpack and where I put stuff. I maintain a level of worry just below “freak out” over things like getting to the venue/airport on time, and obsessively check and verify addresses and route maps. Not that this level of preparation is a bad thing, mind you; I hardly ever have to hurry in the morning (which is good because I’m usually groggier than crap), I hardly ever forget to take stuff along that I need, and I’ve been able to just hand our taxi driver a post-it note with the address of the venue the last two cities. But this level of obsessiveness takes it too far, and dumps me right in the middle of awkward, paranoia mode, which is so not helpful. If I’m chatting with one of my co-presenters and there’s a lull in the conversation, I’m immediately worrying that it’s a direct result of something I’ve done or said; if I don’t get included in a casual conversation, I spend minutes trying to figure out why. Take me out of my routine, and my Asperger’s isn’t at all far underneath the surface, no matter how well people tell me I conceal it.


The funny part is, I really love speaking — and the bigger the audience, the better. Smaller audiences require me to deal with a collection of individuals, which taxes my social skills to the limit. I find smaller audiences usually tend to be “flatter” — they don’t react as well to the jokes I make, they don’t tend to ask questions of the same intensity, and I just don’t seem to “click” as well with them. This is disappointing; I want my audiences to feel like they’re getting not just the technical information they paid for, but I want them to be entertained. I want them to feel like I’ve helped them. Hardly anyone who comes up to me afterwards and asks a good, hard question ever takes me up on my offer to email me so I can research and give them an answer. I tend to get good marks and comments on my feedback sheets, so if the people listening feel like I’m doing them a disservice, they’re not complaining about it, but I just can’t read them.


Give me an audience of 150+ people, though, and I start to have fun. I’m only nervous for a few seconds, and then something clicks and I turn on. The few times I’ve talked in front of a really large audience, I had great sessions — lots of fun, lots of laughter, and lots of good questions. 


Having said all that, for being a smallish group today, the crowd here in Dallas was just plain fun. At the other venues, I’ve left my cowboy hat off when I got on stage; I left it on here and was able to get a laugh from it (at Anaheim’s expense; sorry, SoCal!) My post-lunchbreak observation that bringing in 100% clouds and rain to make the Seattle boy feel more at home was probably overkill got another good laugh. Thanks to everyone who showed up and had a kind word or question for me; y’all were great.


Now to see if I can get an hour or two of sleep before the wake-up call drags me out of bed. Have to get to the airport early enough to be sure to get on my flight home, since I’m not sure what kind of crowding has resulted from last night’s weather-related ground stop. I’d be more than mildly stressed at this point if I didn’t make it home tomorrow reasonably on-time.

bps

Big conference, lots of geeks. With two laptops and a Windows Mobile cellphone, I’m not at all unusual here. The conference organizers have wisely established free Wi-Fi for us all.


Well, guys, that’s a great start, but how about you give us some bandwidth? I’ve had faster connections over my old 300 baud modem.
 

Drive-by memeing

First, I would like it to be known that I am only doing this under protest. I don’t usually participate in blog memes, because most of them are damned silly. However, I got tagged on this one by Paul in what looks like a fairly typical spree of spreading the love, so I’ll go ahead and do it.


So here’s the meme, in Paul’s words:



The latest craze sweeping the series of tubes is “5 Things”, a sort of chain letter in which victims participants are supposed to list 5 things that others may not know about them, then pass the baton on to some other people.



And here are my responses:



  1. Those folks who read my professional blog (e)Mail Insecurity have already figured it out, but those of you here have probably not heard about it yet. This week, I got the official notice that I had been awarded the the 2007 Microsoft® Most Valuable Professional (MVP) Award in the technical community of Microsoft Exchange. Basically, this means that Microsoft has noticed and appreciated the work I’ve done out in the real world (blogging, speaking, writing, spending time on mailing lists) helping people learn about and use Microsoft Exchange Server. It’s has some neat perqs that come with it, including a great network of other MVPs (many of whom I’ve already been blessed to work with over the years) and more direct access to the Exchange product group at Microsoft. Of all the things I’d envisioned for my five-year goals, this wasn’t one of them, and I’m truly blown away that I’ve been selected.

  2. Most of you know that my ambition is to be a full-time sf author and have many novels and story ideas in progress. Many of you know that I also enjoy singing and writing music, going so far as to dabble with guitar and keyboard. What almost none of you know (hush, Steph) is that I have the ambition to write and produce my own professional fully-sung Eucharist liturgy (a Christian Communion service, for those of you not up on high church terminology). I’d write it so that the congregation would definitely have parts that they’d join in, but there’d be four main celebrants (SATB, of course) with much harder parts to perform. In my perfect world, I’d be able to entice Jason Michael Carroll, Sting, Alison Krauss, and Sarah McLachlan into performing at the inaugural celebration of the liturgy.

  3. Taking a cue from Paul, I had my first paid computer job when I was 12. The secretary at the resort Dad was working at needed someone to do some data entry for her, as they’d just switched her IBM PC from one accounting package to another and she needed to get the accounts receivable data into the new software. IIRC, I was offered the princely (for the time) sum of $8 an hour. We estimated that it would take around 24 hours or so, so I was standing to make quite a decent chunk of change. The first morning, I went into the office, acquainted myself with PC-DOS for the first time, and spent the first four hours doing data entry. When lunch came around, I grabbed the manuals and read them while I ate. I noticed that the new package talked about being able to import data from a variety of programs (none of which was the old package) and formats, so I checked the manual for the old program. Sure enough, it could export to one of those same formats. I backed up the work I’d done so far and tried the export/import. Perfect! You’d think she would be happy, but no — she was quite upset that a 12-year-old had figured this out and somehow made her look bad. She paid me for one single hour of my time — since the actual export/import work had taken one hour and was in a separate data file from the one I’d spent the morning on, she claimed that it was the only work that counted — and that was that.

  4. While I grew up in Oregon and have spent the majority of my post-college years in Washington, I am not in fact a native Pacific Northwesterner. My family actually comes from back ’round Wisconsin and Michigan, and we moved out to Corvallis, OR when I was just 11 months old. The Pacific Northwest Native Advisory Board did, in fact, take this into consideration, decided that it wasn’t my fault I couldn’t get my parents to move out here before I was born (and in fact one member of the panel commended me for “extraordinary action in relocating his family while still shy of his first birthday”), and granted me PNW native status anyway. This is good, because if I didn’t have that status, I wouldn’t be able to gripe about the Californians as is the right of all native PNWers.

  5. During high school, I participated in an academic competition at our local community college. To fill out an empty time slot so I could take the entire day off, I picked the radio broadcasting competition, since when I was a young lad I used to spend hours in my room with cassette recorders pretending to be a DJ. The next year after the competition, I spoke to the college radio faculty director about doing a 15-minute radio show focused on events at the high school. Suddenly, I found myself gathering information for, recording, and producing a weekly radio show. The poor college DJ who had to run my piece before his own show quickly grew to hate me, as I pushed the envelope of what I could do by including clips of favorite pop songs and completely harshed the mellow of his own show (which was heavy metal, IIRC). I had the complete backing of the faculty director, though, so there wasn’t much he could do. My first year of college, I took radio as a pass/fail credit and continued harshing the mellows of the broadcasting program students; my format, right in the middle of a highly-desired timeslot, was an eclectic combination of news commentary, music selection and experimentation from all genres (there was literally nothing I wouldn’t play), and pure naked listener gratification. I must have been making someone happy, though it wasn’t the “serious” broadcasting students; I enjoyed a constant high level of feedback from the surrounding community. Again, that kept The Powers That Be from stepping in and messing with my groove.

I’ll just note here, for the record, that I’m only doing this because I already have a couple of things I wanted to blog about and I can twist this meme to my service. The fact that I’ve been needing to update here is just extra gravy. The fact that one of my other co-victims needs to actually fix his blog server before he can respond just makes me feel better about the whole thing.


And now on to my victims, which is the hard part. I’ve been seeing this meme running around the tubes for a while, so anyone who hasn’t already done it is either less connected than I am or just as likely as I am to say “Poppycock!” at the whole concept and just not participate. With that caveat in mind….


….I choose you, Alistair, AndrewBrian, Nick, and Steph (in alphabetical order so no ranking is implied).

3Sharp releases an anti-phishing study

Today, my company released a report comparing several anti-phishing technologies, including a beta version of Microsoft’s Internet Explorer 7. I note this because I was one of the researchers and authors of the report. Needless to say, we’re already catching a lot of flack in the press because we scored IE7 the highest of the various products.


Many of you have heard my opinion of IE6 (I switched to Firefox back in the 0.4 timeframe). I just want to say that through my exposure to IE7 as gained through the testing for this project, I ended up switching back to IE7. I’m now using IE7 RC1 as my main web browser, and slowly converting my mass of Firefox bookmarks back over to IE7 RSS feeds and favorites (as appropriate).


So, make of that what you will.
 

Ode to Monday: a rant on insensitivty in the workplace

Not fond of Mondays in theory or in practice. Even with all the things that have gone right today, I’m still pretty cheesed off about today:

  • Getting up at 5:20 so I could be on the road before rush hour and into the office early: good.
  • Having to shift back into the office: enh. We’ll call this a draw.
  • Forgetting the box with my laptops, thus necessitating a second round-trip: bad.
  • Having to deal with my schedule and a new dentist to take care of the tooth (top front left) I broke on Friday: bad.
  • Having a co-worker make repeated references to my broken tooth, laughing and saying it makes me look like a hick: very bad.

You know, dude, the first time you laugh at how my broken tooth makes me look like a redneck, then apologize because it was insensitive, I’ll give it to you for free. When you go on to say it again and keep laughing…

…why was I moving back into the office again? Oh, yeah, to increase my productivity and help re-integrate me into the team. Newsflash: I feel so integrated right now.

Just for reference, the previous sentence was sarcasm and not to be read literally.

That went well

The current project I’m on at work? I’ve decided it needs a new name. I’m henceforth calling it Operation Zeno’s Paradox.

You know you’re in trouble when you’ve finally found a solution to a
roadblock that is keeping you from finishing a major portion of your
project, serendipity has dropped a convenient piece of hardware in your
lap with which to effect this solution, and then in order to ensure
that your day of brilliance is turned into wasted effort — not to
mention stalling your continued hopes of forward progress — that
hardware blows up. And spectacularly, I might add. I’ve seen power supplies blow up before, but never as enthusiastically as this. Huge clouds of white smoke pouring from the machine.

My first thought was literally, “Wow. I knew somebody didn’t want me to finish this project, but this
is a little much.” Then I sighed, calmed Treanna down (she was standing
two feet away from the machine at the time), and hauled the now-carcass
of a computer back to 3Sharp to rebuild.

Let’s hope the redeployment goes more smoothly, yes?

Disgruntled

I hate putting in long hours to compensate for fucked-up technology. Even before my laptops bit the big one, I was having problems with one of them. Since then, they’ve gotten a lot more flaky because I haven’t had the time to migrate all of my data, pare down the cruft and crap, and perform fresh clean installs.

So instead, I get to put up with a piece of shit that insists on dropping a vital network connection at random and inconvenient times, which of course cannot merely be restarted as I’m supposed to be able to do, but requires me to save all of my data and reboot the damn thing. This is the third time in less than an hour I’ve had to reboot this bastard. I’m already depressed about this project to begin with, and running hard against a deadline. I don’t need the additional stress and aggravation of a laptop that’s trying to get me to blow my fuse.

Message to my T30: Just. Fucking. Work. I’ve not been working my ass off and dealing with all of the other bullshit I’ve had to, I’ve not been stressing over my complete and spectacular lack of stellar performance at work on this project, I’ve not been worrying about getting fired just so I can be sabotaged by a Thinkpad with delusions of Dellhood.

Back from Europe

I’m home from Europe a couple of days early; I was originally supposed to be in Oslo today, doing a second presentation, but I got really sick once I got to London and my bosses couldn’t take the chance I wouldn’t be better (sorry, Jim, for making you go to Oslo!). Lisbon was interesting. Paul talks about it a bit and relates some of the interesting bits of what happened there, but I have a couple of comments to add:



  • I was less than thrilled to find that they seem not to serve straight orange juice, but like to mix in other things. From the one sip I had my first morning, those other things might include pineapple, which would have been a huge problem for me.
  • The steak was awesome. The bacon was not, and the McDonald’s burger just didn’t taste like real beef.
  • Walking on the sidewalks was an interesting experience. First, they were pretty narrow — combined with narrow streets and fast city drivers, it made it feel like you were taking your life in your own hands on certain streets. Next, they were cobbled with irregular stones, and the resulting surface wasn’t at all level. Added strain to my feet.
  • I found out on Monday afternoon that my track (Exchange 2007 for the IT Pro) was in the main auditorium. It was a big auditorium, and there were a lot of people who had signed up for it. It seemed to go well, though; I got a good laugh from my intro joke and a lot of good feedback.

Being sick sucks. Being sick in a foreign country in a shoddy hotel really bites. And I was pretty out of it — the only reason I wasn’t in more serious trouble is that a couple of college friends (Mickie and Jen) were at the same hotel and keeping an eye out for me.


I’d been planning on going and staying with Jen and her husband Rich in Harrogate (up by York) on the weekend anyway, but instead of going back to London Sunday night I just stayed a couple of extra days and got better. It was definitely the right thing to do; the area right around Kings Cross station (where my dodgy hotel was) was a bit grimy and depressing.


The highlight of my trip was going to York on Sunday. I still wasn’t fully up to speed, so we focused on the Minster (the giant cathedral there; I’d been planning on trying to catch Sunday services there, but we didn’t get out the door quite on time and I didn’t feel like hanging around for the Evensong service at 4pm). That is a lovely cathedral. I now plan on going back there with the family at some point.

Euroadtrip, Day 1: Amsterdamned!

I’ve not been blogging much lately because I’ve been buried under a ton of work. One of the things I’ve been busy with has been developing a full day of content on Exchange 2007 to present in Lisbon (May 30th) and Oslo (June 8th).


Astu Are thte readers will have noticed the timestamp of this entry, done some math, and realized that I’m probably not in the US anymore. Those readers would be correct. I’m sitting in one of the food courts (near the B and C concourses) of the Amsterdam Schipol airport, waiting for my connecting flight to Lisbon in just under three hours.


This is my first international flight, and other than some business trips to Hawaii, my first time off the North American continent. Since I’m sorta wiped out and still have work to do before I take off, I’ll share just a short list of observations (in no particular order):



  • Airports look like airports, no matter where you are.
  • I’d not realized exactly how much I appreciated the US airport (and Washington state) total ban on smoking until, halfway through my meal, the nice people at the next desk lit up.
  • What’s up the the “T” gates (Transfer) they have here? What are those for? They look like ticket counters…but surely you buy your tickets ahead of time, not hop-by-hop, right? Are they an Amsterdam peculiarity, or are they an EU thing?
  • Wifi providers here are just as willing to take adavantage of your wallet as they are in the US.
  • Amsterdam is flat. Really flat.
  • You don’t hear announcers in US airports naming specific people for specific flights, telling them that they are delaying the flight and threatening to offload their luggage. You do here.
  • They have a Star Wars Transformer in one of the toy shops here. No shit. Anakin’s Jedi Starfighter turns into a fighting Jedi mecha. Who knew? And it’s using the real Transformer brandname — has the trademarked “More than meets the eye” tagline and everything.

I might do more later, if I’m not complete tapioca after 17 hours of travel plus an afternoon of session prep.

Study monkey, but with good beer

I have to acquire some Microsoft certifications in a short amount of
time. Not a full MCSE, although that’s always been in the gameplan, but
at least one passed exam (and thus Microsoft Certified Professional
status) from a specific list of exams, Real Soon Now. And hey, if I’m
going to go into MCSE Study Monkey Mode (MSMM), I might as well do it
right and get my MCSE out of it, yes? Then I can worry about getting
some of the hard, security related certs.

So I’ve been cracking the books and taking practice exams. For the
most part, I know the material — if I didn’t have to worry about
finishing ebook chapters and developing talks for Exchange Connections,
I could easily have my MCSE within the next couple of months — I just
have to learn Microsoft-think. Most of my errors on my practice exams
so far have been because I haven’t acquired the particular mental
shorthand the tests expect you to have. Example: one of the Exchange
questions I hit asked you to select, from a list of tasks, the three
tasks you needed to complete to install Exchange in a certain scenario with the least amount of administrative effort.
It turns out the three tasks were all concerned with making sure the
account you were installing from had the proper permissions. I missed
one of those permissions because I selected the task where you ran two
particular pre-installation routines (Exchange admins know them as ForestPrep and DomainPrep).
I’ve been dealing with convoluted Exchange organizations for so long
now that manually running those steps has become second nature and I
just plain forgot that if you have the correct permissions on your
account, the Exchange installer will automatically perform those steps
for you when you install the first Exchange server. I missed two
questions because of that particular thinko. Le sigh.

It’s all good, though; I’m confident I’ll be able to get a decent
set of successful exams under my belt in the next couple of weeks and
be well on my way to my MCSE cert (seven exams required, with two more
if I want the messaging competency, which I do). And since we had to
run into Redmond today (I needed to swing by the office and grab some
software I need for a couple of current projects), we swung by the
British Pantry, a neat little shop that sells all sorts of great
British items including food and beverages. The Strongbow cider is hawesome (don’t know about hawesome yet? This blog entry from Nickerblog
should help), and even the Newcastle Brown Ale (which I bought on the
advice of a British gent who admitted it’s crap beer but spun a funny
enough story about it that I agreed with his conclusion that it was
worth trying at least once) was far tastier than American beer. (When
he said it was less than stellar beer, I replied that I was used to
that, being an American.) Clearly, I must tour the UK and sample their
beers when even their crap beers taste better than the horsepiss we’ve
got here.

Oh, for those who know, my homebrew experiment failed utterly, so
there will be no tasting parties in the near future. I’m not sure if
I’m going to sink another $65 into getting supplies for a second
attempt, so we’ll have to see what happens. There might be some
alternatives — or I could just drop the whole project altogether. I
need another vice at this point like I need another hole in my forehead.

Mileposts

Today was a day of mileposts.


It was my first working day of 2006. Because Paul and I got our various projects wrapped up before vacation, I went into the 3Sharp offices and spent a good part of my day doing something I’ve never done before — pack up my office. No, we’re not moving locations, and yes, I’m still with 3Sharp. However, we’re getting tight on office space and since my average in-office time over the last year has probably been close to one day a week, I realized it was going to be a lot less disruption if I offered to give up my nice one-person office and officially move everything to my home office. It made all sorts of sense, but I found myself feeling very uncomfortable getting the first couple of boxes packed up until I realized that my only previous experiences with packing up my office were at the end of a job, not switching locations while staying employed. Once I realized that, the rest of the packing went a lot more smoothly.


Today was also the day I won my first Texas Hold’em tournament. One of the local bars hosts a couple of freelocalpoker.com tournaments every Tuesday and a friend and I checked them out last November. Now that the anti-smoking law has kicked in here in Washington, I can go play without coming home reeking, so I showed up tonight for the 9:30 game. We had ten players — a smaller turnout than normal, even though the later game is usually a bit smaller than the 6:30 game — so we all crowded around a single table. Two hours later, I was the proud possessor of a $15 gift certificate for the bar. Not only that, I’d earned the bounty (extra points received for being the player to knock the previous winner out of the game). Unlike previous games (where I can remember all the hands I lost all too clearly) I don’t remember much; the only hand that really sticks out is the hand where another player and I both got a straight and split the pot. There are bits and pieces of the rest of the game, but I have no idea how I won. I do remember (if my math is correct) that I knocked out five people — I had a couple of hands near the end where two players had gone all-in, so when I won those hands, I knocked out two players at once.


Finally, today was Steph’s birthday. And now that she’s gotten her nose out of the new Honor Harrington book I got for Christmas, she gets the rest of my attention before sleep time.

San Diego

This last week (Monday through Thursday), I was in San Diego for the Windows/Exchange Connections conference. It was my first Connections conference as well as my first trip to San Diego.


San Diego was a nice town; it reminded me somewhat of Honolulu with enough elbow room. Mind you, I didn’t see too much of the city; we were at the Hyatt down on the water, only a couple of blocks away from the Midway memorial (a two-hour tour that I didn’t get time to take). What I did see, though, was fairly clean and had character. I met up with Mickie and one of her local friends for dinner at the Cheesecake Factory on Monday night and can report that Californian drivers are still nuts. Although I did notice that both of the taxi drivers I used drove perfectly well — signals, ample lane-change space, slowing to let people merge — and showed signs of sanity I’ve not seen in taxi drivers anywhere else. Apparently the combination of California + taxi driver undoes the evil of both stigmas.


The conference went well; check out my work blog for more specific information on things I did and saw (I’ve written parts one and two so far, with probably two more on the way).

Sick days and publication news

Both kids woke up with an excess of phlegm, so we kept them home from school today. Sounds like a good day for them to work on updating their web sites and blogs, especially since I just handed out the URLs to the members of my family.

In other news, Microsoft has produced printed versions of one of the solutions I worked on in 2004, the Windows NT 4.0 and Windows 98 Threat Mitigation Guide. They’ve printed quite a few of the solutions produced by this group. Printed copies are $5 from the Microsoft web site.

Also, Chapter 2 of my ebook on DCAR (discovery, compliance, archival, and retention) is now up for download. All in all, a busy week. I’m very thankful for this project; I just got my first payment last weekend and it means Christmas shopping is going to go much easier this year.

If it’s a meltdown, it must be Monday

It’s a Monday. With a vengeance.

Work has been better, the car needs $1K+ in repairs soonish, some personal projects have been hit with unfortunate and untimely delays that I really didn’t need right now, oh no, and I’m stressed beyond all belief.

I really just want to stop coping and have a full-scale meltdown. Preferably in front of the idiot psych dude at UW who said there was no way I could be Asperger’s because I’d managed to stay married. I have news for you, Psych Dude — my marriage is one of the few stable things in my life and it gives me the strength to cope with all of the rest. And even then, there are days where it’s a near thing.

Where do I find the spiritual and emotional equivalent of bailing wire and duct tape? My supply is running out fast.

My new ebook on email DCAR is now available!

DCAR (Discovery, Compliance, Archival, and Retention) is a big topic these days. I’ve done some work with 3Sharp in this space, and I have been fortunate enough to be given the opportunity to write an ebook for Windows IT Pro on the topic.

The first chapter is now available for free download with registration; the remaining chapters will be coming out over the next few months. So if this is something that interests you, feel free to go take a look.

How big of a place setting did you say you need?

I just got off of a weekly conference call for a biggish project we’ve been working on. We’ve had some unfortunate changes lately that have impacted the scope of the project in a fashion that would not be exaggeration to call “dramatic”. One of those changes dumped a fairly involved and possibly gnarly piece of work square in my lap, which of course left everyone else free to not worry about it. Instead, they’re all worrying about the other major change of scope.

As is my wont, I was blowing off steam by IMing one of the other participants in the call and mentioning that the only reason I’m not stressing about Major Change #2 is that I’ve got to focus on Major Change #1 or I’ll be completely overwhelmed (and in all honestly, I’m almost there just with Major Change #1). The response I got was this:

eat one elephant @ a time

That about sums it up perfectly. Now to find my elephant fork…

Yo Missy!

Um, Missy dear, what exactly have you been telling the fine Exchange ladies and gentlemen about me? I’m taking an incredible ration of shit from the assorted suspects and they keep mentioning your name. And Tom, your name has gotten taken in vain more than once.


And for the record, my kilt shows off what a stud I am. Disagree all you want, but a certain luscious redhead says so and she trumps.


That is all.

All-nighters and Mutual of Omaha’s Wild Kingdom

I’m staying up late to get a paper done for work. While I’d rather not pull late nights and long days, in reality I have an extremely high amount of flexibility and self-determination in the day-to-day details of my job. The flip side of that coin, however, is that I keep that freedom as long as I produce the deliverables I have been assigned by my deadlines. Having the freedom to take a bit longer lunch break and go on a walk with my wife, or work from home, or not have set hours that I have to be in front of the computer working, means that I have in turn committed to putting in the time to get stuff done.

In a way, I think I would do superficially better with a more strict office regimen, but I think my production would suffer accordingly. (One of my bosses seems to agree; last time I was in the office and we chatted, he mentioned that he’s seen my productivity climb noticeably since I’ve been working from home again). Not having to toe the line means I don’t have any excuses for bad performance. The end result is that I’m developing routines and better work habits because I want them, in order to avoid late nights and long hours. It’s an ongoing process.

Every now and then, my late nights have an unexpected side benefit. A few minutes ago, I heard what sounded like a heck of a catfight in progress out in my carport. After it continued for a few minutes, I realized that, no, it sounded like a dog — some small yappy thing — and a cat. I got a flashlight and went outside to break it up. At this point, I’m in pajamas and sandals.

About 10 seconds after I get outside, the flashlight is useless. Great, dead batteries. The dog is cowering directly under our car, giving out weird yips. It’s been hurt, I think, and I have a momentary fleeting sense of admiration for the cat (since I really don’t like small dogs and I love cats). But something is breathing very heavily, and I realize that it’s not the dog. So I walk around to the end of the car to take a look, because I’m starting to get the bad feeling what we’ve got is a poodle getting beat up by an enraged possum. Sure enough, I get around to the end of the car and look on the other side (still waving the useless flashlight, because it is large and has a comforting weight), and there’s a large shadowy figure that’s too large to be a cat. This thing is panting and hissing and is paying absolutely no attention to me; it’s fixed on the tasty treat that is probably busily pissing itself all over the concrete underneath my Ford Focus.

It’s about this time that I finally realize what is really going on.

I’ve wandered into the middle of two very pissed off, fighting raccoons. I’m standing out in my carport with a dead flashlight in plaid flannel pajamas and open-foot sandals at 3:30am in easy striking range of two big, nasty, energized scavengers who are more than willing to take a swipe at me if they feel that’s their best way out of danger. A year or two ago, we had a raccoon get treed in our yard; when the cops came to deal with it, the not-so-little bastard charged one of them. These are junk-fed raccoons with a sense of entitlement. So what the heck am I going to do?

Okay, yes, I spent a couple of seconds wishing I’d followed my impulse to grab my thick quarterstaff before coming outside. At least then I’d have a long stout stick to try to beat them with and gain precious seconds before my toes get gnawed off by the rabid little berserker sizing me up from the driver’s side of my car. And then I decided turnabout was fair play and started yelling at them. I wasn’t yelling too loudly — I really didn’t want to wake the neighbors up — but it was enough to get their attention. The raccoon under the car stopped his “Shit, I’m BLEEDING!” yipping while his opponent looked over at me and growled even more loudly (don’t worry, buddy, I don’t care if you beat him up, just go do it in someone else’s yard) as if to say, “Zip it, you wimp.” I refused to be cowed (and here’s where I do NOT believe myself) and even stepped closer to him.

At that point, he backed up. Trapped Raccoon saw this as a great opportunity and took off like the fat waddling bastard he was, running to his freedom across my front lawn. Mean Raccoon is justifiably pissed at this turn of events; garbage day was a couple of days earlier, so pickings have probably been slim for the past few days and here some dork in pajamas just chased off his midnight snack. He and I get into a real macho staring match; he’s growling and telling me nasty things about my parents while I’m doing my best crazy Walter impression trying to break his nerve. Eventually, he realizes there’s no point in standing there arguing with me; I’m too big to eat and Trapped Raccoon has a heck of a head start. He finally breaks off and heads back around the back of the house. Within a minute, I hear the fight break out again, but this time it’s a running fight that is half a block down and moving farther away all the time.

Mission accomplished. Crap, that was some adrenaline.

It’s done!

I just shipped the first draft of Chapter 7 “Routing, Transport, and SMTP” of The Exchange Cookbook to my editor and co-authors. I wrote this chapter entirely by myself. It started off with 47 recipes (including the introduction); final size was 27 recipes. Many of the missing 20 were duplicates of either recipes in other chapters, belonged better in other chapters, or could be added in with existing recipes in chapter 7. Only a few got dropped because of time pressures.

It was a ton of work. I have a lot more respect for technical authors.

Now comes all the revising and technical reviews…but the draft is done!

Scary revelations at 2am

Put Steph to bed a couple of hours ago and came back downstairs to finish up a recipe or two for the Exchange Cookbook, which I am co-writing with some awesome co-workers, and which is way behind schedule. When researching one particular twist got me frustrated, I dropped over to Yahoo!’s Launch website and fired up some streaming music videos in the background.

Holy crap, dude.

Found out that Lindsay Lohan — that precocious red-head who did a pretty good job of filling Hayley Mills’s shoes in Disney’s 1998 remake of The Parent Trap a few years back — is doing the “Disney girl” thing and has a record album out, and one of her music videos (Rumors) has hit the top of the various countdowns. Decided to watch it.

For those of you on slow connections, let me save you the trouble; Ms. Lohan has put out a video that pretty much leaves her marching squarely down the trail of rebellion, “strong woman” independence, and barely-legal sluttishness that Britney Spears has perfected to a tee. The song isn’t too bad — certainly better than Britney’s recent efforts — but it’s tired and trite.

However, I’m not writing to rant about yet another 18 year-old sexpot turning out crap music videos. What scared me was that I remember seeing Ms. Lohan for the first time in 1998. The Parent Trap was one of Treanna and Alaric’s favorite videos for a long time, largely because of Lindsay’s performance. She was a spunky kid, and when Steph and I saw Mean Girls earlier this year, I thought she’d managed to hang on to that wholesome appeal while growing up and making the transition to being a young woman rather than a kid. But I was still thinking of her as “that kid.” She’s not — she grew up, with a vengeance, and now she’s definitely proven (to me at least) she’s not a kid anymore.

I’m just old enough — and just enough of a parent — to want to yell at her to put some clothes back on already, dammit! Because Treanna is almost 8, you see, and Alaric is 6, and it wasn’t all that long ago they were watching Lindsay’s debut. They are growing up, fast. Treanna’s going to become one of those hotties overnight; Alaric is going to be noticing the hotties (and getting some notice back from them) all too soon. If I’m doing my job, they’ll have good heads on their shoulders and will be better prepared than most of their peers, but they’re still going to get to a point where I’m cheering from the sidelines.

Bah. I should go to bed.

Oh, yeah — hot damn, Boston, what got into you? As long as it wasn’t the Yankees, I suppose it’s okay (but don’t tell the guys at work I said that, because I’m going to get really sick of their boasting about the Sox over the next year. Imigrants to the Pacific Northwest are okay until their hometown teams win; then they become annoying), but give a guy some warning, willya?

One of many

So why did I decide I needed my own blog? How come LiveJournal wasn’t enough?



  • Well, I’m not terribly keen on LiveJournal’s interface, to tell the truth. I like having categories, since I write about a lot of different things. I like that you, the reader, have the choice to only read certain posts that interest you instead of the boring ones too.

  • I like having the server bits to play with. When my stuff breaks, I usually know how to fix it (or choose not to fix it for a time). When LJ breaks or gets clogged, I’m hostage.

  • I’m somewhat vain. I think I’m interesting enough that people will keep reading, especially once I get the XML syndication working so that my LJ friends don’t miss a thing.

  • I already am scattered across multiple blogs, and I like the trackback feature that real blogs offer.

So, what other blogs am I on?



I’m almost ashamed to point to these blogs because I post to them so infrequently, but perhaps I can harness that shame into actually using them and posting to them. However, I just posted something new to my 3Sharp blog, so go read that.