Building the transaction protocol documents

You've undoubtedly read about the announcement last week to publish our protocol documents on msdn.  If you've gone and looked, there's an extensive set of documents posted.

You may also remember that I mentioned that I'd been involved in some projects that had taken up quite a bit of my time over the past 18 months.  One of those projects involved some of those documents -- the MSDTC protocols [MS-CMPO], [MS-CMP], [MS-DTCO], and others.  If you take a look at these documents, you'll find that they describe a fairly complex set of messages, state machines, and inter-component interactions.  Constructing this documentation was a pretty interesting challenge, and one that I thought might interest you.  What follows are my personal observations about what we did to produce these transaction documents.

Much like the product code, the transaction feature team was responsible for the construction of these documents.  We were the experts on the code, and therefore made the best sense for understanding what the content of the documents needed to be.  There were ancillary groups that provided editing, review, and formatting support.

One of the first big tasks we faced was how to describe 'the protocol'.  Like all transaction managers, we had a set of interlocking protocols and state machines that arbitrated operations between applications, resource manager, and other transaction managers.  Each arm typically has one or more 'protocols', but their operation is dependent on the aggregate state across them.

Or, to put it another way, if I only look at, say, a resource manager to transaction manager exchange in isolation I'll find nondeterministic behavior around whether or not a transaction enlist request will succeed.  What makes it nondeterministic is that the failure is a result of the aggregate state the transaction is in, which is effectively invisible to our particular exchange (for instance, the app may have already requested that the transaction commit, and the transaction manager may have already issued request to prepare messages to the other participants).

This is a problem that the various standards have faced.  In my experience, none have dealt with it terribly well.  WS-AtomicTransactions takes a stab in the state tables at identifying the existence of this aggregate state, but is pretty terse on how it works.  XA is a bit more verbose, but it is also pretty hard to work out the aggregate state machine from its various tables.  Frankly, we had to find a way to do better.

Before I go on, please do note that this is not a criticism of these standards -- it's more a side effect of the inherent tradeoffs a standards org has to make in the face of multiple vendors with differing implementations.

The second big task was to then determine exactly how the protocols worked.  What we needed to ensure was that the documents reflected the actuality of how the protocol was implemented, across the range of Windows versions.

There's pretty obviously only one solution -- go to the code.  The dev team writes the support fixes for MSDTC, so it had current experience with the various code bases.  That helped, inasmuch as it made the archeology effort more efficient.  Of course, most of the changes were additions.  The protocols are pretty much upwardly compatible, after all.  But still, over the years there have been quite a few changes.

At this point the team had an outline for the documents, and a large amount of data that it had accumulated.  The actual authoring and revising then began, and were run essentially like any other project.  There was a coordinator that assigned out tasks, monitored progress, and kept an eye on how the docs were shaping up.  As we learned details, he oversaw ensuring that this information was quickly shared with the rest of the team.  This allowed us to produce the documents on the aggressive time schedule that were required for delivery.

One special aside is that when you look at the documents you'll note that section 4 contains examples of message exchanges.  What we did was to assign section 4 to one of our senior testers.  He took the draft documents, constructed specialized test programs that spoke the protocols, and captured the actual message contents in the samples.  This turned out to be an effective early QA test of the document, and found a couple of product bugs as well.

This was an intense period of activity.  I'm happy with the technical content of the transaction documents.  I’m also proud of how the team came together to produce them.  But you have to ask, who might want to use it?  I can't speak for anyone here, but I know folks that could have used it to construct transaction bridges to OpenVMS, and I could imagine one or more of the J2EE vendors doing so today.  Only time will tell.

Jim.