Jim Johnson's Blog

Plumbing 101: A bookshelf list

2008-11-05T13:46:00Z

As I mentioned in a previous post, I'm essentially a plumber. I work on infrastructure software, and have focused mostly on how to hook things together to get work done reliably. I'm a reasonably literate plumber, though. I've worked, with moderate success, to stay up to date. I personally think that this is key to being successful in the sorts of roles that I've had.

Like pretty much everyone else that I know, I've built up a reading list over time. This varies from foundational books through to insightful and subversive papers. So, let's imagine that you're starting on a team that builds transactional infrastructure. What should be on your reading list?

First, there are the journeyman books that you probably already know about. They'll tell you about the tools you have, and how to best use them to produce a product. Aside from the your-favorite-language books, you'd undoubtedly want a couple like Code Complete and Writing Secure Code.

Next, you'll need at least one of the foundational texts. My personal favorite is Gray & Reuter, Transaction Processing. It's an imposing black book with a picture of the Rosetta Stone on the front. Frankly, it weighs almost as much as the Rosetta Stone. However, it is the first real 'how to' book for building a TP system. It's almost 15 years old and still highly relevant. I'd also recommend getting Weikum & Vossen, Transactional Information Systems. It is also a truly excellent how to book, and newer.

[A brief aside: I started working on transactions some years before these came out, at a time when transaction management was a pretty active area of research. Consequently, I got into the habit of reading the journals, and saving the ones that had some key insight. I suspect I was typical. I had my stack of papers and notes that I referred back to as needed. It was all too reminiscent of Dr. Jones' grail diary. When Gray & Reuter came out, much of the need to build your own grail diary disappeared. Man did that make lists like this much simpler to construct!]

With these texts in hand, there are two old papers, both by Butler Lampson, as it happens, that I'd recommend as part of any basic library:

Lampson, "Atomic Transactions", Distributed Systems-Architecture and Implementation. This is a real gem of a paper. It derives the two phase commit from a world where anything can fail, which is why I first read it. Along the way gives you a great way of thinking about the edges of any design, which is why I recommend it.
Lampson, "Hints for Computer System Design". This is a paper that anyone building system software simply must read. And anyone building commercial software should read.

Then there are a couple of more recent items that you should understand. These are:

Lamport, "Paxos Made Simple". If you're working on consensus protocols these days you should understand Paxos, and this is straightforward introduction to the topic. You can also pick up Gray & Lamport, "Consensus on Transaction Commit", which relates Paxos and 2PC.

Brewer, "Towards Robust Distributed Systems". These are slides from the talk that described the "CAP conjecture" -- the inevitable tradeoff between consistency, availability, and tolerance to partitioning. It's something that I think (hope) was reasonably widely recognized, but not clearly stated before this.

Finally, there are a couple of papers that I tend to recommend because they talk about ideas that are a little different. They're good ideas, and they helped me flex my imagination when I read them:

O'Neil, "The Escrow Transaction Method". A clear example of where you can get to if you don't assume that locking implies CRUD.
Haskin, et al, "Recovery Management in QuickSilver". A pretty old paper, but one that asks what if transactions could be baked really deeply into your system.

Finally, these days, pretty much anything by Tim Harris on transactional memory.

Jim.

I'm late -- the PDC was last week

2008-11-02T16:31:00Z

I had meant to file something before PDC, but never quite got around to it.

Our group had a few people at the event, and had a presentation based on what we've been doing. If you didn’t see it, please go check out the channel 9 video of Max Feingold's talk on building for scale.

We've been building some base scaling and messaging infrastructure for the .Net services in Azure. Max's talk describes some of that technology, and also talks about design considerations that we've seen to work well for scaling -- both with our infrastructure and in general. I hope that you'll find it interesting.

Please take a look at the talk.

Upcoming webcasts

2008-07-03T21:04:00Z

There are some WCF-related webcasts coming up during July, with monday's one on transactional WCF support. You can find links to them on the MSDN Webcasts site. This month's talks are:

Webcast Title	Date/ Time	Presenter
Transactional Windows Communication Foundation Services with Juval Lowy	7/7/2008 10:00 AM PST	Juval Lowy
Using Windows Workflow Foundation to Build Services with Jon Flanders	7/9/2008 10:00 AM PST	Jon Flanders
WCF Extensibility Deep Dive with Jesus Rodriguez	7/11/2008 10:00 AM PST	Jesus Rodriguez
Bringing Enterprise Data to Life with SharePoint Server and Windows Communication Foundation	7/18/2008 10:00 AM PST	Joe Klug

Update (July 14, 2008): I've updated the links to the talks.

Jim Gray

2008-06-01T01:30:00Z

There was a tribute earlier today for Jim Gray at Berkley. I want to share some thoughts about Jim.

I have had the privilege to know a lot of great engineers, a number of scarily bright people, a smaller number of truly wise people, and some solidly good people. And very, very few that are all of these. Jim is one.

I have met Jim maybe 2-3 dozen times over the last 15 years, and I've exchanged a similar number of emails with him. But his effect on me, and on my career, was far larger than those numbers would suggest. He has provided guidance and advice, to which I am seriously indebted. His only charge was that I pay it forward when I could. And I try.

If it isn't clear from my other posts, I am basically a plumber. I build piping, and worry about how to hook stuff together so that it works, and so that it goes reasonably fast (or so I hope). And, I'd have to say that I wasn't really a professional at this game until I'd put 5 years or so under my belt. Indirectly, Jim was partially responsible for that transition.

My first exposure to Jim was the same as a lot of people's, I'd imagine. It was through his writing. At the time I was a young developer working in the VMS OS group working on its distributed cache management layer (inside RMS). Steve Beckhardt gave me a copy of a paper and told me to read it -- that it would explain what the purpose of the lock manager was. The paper was "Notes on Database Operating Systems" by Gray, et al. It was stunning -- it was clearly an academic paper, but… it was written so that it spoke to plumbers like me.

Eventually I met Jim in person. Others have written much more eloquently than I could about how Jim can immediately connect with a person. All I can say is that was my experience. He was inspirational, helpful, and seemed honestly pleased to meet me. Over the following years I've met Jim occasionally -- sometimes years would pass between times. And every time we picked up from where we left off. I've always been amazed that he could keep track of what I'd been up to. And every time we talked I came away the better for it.

On the day that Jim was reported missing I came into the office with a question for him. I was working on the email when I heard the news. I still haven't asked my question, and I miss his counsel.

Jim.

PS: While writing this I came across Pat's post. As usual, Pat is very eloquent. I can only recommend that you read it.

MSDTC changes in WS2008, Part 4: Miscellaneous features

2008-04-07T00:02:00Z

Parts 1, 2, and 3 covered most of the mainline MSDTC features that show up in WS2008. And now we come to the end with three more, relatively minor, topics: VSS integration, changes in XA support, and some changes in MSDTC tracing.

VSS Integration

This isn't strictly speaking a WS2008 feature, as it was mostly present in Vista. However, it is a good example of a 'good Windows citizen' change.

Traditionally, transaction managers have had little to do with backup. They manage a log that is assumed to hold very short lived data. In turn, that data only makes sense in conjunction with other data stores (i.e. either other transaction manager or resource manager logs). And, there is normally no automated way to handle the multiple data stores being out of sync. MSDTC is no different in any of these regards, and all of these reasons are still valid with Vista.

However, there is a role that a transaction manager can play. Consider taking a live backup of a volume where you have a number of transaction aware resources. The ideal backup is no longer a crash consistent point in time view of the data. It is a crash consistent point in time where all the active resource managers will recover their transactions in the same way on restore -- in other words, a transaction consistent point in time.

VSS, the Kernel Transaction Manager, and MSDTC were changed during Vista to support this mode. VSS now asks the transaction managers at the appropriate times to establish a transaction consistent point. This is one where all impacted resources that have received a phase 2 (commit) notification have acknowledged processing it. Once that has been reached, all the resources are directed to take their snapshots, and then the transaction manager is released to continue processing.

Each resource manager, when recovering from a VSS restore, will automatically abort all transactions that were in flight or in doubt. This means that they will all restore to the same logical point in time, given the volume a transaction consistent restore.

XA Support

In part 3, I described how MSDTC now automatically reconnects to the correct MSDTC instance during recovery for any given transaction. By default, that feature also works for XA-based resource managers.

In order for MSDTC to do this, it had to add some data to both the PrepareInfo structure and to the XA XID. The PrepareInfo structure was already variable length, but the XA XID was packed into a format that had typically been more static in its size. Normally, there is no problem with the new fields. However, the format and size of these structures did change, and that could cause some resource managers difficulty.

Because of this, as of Vista SP1 and Windows Server 2008, MSDTC supports a degraded mode of operation which restricts a resource manager to both the pre-Vista recovery data formats and to working with, and only with, the default cluster resource MSDTC instance.

In order to select this mode, the resource manager needs to call DtcGetTransactionManagerEx as follows:

hr = DtcGetTransactionManagerEx (

NULL, NULL, riid,

OLE_TM_FLAG_NOAGILERECOVERY, // 0x00000002

NULL, &pvRiidObject);

As you can see, this requires that the default be selected, and the flag will force this to use the default cluster resource instance.

Changes in tracing

MSDTC has long had both the administrative transaction tracing and a growing set of fault analysis trace features. The latter are increasingly useful in tracking down issues quickly and resolving them. During Vista and Windows Server 2008, the transactions team worked on rationalizing the tracing, extending it to cover more of our own problem resolution scenarios, and incorporating suggestions and feedback from PSS. From what I saw, we ended up with much more extensive tracing than we'd previously had, with a result that we were able to debug problems during development much more quickly than in the past.

While they are mostly meant for internal consumption, they might be interesting for some intrepid spelunker. I was going to provide a rundown of the traces, but I ran across http://support.microsoft.com/default.aspx/kb/926099, which already does that, and better.

And that brings my lap around the changes in MSDTC to a close. It has some very significant new features, ones that I believe and hope will prove useful to many customers.

Jim.

MSDTC changes in WS2008, Part 3: Cluster features

2008-03-23T17:52:00Z

As I mentioned in parts 1 & 2, WS2008 clusters are designed to have more than one functioning MSDTC instance. Once the DtcGetTransactionManager[Ex] had been upgraded to connect to multiple MSDTC instances simultaneously, the issues next focused on configuration and management.

MSDTC Configuration

An MSDTC instance is defined by the information to establish an RPC connection, a log file, and some security and accessibility information. The connection information consists of a NetBIOS name and four GUIDs, also known as connection IDs, or CIDs. In a non-clustered system, the NetBIOS name is the one used by the system, and the rest of the information is located in the local registry. The CIDs and log file information can be found under HKLM\Software\Classes\CID, and the pretty much the rest can be found under HKLM\Software\Microsoft\MSDTC.

These settings represent three roles: what does the MSDTC service need to configure itself, what does the system provide, and what does the MSDTC proxy need to contact the MSDTC service. On a single node system it can be hard to tell these apart.

But then you look at a pre-WS2008 cluster and you'll see that some of the settings, specifically the CIDs, the security settings, and the log location, have been migrated into the cluster registry hive (the name is calculated, so there isn't a simple path to point out). Information pertaining to the capabilities of an individual system, such as tracing, XA DLL registration, and version information remain in the various local hives. The MSDTC proxy understands when it is in a cluster, so that information is not a clearly derived.

At one level, the WS2008 changes amounted to supporting multiple entries in the cluster registry, each of which is attached to a different cluster resource. For backwards compatibility, one of the MSDTC cluster instances is designated the default cluster instance, and uses the cluster alias as the NetBIOS name.

Add a command line feature to start an MSDTC service for a specific configuration, and we're done.

Picking 'the right MSDTC'

There's a big problem with the solution so far. If the application or resource manager specifies the MSDTC instance in DtcGetTransactionManager[Ex], then the correct MSDTC instance will be used, and you'll get the benefits of from being able to spread the traffic around, from the instance always being local to your resource, and so forth.

Unfortunately, almost no one specifies the MSDTC instance to use. This is partially due to the assumption that there was just one, hardened by the MSDTC proxy behavior to bind to the first MSDTC instance specified in the process. Figuring out how to select a 'good' MSDTC instance was the next big hurdle.

A good MSDTC is one that isn't overly remote from the caller, that has the same availability characteristics, or better, of the caller, and one that doesn't add additional MSDTC instances for no good reason. Oh, and did I mention that existing programs need to continue to work?

This then produced the following rules:

Any caller that does not specify whether it is a local application or a cluster resource uses the default cluster instance.
If the caller is specified as a local application, the local MSDTC instance, if available, is used. Otherwise the cluster default instance is used.
If the caller is specified as aligned with a specific cluster resource:
- If there is an MSDTC instance that is part of that cluster resource, it will be used.
- If not, and there is a cluster default MSDTC instance available, it will be used.
- If there are no cluster instances available, then a local MSDTC instance will be used with a notification of a potential loss of availability.

A caller can be identified in two ways: programmatically or via configuration. For the first, the DtcGetTransactionManagerEx call has been extended. The call has the signature:

EXTERN_C EXPORTAPI __cdecl DtcGetTransactionManagerEx (

tchar * pszHost, // NetBIOS name of the MSDTC instance, or null

tchar *pszTmName, // must be null

REFIID riid, // IID for the interface to retrieve

DWORD grfOptions, // Configuration options

void * pvConfigParams,// Configuration parameters

void **ppvObject // returned object instance

);

The structure passed in pvConfigParams is what has been extended to now look like:

typedef struct _OLE_TM_CONFIG_PARAMS_V2

{

DWORD dwVersion; // Must be set to 2

DWORD dwConcurrencyHint; // Set to 0

APPLICATIONTYPE applicationType; // Specifies alignment to local or not

GUID clusterResourceId; // If not local, align to this resource

} OLE_TM_CONFIG_PARAMS_V2;

typedef enum APPLICATIONTYPE

{

LOCAL_APPLICATIONTYPE,

CLUSTERRESOURCE_APPLICATIONTYPE

};

Note that the call to DtcGetTransactionManagerEx can specify the specific MSDTC instance to use. The new configuration parameter settings allow for a different approach. Rather than having the caller specify the MSDTC instance to use, by name, these new settings allow a caller to make statements about itself that the MSDTC proxy will use to select a reasonable MSDTC instance. The assumption is that statements that a caller makes about itself are less likely to change due to a configuration change in the cluster, and leaves more leeway for later optimization algorithms -- such as continuing to use an existing cluster MSDTC instance if it is already in use for that transaction.

It is also important to realize that in the case of the CLUSTERRESOURCE_APPLICATIONTYPE what the caller is specifying is the preferred cluster resource to align to. In the case of a cluster resource this is most likely to be itself. In the case of, say, a node-local COM+ component, this could well be the cluster resource that the SQL Server that it uses is in.

The second way that a caller can be identified is via external configuration settings known as TM mappings. The configuration specifies executables, COM+ application IDs, and cluster resource IDs that are to be mapped to specific MSDTC instances. These settings are used when the caller to DtcGetTransactionManagerEx supplies neither the name of an MSDTC instance or a OLE_TM_CONFIG_PARAMS_V2 structure.

The configuration settings can be found under HKLM\Cluster\MSDTC\TMMappings. The values can be set, viewed, and deleted via options on msdtc.exe, as follows:

To set up a mapping, use
msdtc.exe -tmMappingSet -name

[-exe <pathname> | -service <full service name> | -complusappID <app ID>]

[-local|-clusterResourceName <resource name>]

To view a mapping, use
msdtc.exe -tmMappingView -name
To view all known mappings, use
msdtc.exe -tmMappingView *
To delete a mapping, use
msdtc.exe -tmMappingClear -name
To delete all known mappings, use
msdtc.exe -tmMappingClear *

For instance, let's say that I have a application "JoesGarageShopApp.exe" that only works against local files and only on a single node. I'd like that application to only use the local MSDTC instance, whichever that one is. I'd do that with:

msdtc.exe -tmMappingSet -name JoesGarageShopMapping -exe JoesGarageShopApp.exe -local

What happens to resource managers?

From the point of view of the resource manager, effectively nothing. They make the calls to the MSDTC proxy, and receive notifications in the way that they always have. There are two significant changes, but they are done transparently behind the MSDTC proxy interface.

The first is that a transaction passed into a resource manager from an application may be using a different MSDTC instance than the one that the resource manager wishes to use. If that is the case, the MSDTC proxy will emulate the operations to pull the transaction to the new MSDTC instance (see http://blogs.msdn.com/florinlazar/archive/2004/10/02/236965.aspx for a good description of how this normally works).

The second change is that pPrepInfo presented at IResourceManager::Reenlist may specify a different MSDTC instance than the one that is currently being used. In that case, the MSDTC proxy, as part of handling the Reenlist request, will internally issue the correct DtcGetTransactionManagerEx call to connect to that MSDTC instance as well.

Kernel Transaction Support

As something of a detail point, there is some underlying plumbing in place to handle failover of transacted NTFS and transacted registry resource managers. The overall logic is the same as I've talked about above. The one difference is that the kernel interfaces are not themselves remotable, so there is an agent service (the KtmRm service) that acts as the intermediary between the MSDTC instance and the kernel transaction manager.

Jun 1, 2008: Updated to fix a bug in the msdtc.exe command line syntax.

MSDTC changes in WS2008, Part 2: Cluster requirements

2008-03-16T16:24:00Z

As a preamble, I have to point out that most customers have little to no trouble with MSDTC in a cluster. It has been a great solution for those that need transactional consistency and high availability. Nothing in this description should suggest otherwise.

However, one fact of life is that Windows shows the power of large numbers. We have vastly more users than I've encountered in any previous company. Because of that, you can see trends and identify opportunities for improvement much more easily than I've seen in the past.

The clustering changes came about as a response to a number of issues that we'd been seeing with MSDTC in a cluster. They basically fell into three categories:

Complexity of setting up a cluster. Setting up a cluster with MSDTC was complex -- it involved a number of steps, those steps had a fragility in how they related to setting up other cluster resources, such as SQL Server, and they required that the system actually degrade in capability as an initial step.

To put it another way, setting up a cluster first disabled the MSDTC instances you did have, then you had to know to create a cluster MSDTC instance, and then you could add in new transaction-aware cluster resources (e.g. SQL Server). The fewer steps and the fewer ordering dependencies that we could get to, the better.
Asymmetric performance patterns. Once the cluster was correctly set up, there was a second surprise. As the cluster lived through node failures, and depending on the application design, the performance of e.g. SQL Server or MSMQ could change significantly from one failover point to the next.

We all know that full distributed transactions have overhead. In some cases, quite considerable overhead. That's why so much of the industry's work is and has been around optimizations. In the case of MSDTC, the performance when MSDTC is on the same node as the application and / or resource managers is higher than when it is on its own node.

Given that there is one MSDTC cluster instance, any reasonably complex configuration will have SQL or MSMQ instances that move from being remote to that MSDTC instance to being on the same system as MSDTC, and then back. As that happens the performance profile for their interactions with MSDTC change significantly.

This means that you should do your performance planning against the case when MSDTC is located on another node from the application and resource managers.
Migration of nodes into and out of a cluster. Note above that the local MSDTC is disabled upon entry into a cluster, and the MSDTC cluster instance replaces the role of that local MSDTC. This has a direct implication that when a node joins a cluster it needs to be fully recovered, and the same is true when it permanently leaves.

Frequently this isn't a problem when joining, if you assume that nodes that are put into a cluster are pretty much idle nodes anyway. It is only in more dynamic load management cases that this one could be a concern.

At the root of each of these points was a longstanding design decision to have only one active MSDTC instance in a cluster -- and it had to be a cluster resource. This was the cluster representation of the assumption that topologies were pretty much fixed: that a resource manager would be in a configuration where they were always using the same transaction manager, and likewise for the applications.

And to be clear, this wasn't a bad assumption for how systems have actually worked. It reflected how systems were configured, how COM+ was configured, and how the hardware tended to be deployed.

Beginning around .Net 2.0, though, we began to explore ways to remove or reduce this assumption. This was driven by an assumption that we'll be seeing increased mobility of applications, and a desire to be able to isolate one application, or one resource, from another.

First, we had to deal with the assumption as present in the MSDTC proxy. It would bind to the MSDTC instance specified by the first call in that process. Until the last few years, you've not been able to find out which one that was. Now you can for any version that supports the ITmNodeName interface.

Next, in Vista we changed the proxy further to bind to multiple MSDTC instances, once per unique MSDTC name specified. With this in place, you could change MSDTC instances on a per transaction basis, both in normal operation and during recovery.

This was a sufficient basis to then begin working through the various design issues to handle multiple MSDTC instances in a cluster. Interestingly, most came from configuration choices or management operations. On the other hand, the protocol had sufficiently abstracted the idea of an MSDTC instance that I can't remember any protocol changes that were needed specifically for this, although I'll admit that I've not checked recently.

The design issues that I remember being at the top were:

What is a 'good' MSDTC instance to pick? If I take an application or resource running today, it won't specify which MSDTC instance to use. I don't want to just use a cluster-wide default, or I'll get largely what I get today -- one cluster instance handling everything. So, how should a particular instance be picked?
What does the cluster default MSDTC instance mean? Is it anything more than an instance that happens to have the same name as the cluster alias, or is it deeper than that?
What configuration needs to be consistent across all MSDTC instances, and what should be specific to a given instance? Also, much more mundanely, where should all this configuration data go?
What are the management changes to support these configurations? What are the command line changes, and what are the MMC plugin changes?

I'll pick up with how we approached these questions in part 3.

Jim.

MSDTC changes in WS2008, Part 1: Introduction

2008-03-15T20:44:00Z

Windows Server 2008 has a number of significant changes to MSDTC, addressing issues from performance, simplicity, and reach. These changes are the culmination of a stream of work that began with Vista, and was heavily influenced by .Net from 2.0 onwards.

The changes in WS2008 had the following goals:

Support the use of transacted NTFS and transacted Registry operations.
Increase flexibility of deployment, predictability of performance in a Windows cluster.
Improve the diagnostics available for troubleshooting.
Be a 'good Windows citizen'.

Much of the visible changes to support transacted NTFS and registry operations showed up in Vista. The changes in WS2008 are largely to support those features in a cluster. I'll talk about them in passing as part of the overall cluster changes.

The biggest set of changes, by far, had to do with the cluster support. This is what I'm going to focus on in parts 2 & 3.

Through WS2003, MSDTC could only have a single instance on a cluster. This paralleled the single-instance-per-node assumption that holds for non-clustered systems. However, it meant that entering a cluster meant that the local MSDTC was disabled, thus preventing it from completing any outstanding recovery. It also meant that you could be very asymmetric performance patterns depending on where failovers had taken the recovery resource, such as a SQL server instance, and the MSDTC instance. As you can imagine, you'd get much better performance when they happened to be on the same system vs. when they weren't.

In WS2008 the underlying assumption has been removed. You can have any number of MSDTC cluster resources, they can be affinitized to other cluster resources (so that you can keep the transaction on the same system, regardless of failover patterns), you can retain the use of the local (per-node) MSDTC instances for those cases where the application is not cluster-aware, and you can split a cluster safely without requiring that all transactions be resolved.

Finally, there were two sets of changes that we made that should be considered largely internal. One was a set of changes to unify and extend the advanced diagnostic tracing features that PSS would typically use when resolving a customer problem. The other was a set of changes related to the quality and feature expectations for any component going into Windows. Although neither should directly impact any customers, I'll touch on those in part 4.

Jim.

Building the transaction protocol documents

2008-02-28T18:49:00Z

You've undoubtedly read about the announcement last week to publish our protocol documents on msdn. If you've gone and looked, there's an extensive set of documents posted.

You may also remember that I mentioned that I'd been involved in some projects that had taken up quite a bit of my time over the past 18 months. One of those projects involved some of those documents -- the MSDTC protocols [MS-CMPO], [MS-CMP], [MS-DTCO], and others. If you take a look at these documents, you'll find that they describe a fairly complex set of messages, state machines, and inter-component interactions. Constructing this documentation was a pretty interesting challenge, and one that I thought might interest you. What follows are my personal observations about what we did to produce these transaction documents.

Much like the product code, the transaction feature team was responsible for the construction of these documents. We were the experts on the code, and therefore made the best sense for understanding what the content of the documents needed to be. There were ancillary groups that provided editing, review, and formatting support.

One of the first big tasks we faced was how to describe 'the protocol'. Like all transaction managers, we had a set of interlocking protocols and state machines that arbitrated operations between applications, resource manager, and other transaction managers. Each arm typically has one or more 'protocols', but their operation is dependent on the aggregate state across them.

Or, to put it another way, if I only look at, say, a resource manager to transaction manager exchange in isolation I'll find nondeterministic behavior around whether or not a transaction enlist request will succeed. What makes it nondeterministic is that the failure is a result of the aggregate state the transaction is in, which is effectively invisible to our particular exchange (for instance, the app may have already requested that the transaction commit, and the transaction manager may have already issued request to prepare messages to the other participants).

This is a problem that the various standards have faced. In my experience, none have dealt with it terribly well. WS-AtomicTransactions takes a stab in the state tables at identifying the existence of this aggregate state, but is pretty terse on how it works. XA is a bit more verbose, but it is also pretty hard to work out the aggregate state machine from its various tables. Frankly, we had to find a way to do better.

Before I go on, please do note that this is not a criticism of these standards -- it's more a side effect of the inherent tradeoffs a standards org has to make in the face of multiple vendors with differing implementations.

The second big task was to then determine exactly how the protocols worked. What we needed to ensure was that the documents reflected the actuality of how the protocol was implemented, across the range of Windows versions.

There's pretty obviously only one solution -- go to the code. The dev team writes the support fixes for MSDTC, so it had current experience with the various code bases. That helped, inasmuch as it made the archeology effort more efficient. Of course, most of the changes were additions. The protocols are pretty much upwardly compatible, after all. But still, over the years there have been quite a few changes.

At this point the team had an outline for the documents, and a large amount of data that it had accumulated. The actual authoring and revising then began, and were run essentially like any other project. There was a coordinator that assigned out tasks, monitored progress, and kept an eye on how the docs were shaping up. As we learned details, he oversaw ensuring that this information was quickly shared with the rest of the team. This allowed us to produce the documents on the aggressive time schedule that were required for delivery.

One special aside is that when you look at the documents you'll note that section 4 contains examples of message exchanges. What we did was to assign section 4 to one of our senior testers. He took the draft documents, constructed specialized test programs that spoke the protocols, and captured the actual message contents in the samples. This turned out to be an effective early QA test of the document, and found a couple of product bugs as well.

This was an intense period of activity. I'm happy with the technical content of the transaction documents. I’m also proud of how the team came together to produce them. But you have to ask, who might want to use it? I can't speak for anyone here, but I know folks that could have used it to construct transaction bridges to OpenVMS, and I could imagine one or more of the J2EE vendors doing so today. Only time will tell.

Jim.

The end of an era

2007-12-31T18:41:00Z

I read a couple of weeks ago that HP is closing ZKO (Spitbrook Road, Nashua). I have a lot of fond memories of working in VMS development in ZKO1/1 (and later ZKO3/4) in the 80's and early 90's. It's sad to see that it is going.

Of course I miss the people -- too many to try to count. It was a great place to work, and I'm proud of the work that we did back then. VMS was an excellent product for its time, and portions of it compare favorably to anything being done today.

I miss the DEC traditions around April 1st, especially those that Andy had invested in. In ZKO, the decision to name conference rooms after people I'd heard of, and the quotes they picked -- some of which I use to this day (e.g. "anything not worth doing is not worth doing well").

I also wonder what will happen to the nonpaged pool behind the cafeteria, or the 7-bit ASCII colored bars at the main entrance (that used to say "DigitalSoftwa Reengineering" due to an unfortunate line break).

And, yes, I enjoyed DEC, but despaired of Digital.

Finally, one small personal tidbit: I remember giving a lecture on 'what is this 2PC thing anyway' to some VMS devs around 1990. The thing that makes it memorable was that it was in the [Nancy] Lynch conference room. The quotes on the walls were at least as good as anything in my slides :).

Where has Jim been?

2007-12-31T18:39:00Z

I have clearly been away for a while. This blog effectively shut down about 18 months ago. For the last few months I've been trying to figure out if I should reopen it, and I set a date of Jan 1 to make that decision. I've decided to attempt to return to writing entries in this blog.

So what happened?

At work, the simple answer is that I got extremely busy. There was a 12-15 month period where I had a series of projects with very short term deadlines. Almost none of them involved my normal day job (which also pretty much stopped for that same period).

At one level it was a pretty interesting period -- I got involved with teams and people that I normally wouldn't have, and was exposed to technologies and groups that I'd previously been fairly ignorant of. I always enjoy the opportunities to be exposed to an learn new things. However, it did come with the cost that it was extremely time consuming and very exhausting. And I'm glad that it mostly seems to be over for now.

About 10 months ago the biggest time sinks came to an end, and I returned to thinking about the transactions directions and what we could be doing next. Because so much time had passed, we were now in a rush to get our planning in shape for the next round of releases. I spent the next few months getting a set of a feature ideas birthed for that next round. That was as intense a period as any that previous year. The team is now off making them real, and I believe that they'll prove to be exciting when they ship.

Then, having gotten the work moving, I took stock of where I was. I realized that I needed a change -- that while I loved the technology and the team, the excitement was ebbing. I needed to push my envelope again and try something that would stretch me.

So, about six months ago I changed teams and took a position with the team that does messaging infrastructure -- from the WCF channels through to our enterprise messaging work (which includes MSMQ). It is a very different role for me in many ways: it is more breadth based, while I've been mostly a depth-first architect. It involves technologies that I've been aware of, but don't have the same level of involvement that I've had with transactions. And it is with different people, although, again, it is with folks that have been nearby in the past.

I can't say that life has gotten slower. I'm still on a learning curve. I do, however, think that I understand the slope more than I did when I started. I think I'll be able to find time for this blog now. We'll have to see.

What does this mean for this blog?

I intend to continue writing here about transactions. It is still a long standing passion of mine. However, you should expect to see a somewhat broader mix of postings. Some may even be about messaging…

I'll try to make sure that I mix in articles with code (since those seem to get the highest readership ;)), but I've never posted purely coding articles, and won't start now.

Oh, and, time willing, I will return to the resource manager project series.

Vista: An MSDTC proxy enhancement (MLTM)

2006-09-01T17:47:00Z

MSDTC has had a structure where there is a proxy (msdtcprx.dll) located in the user's process. This implemented the Oletx API, managed some local state, and a set of communication channels back to the MSDTC service (msdtc.exe). Much of the processing went on in the MSDTC service -- almost all of it, in fact.

Given the transaction topologies that MSDTC would be involved in, this had been a reasonable design choice. Transactional resources were located in server processes, so multiple processes, and quite often multiple systems, were involved. That meant that these transactions were going to migrate to the MSDTC service process anyway. The only thing that could be up for debate was whether or not the transaction went to the MSDTC service when it was created, or a millisecond or so later.

That began to change with System.Transactions. The amount of logic in the user process went up, the capability went up, and new mechanisms (such as the promotable single phase enlistment) were introduced that kept the transaction local to the process, even if it involved a database. Even if the transaction should promote to Oletx, System.Transactions would continue to do what it could to limit the amount of information that would flow back to the MSDTC service. For instance, volatile enlistments and transaction outcome notifications are multiplexed in the user process, and may not be represented back at the MSDTC service at all.

With Vista, we also had some new topologies to consider, most especially ones that involved the transactional file system (TxF) or registry (TxR). In those cases the transacted resource is not in another process. More importantly, there is no failure mode where TxF or TxR can fail catastrophically and the user process not be impacted.

Consequently, we've added logic in the MSDTC proxy to be able to create transactions, multiplex voters and outcomes, and manage TxF or TxR resources without going to the MSDTC service. Much like System.Transactions, the proxy will automatically promote to the MSDTC service if the topology requires it. This feature is internally called "MLTM".

Consider the implications of this. If you look at the example code shown here, you'll note that System.Transactions acquires the DTC ITransaction interface in order to pick up the kernel transaction handle. Prior to Vista, this would have promoted the transaction to the MSDTC service, but no longer. That example should never leave the user process, never add any additional log writes, and only involve the minimal transaction mechanisms in order to execute.

A Vista update: integration between System.Transactions and the transacted file system and registry

2006-08-31T18:03:00Z

I have written earlier (here and here) about how to integrate the transactional file and registry services in Vista with System.Transactions. With the upcoming Vista RC update, there are a number of changes in the APIs for TxF and TxR that impact how to do this integration.

First, TxF and TxR have changed from APIs that automatically picked up the ambient kernel transaction and joined it, to one where the developer needs to explicitly show an intention to participate in the transaction. This is accomplished through the addition of several new APIs, such as CreateFileTransacted. These new APIs are explicitly transaction-aware. All existing file and registry APIs return to being explicitly transaction-unaware.

(For details on the new transaction-aware APIs, see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/win32_functions_changed_by_transactions.asp.)

All of the new APIs take a kernel transaction handle as an explicit parameter. This means that if I want to use these new routines from inside a System.Transactions TransactionScope I will need to do some work. That is what I want to concentrate on here in the form of a partial example.

Let's start by setting up a couple of helper classes and declarations.

First, we know that we'll be manipulating a kernel transaction handle, so we'll specialize a SafeHandle appropriately:

using System.Collections.Generic;
using System.Runtime.ConstrainedExecution;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using System.Security.Permissions;
using System.Text;

namespace TxFile
{
    [SecurityPermission(SecurityAction.LinkDemand,UnmanagedCode=true)]
    public sealed class SafeTransactionHandle: SafeHandleZeroOrMinusOneIsInvalid
    {

      private SafeTransactionHandle () : base (true)
      {
      }

      public SafeTransactionHandle (IntPtr preexistingHandle, bool ownsHandle)
       : base (ownsHandle)
      {
       SetHandle (preexistingHandle);
      }

      [DllImport("Kernel32.dll", SetLastError=true)]
      [ResourceExposure(ResourceScope.Machine)]
      [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
      private static extern bool CloseHandle (IntPtr handle);

      [ResourceExposure(ResourceScope.Machine)]
      [ResourceConsumption(ResourceScope.Machine)]
      override protected bool ReleaseHandle ()
      {
          return CloseHandle (handle);
      }
    }
}

We also need to declare the new CreateFileTransacted API. This looks a lot like CreateFile, but with a couple of extra parameters. The new declaration is:

[DllImport("Kernel32.Dll",
           CallingConvention=CallingConvention.StdCall,
           CharSet = CharSet.Unicode)]
internal static extern SafeFileHandle CreateFileTransacted(
    String lpFileName,
    int dwDesiredAccess,
    int dwShareMode,
    IntPtr lpSecurityAttributes,
    int dwCreationDisposition,
    int dwFlagsAndAttributes,
    SafeFileHandle hTemplateFile,
    SafeTransactionHandle txHandle,
    IntPtr miniVersion,
    IntPtr extendedOpenInformation
   );

Our final declaration is to define the new COM interface that DTC supports, IKernelTransaction. This interface is available off MSDTC's ITransaction objects. It's declaration looks like:

[ComImport]
[Guid("79427A2B-F895-40e0-BE79-B57DC82ED231")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IKernelTransaction
{
void GetHandle( out SafeTransactionHandle ktmHandle );
}

With these helpers and declarations out of the way, we can look at what to do in the mainline code to use CreateFileTransacted in a TransactionScope. We know that we need the transaction handle, that we can get that from IKernelTransaction, and we know that we can get that from a DTC ITransaction. That turns into:

SafeTransactionHandle txHandle;

IKernelTransaction kernelTx =
(IKernelTransaction) TransactionInterop.GetDtcTransaction
(Transaction.Current);
kernelTx.GetHandle (txHandle);

Putting it all together, we would get a TransactionScope pattern that looks like:

using (TransactionScope s = new TransactionScope ())
{
SafeTransactionHandle txHandle;

  IKernelTransaction kernelTx =
   (IKernelTransaction) TransactionInterop.GetDtcTransaction
    (Transaction.Current);
  kernelTx.GetHandle( txHandle );
  ...

// Do the Win32 file operation such as...
SafeFileHandle fileHandle = CreateFileTransacted(..., txHandle, null );

s.Complete ();
}

An article on 'scopes'

2006-08-25T17:34:00Z

Stephen Toub has written an interesting article on the construction of generic 'scope-like' classes over at .Net Matters. It captures a lot of the ideas that were behind TransactionScope and generalizes them nicely.

Jim.

Spamming -- ugh.

2006-08-25T17:30:00Z

You may have noticed that this blog is getting nailed by a spam storm. The great guys at Pluralsight are working on it, but in the meantime replies have been disabled to try to slow it down.

Hopefully things will be back to normal soon.

Jim.

UPDATE: Comments have been re-enabled.