On messaging and distributed system design

It's been 2 months since I last wrote a post. I moved out of my turn-of-the-century Victorian house quite abruptly about then because we discovered that my little boy had elevated lead levels in his blood. We were on the road staying with family and friends until last week, when we closed on our new home and moved in. With all that behind us and Gus getting treatment, it's finally time to start again. I can't think of a better way to jump back in than to respond to this recent post from John about advice on distributed system design. He was rebutting Richard Turner's MSDN article, which offered the official MS advice on the same topic. I figured I'd weigh in with a bunch of thoughts in no particular order...

First, I think Richard has a tough job. He has to rationalize current and future distributed technologies in a way that makes all of them make sense. This is part of his job at MS, which wants to present a unified view of the platform it offers for building distributed apps. It isn't a fun job, because different people want certain technologies positioned certain ways. With more and more of these technologies in the hands of the Indigo team, this should in theory get simpler. Of course BizTalk and the SQL technologies are still separate, so that keeps things interesting. Mary Kirtland did a similar job trying to position all the distributed technologies in .NET 1.0 (and COM+, MSMQ, DCOM); I felt sympathy for her too.

As one comment on John's post observes, in a lot of ways the focus on particular implementation technology isn't as important as the design considerations around message payload, asynchrony, message exchange pattern, state management, programming model, etc. These things are often conflated, which is a mistake. In general, I encounter people in one of two camps. There are people who want the world to look like method calls in their native programming language. In this world, everything is request/response and XML message bodies are completely hidden. There are other people who want everything to be truly (by which I mean disconnected, not just off-thread) asynchronous and XML message bodies are processed natively as documents. The latter group generally spend a lot of time trying to get the former group to see the world their way. Both groups are being to extreme.

John comments that if you're programming model requires a proxy, you're being forced into distributed object semantics. But the WSE messaging API includes SoapSender and SoapClient, which feel like proxies to me. Okay, they are generic proxies that aren't typed according to your interface (yes, you can use SoapClient that way too), but they are proxies. What I think he's really say is that if you use a proxy that marshals to/from your native object model, then you're headed for down the distributed object path. But it's more complicated than that, because you can actually mix access to objects and raw XML in a single proxy or service if you want to. We should also throw in the issue of instance lifetime management, which some would argue is a requirement for distributed objects, and the lack of it suggests that that's not what we're really doing.

The point I'm trying to make is that we really need to be more precise about what we think is a good or bad thing to do, independent of the technology used for implementation. This is hard though, because the answer about whether something is good or bad is typically “it depends.“ Should you use OX mapping? Should all messages be sent asynchronously via queues? Should asynchrony be layered in at the application level via synchronous invocations? I can think of cases where the answers to all these and many more questions are both yes and no.

So having made the observation that we're thinking too much about specific technologies and not about the actual architectural issues, let me take my turn on the specific technologies too. ;-)

My last post was about Biztalk and how appealing it seems compared to Indigo because XML is it's native language. Steve Swartz from Indigo responded that they are trying really hard to make working with raw XML in Indigo as easy as possible. I've been feeling bad about my post, because, I have to say that they have in fact included every feature I've ever requested to make working with XML in Indigo as simple as possible. The real issue is that it isn't typically presented that way. Biztalk, on the other hand, is presented that way and gets a lot of traction (at least with some folks) as a result. What I really want is a seamless experience across all these products. Specifically, I want to see three pieces:

1) XML dev tools integrated into the IDE - basically the stuff in Whidbey plus the BTS tools

2) A basic declarative XML pipeline for validation and transformation in Indigo

3) Seemless integration with the BTS orchestration engine when I need it.

I need 1 and 2 a lot more than I need 3 right now, and I don't want to have to go all the way to BTS just to get them.

With those pieces in place, I'm pretty confident that you'll be able to build whatever architecture you like, whether anyone else likes it or not.