The state of Web services, continued

There are a couple interesting comments about last night's post. Steve agreed that today's SOAP stacks are too focused on hiding XML behind objects (a point he and I have agreed on before). He pointed out that one of the benefits of using HTTP sans SOAP is that you can fetch other types of data, e.g., images. He also pointed out that a downside with using HTTP is that you don't get fault information reliably. In theory, you could get the same benefits with a mix of SOAP and some sort of attachment (SwA, MTOM) mechanism. The problem I have with all of those is that current implementations do too much buffering, so the memory overhead for doing that is huge compared to just pulling data off the wire. Still, I think the key point is that SOAP can offer value in this space if we had tools that made it more straighforward for people used to working with HTTP to do what they really want. SOAP can add value in this space, but right now toolkits charge too high a price in terms of what they do to your programming model.

The other comment was from Kevin, who had three interesting observations. First, that XML was a plumbing detail and that “while some might have the urge to wallow in it like pigs in muck, I don't want to fart around with it any more than I have to, because that is simply a distraction from the actual payload.” I know that I have a reputation for being very XML-centric, but believe me, I don't want to wallow in it either. Rather, I'm interested in a practical approach to building systems using Web services. What I've found, though, is that hiding XML makes this harder. Interop, versioning, etc. all get easier if you are willing to deal with XML sometimes. That doesn't mean that you never want to map to objects, either yourself or automatically. It does mean that you want to control that process if and when it occurs or you're toast.

Second, “the issue with WS-everything-in-sight is complexity. It seems like too much configuration to do simple things - and those policy files are a nightmare: wasn't XML supposed to be human-readable?” I completely agree. The whole thing needs to be simpler so that people can understand it, debug it, etc. People focus too much on tooling that magically just makes sure everything is right. Even in the basic WS stack, that adds two levels of indirection on top of what's really going on - an XSD/WSDL description and a CLR or Java based meta-description of what's supposed to happen. In WS-* it gets worse: you need policy files to describe every aspect of the semantics of your communication.

This tooling-centric declarative style is appealing to a lot of developers because it hides tons of details. On the other hand, it's very complex and is hard to make work right. It would be better if we focused on the low-level building blocks first, even if people have to use them imperatively instead of declaratively (that is, they have to write code). Once we get that working we should think about the higher level abstractions. But until we get that working, there doesn't seem a lot of point. (As an aside, it is interesting to note that WS-Eventing and other app-level specs don't use XSD/WSDL to describe what's going on. You don't actually need that stuff to do WS work, only to drive tooling.)

Finally, Kevin notes that he's been using the SOAP version of the Amazon Web services because it suited his needs on the client platform. He can see the usefulness of the REST version - and he doesn't care if it's REST or not - for building Web sites because there's direct access to the XML. This is exactly my point. I don't think people are embracing REST services because of architectural purity (the rest of the Web isn't pure REST, so I don't know why this would be). Rather, they embrace it because it's easier in a lot of cases. There is no reason that SOAP couldn't be the same, except that toolkits hide raw XML and you have to know how to get it.

What we desperately need to do is a critical analysis of all this from the pragmatic perspective of developers who are trying to build systems. Basic WS are not as useful as they could be because of the dominant object-focused toolkit programming models. Aspects of WS-* are certainly useful, but that's lost in the stack's growing complexity. The idea behind separate specs was that it the behavior would be factored so that you could adopt what you need and leave the rest behind. Increasingly, though, the stack is presented as a single entity that you buy into or not. Fundamentally, we need to get to a place where cost of adoption is low and benefit is high. That doesn't mean we need tools to hide everything so that we don't see the price we're paying. It means we need tools that let us get as close to the wire as we need (not want, need) to be to get interop, versioning, performance, etc. In short, tools that offer direct access to XML and to our transport protocol. We should only have to move up the stack when we're getting something valuable and then the cost of adoption should be as low as possible. Then we should dig into the WS-* specs and really figure out what's valuable and how much it costs to adopt.