The state of Web services

There's been a lot of discussion lately about the state of the Web services stack. I've been thinking a lot about two posts I read recently, Edd Dumbill's recent column at xml.com and Tim Bray's recent blog post. Edd's piece includes a section entitled “no love for Web services”, which starts this way:

General despair and astonishment at the state of web services specifications has been breaking out all over recently. As my friend and colleague Simon St. Laurent notes, the whole web services project seems to have lost its way. Personally speaking, while I admit that the complexity may serve to facilitate some deeply complex e-business scenarios, I've never been a fan of the over-engineered and largely untested web services protocol mess.

And it's not just the WS-mess, even plain old SOAP itself isn't getting as widespread a welcome as you might have thought. Many developers are now admitting in public (and privately in the case of some high-profile companies) that SOAP complicates matters too much and the simple combination of HTTP and XML that is REST will suffice.

I think there is a lot of confusion out there about what's going on with the WS stack and there are pieces that are a mess. But it is far from a lost cause, as I'll explain when we get to the end of this. First, I have to agree that, yes, a lot of people are building XML over HTTP solutions that don't use SOAP. I don't think you can call this REST, exactly, because it doesn't necessarily follow the tenets of URIs and fixed verbs that that ex post facto architecture calls for. (According to Fielding's thesis, such common Web mechanisms as cookies and URIs that contain embedded user ids aren't RESTful either, so at this point I'm not really sure what follows the architecture in practice, other than ATOM) The Amazon Web services, for instance, come in two variants: REST and SOAP. The REST variant isn't really REST as far as I can see - it tunnels service requests through a single URI with query string parameters and it models some (I believe) non-idempotent operations as HTTP GETs - but that isn't really the point. It's a lot more popular than the SOAP variant and it's important to understand why.

For people who are used to building Web based systems, working with URLs and markup is natural and intuitive. They look at the Web service toolkits, most of which focus on exposing the methods of a CLR or Java type via SOAP and describing the whole thing with XSD and WSDL and all they see is unnecessary complexity. I got here myself working on a service for publishing documents at MSDN. Building an ASP.NET Web service that efficiently processes a large document is not impossible, but it's far from easy. In this case, the Web service kits hinder as much as they help, so poeple just use HTTP and XML instead. This is how Office 2003 retrieves it's online help documents. It's also how ASP.NET 2.0's callback mechanism works (actually, it doesn't even mandate XML, you can pass an arbitrary string and input and get an arbitrary string as output).

The important thing to note is that this is a toolkit issue, not a protocol issue. Nothing in SOAP/WSDL requires that you work with objects instead of XML. If toolkit vendors shifted their focus to embrace developers who want to deal directly with XML messages, I think a lot of the people who are discarding SOAP might be more interested. As an added bonus, adopting a more XML focused position can also help address interoperability and versioning problems. If the data is available as a stream - which you get with raw XML over HTTP - you might be able to improve performance as well.

On the .NET platform, both ASMX and WSE let you program directly with XML, but in ASMX it takes effort to setup and in the latter case you have to use the DOM, which introduces nontrivial overhead. Even with this functionality there, the main focus is on marshaling parameters to method calls. Is it any wonder so many people say that the Web service stack is an “RPC for the Internet“ and, with the addition of more and more WS-* specs, continue to compare it to CORBA? Tools shape a technology for better or for worse. As long as the tools make WS about method calls, then RPC is all it will be.

And that would be too bad, because there's a real contribution to make. Consider the recent discussions about the scaling problems of using RSS over HTTP. MSDN dropped full text from one of its aggregate feeds because it was too big and requested too often and that event was picked up as an opportunity to observe yet again that polling isn't the best way to do this sort of thing. Edd has another piece at xml.com about the issues facing RSS distribution, suggesting that perhaps NNTP or BitTorrent might be a better solution than HTTP. I mention this not because I have a solution in mind, but because it provides the solution to a problem that Tim Bray cited in his post, and that Edd quoted in his column

Here's the problem: the people writing these WS-things aren't stupid. In particular, Don Box isn't stupid. And I'm starting to see names of people here at Sun pop up as WS-coauthors; people who aren't stupid.

I think the answer is easy. The authors of WS-* recognize that HTTP is not the only protocol in the world and that it is wildly inappropriate for some things. So the goal is to create mechanisms to communicate securely and reliably using multiple transport protocols. For instance, if you've got RSS data that's available over both HTTP and NNTP, do you want to develop two entirely separate mechanisms for securing access to that information (assuming it isn't just free to the world)? The WS-* answer is no. Ditto for addressing the services that deliver the content, making access to them reliable, making them discoverable, etc.

How long will it take for all this to be developed, tested and widely deployed? It's really hard to say. But WSE is a good start. It's supports pluggable transport protocols, an XML-centric API, and the key WS-Security, etc. features. If they add reliable messaging, an interoperable raw TCP binding and a more efficient way to handle large messages and I'll be where I want to be.

In the mean time, we shouldn't let WS-* et. al. distract us from the immediate, pressing problems of versioning and interop. Most importantly, we have to do address the general perception that Web services are all about objects instead of XML and are too complicated to use in any case. We need to make it easier to adopt the platform and toolkits by focusing on layering. If it isn't easy to get close to the XML and the transport protocol when you need to, and only adopt the higher-level abstractions when they actually add value, people will just walk away.