XML Nation

ROA vs GET/POST

tim-ewald — Fri, 19 Oct 2007 16:51:00 GMT

Bill deHora has a great response to one of my (somewhat) recent REST posts. The timing is perfect, because I've been thinking a lot about what I said in those posts. I've also just finished reading the RESTful Web Services book, which is the most useful technical book I've read in a while, and have been thinking a lot about what it says too (and talking about it with Craig). I've been mulling over a post for a couple of days, and Bill's post is the impetus I needed to share what I've been thinking.

In RESTful Web Services, the authors define and argue the merits of Resource Oriented Architecture. ROA maps directly to the REST-as-CRUD model for thinking of the world, where interaction with your system is modeled entirely in terms of create, read, update and delete operations against data resources. For data-centric services, like the del.icio.us service that they refactor in the book, this model makes a ton of sense.

But what if your problem domain is more focused on processes than data? As the authors show with their proposal for modeling transactions, you can map any process to ROA with the following steps (or ones like them):

PUT/POST to create a new process resource
PUT to update that resource to include the data it needs
PUT to execute that process
GET to fetch the result

This maps any process to a set of resources. From this perspective, my argument that HTTP is all about POST and everything else is an optimization or unnecessary doesn’t make sense.

So could I model all processes the ROA way? Undoubtedly yes. But, what’s the value of this over just doing a POST with the data I want to process and getting the result back? It’s much easier to implement that because I don’t need the build-up of the process resource to span multiple stateless requests. I don’t ask this rhetorically, I really want to understand why people think this approach to modeling processes is better. (This is especially true if core bits of network infrastructure don’t support PUT/DELETE and you end up having to tunnel them through POST anyway.)

Maybe the process perspective is just the wrong way to think about the world, but it’s where a lot of people’s heads are at, including almost all the SOA/WS folks out there. In looking at my own system, I see one interface that is obviously data-centric and fits easily into the ROA model. Another interface is very process oriented. Right now we’re doing it with GET for read and POST for write, and it works and people get it. While I can see how to map it into the ROA model, I’m still unclear on how it helps.

So, to summarize, Bill is right: you shouldn’t follow what I said in that earlier post. (I always reserve the right to get smarter, and I’m using that now.) I’m still debating between the full ROA vs. GET for read and POST for write (again, especially if I have to tunnel through POST anyway), I really need to build some more stuff and see how it plays out.

Don asked for input...

tim-ewald — Sun, 26 Aug 2007 16:17:00 GMT

I haven't been reading or writing much lately - too heads down getting to 1.0. But I did happen across Don's recent post about retiring the four tenets of SOA and asking for input on what we'd like to write when we implement services. Let me take these topics in order. First, the four tenets of SOA...

What people are trying to build are loosely-coupled systems where pieces can be changed without breaking other pieces. I've built a couple of systems that meet that goal to a reasonable degree, so I don't think it's unreasonable for me to say that it's possible but hard and it takes a long time. In fact, it's just like building reusable components, which is also possible, hard and takes a long time. It isn't clear to me that many organizations are actually prepared to put in the work that loose-coupling really requires. What is clear to me is that the 4 tenets and the current tools aren't going to get them there. The only thing they really do is distract Capital-A-architects, which isn't a bad thing.

Which gets me to the second topic, what does the code I want to write look like. Here's my current thinking...

There are 4 essential distributed technologies: sockets, message queuing, RPC, and REST (in the distributed state machine sense I was writing about a couple months ago). There is a place for each in my current system. We use sockets to support certain industry and legacy protocols, both TCP and UDP, including multicast. We're looking at using message queuing for some ancillary processing off the main line of execution. We're using RPC to talk to SQL Server - either TDS, which is essentially RPC with streams, or SOAP via WCF, which is also essentially RPC despite what the proponents of mass customization exhort you to, in cases where a physical deployment separates our client from our database and the IT guys don't want to open 1433. For loosely-coupled cross-component integration we use REST.

I don't care that there are different APIs for all these technologies. In fact, I think it's a good thing because (a) I get an API tailored for the technology I'm using which (b) makes the differences between them clear. The only API that is lacking in .NET, IMO, is the REST one. If you aren't doing pages, ASP.NET leaves you hanging with IHttpHandler. Similarly, HttpListener isn't much of an API. Yes, I could put the new WCF REST bits on top, but I haven't, for several reasons. we needed a solution before they were available, so we already have a UriTemplate-based dispatcher of my own. We don't want to change .NET libraries at this point in the project, and can't move to a beta anyway. And finally, I'm not sure I want to build on a layer designed to factor HTTP in on top of a layer that was designed to factor it out.

The lack of a strong REST API is problematic because REST is a much better alternative for loose-coupling than RPC is. So, what do I want for an API? Easy: I want a UriTemplate matching layer (with a back-door for arbitrary regex's) that works on ASP.NET or HttpListener, abstracting away the differences between their respective context/request/response objects (which is just annoying). And I want a text-templating engine for output, a la NVelocity, with editor support. I'm close to building my own, since I already have the first half, with NVelocity, but haven't had time yet. (Yes, I've looked at monorail, but it feels like more than I want and I don't think it integrates with HttpListener.)

I'd like all that to run on top of 2.0 if possible, or at least in a kit that runs on XP and 2003, even if it requires 3.0 or 3.5. And I'd like it in the next 6 months or so (which I know is a pipe dream). The lack of this interfaces is a big part of my interest in Rails.

As to an example based on TransferMoney, here's what I'd do...

Design an idempotent implementation with a unique request id that can be saved on a client before initiating the request, used by a client in the case of failure to ask if the operation was processed (maybe by looking through a list of the operations processed in the last 24 hours), and used by the server to avoid processing the same request twice. (I'd make this part of the application logic instead of the transport protocol because reliability of the latter doesn't mean I can' t crash between getting a message and executing logic based on it, unless I go all transactional messaging, which introduces other issues.)

I would map out the state machines for the client, the server and the protocol between them. I would map the protocol itself to a set of URLs, as per my recent posts on REST. I would implement the endpoints by mapping URLs to methods using UriTemplates, parse the inbound data using whatever felt right and emit a response using a text templating engine.

I'd test it with the browser. I'd document it in prose, and maybe with an XSD for reference or to generate code if desired.

Don't know if that's concrete enough. It echoes what we're doing in our system now, which works great.

If the question was what do I want to see as a SOAP API, I don't have any issue with the [WebMethod]-esque style that WCF and every other stack uses. SOAP/WS-* turned out to be RPC/CORBA and the API is fine. That isn't as negative a statement as you might think.

What's in a name?

tim-ewald — Thu, 10 May 2007 17:06:00 GMT

My recent posts are my attempt to describe how I think about the Web and how it can be leveraged to integrate systems independent of the browser. I got a ton of feedback in comments and email, which was all great. I’ve been away for a bit, so I thought it would be good to come back and summarize, and to ask a question about what to call the model I’ve adopted. Specifically, some worry that the term REST means too many things to too many people to have any real meaning at all. But before I get to that, let me summarize and respond to some of the key points people made.

Most of the feedback I got can be divided into a couple of bins...

First, that it’s just RPC. That’s wrong for all the reasons I listed in my last 4 posts.

Second, it’s not RPC, so it isn’t appropriate beyond browsing a hypertext graph of multimedia content. The first half is right. I strongly believe the second half is wrong, but we don’t have enough experience applying the model in other systems to say concretely. We need to get that experience.

Third, and most interesting, what you describe isn’t REST, for a range of reasons. Some of this was because I wasn’t crisp in defining the difference between managing system state, session state, and transitions between nodes in a protocol state machine. Clients make HTTP requests to move through a protocol state machine. This is completely unrelated to changing system state or to maintaining session state. Specifically, a GET is a move between nodes (or states) in the protocol, but does not (or should not) affect system state. I adopted the term “node” here to describe moving between states in the protocol state machine. Is that clearer?

Some felt I didn’t conform to REST because I didn’t include the notion of changes over time. Specifically, I said that each state in a protocol state machine has a URI, which I think is true. Each time the client enters this state, they get a representation of it. That representation may change over time. That doesn’t mean that each state in the protocol doesn’t have a unique URI, it means that each representation does not (unless you mix perma-links into the protocol).

The most important feedback worried about the fact that the Web isn’t really RESTian or that my model doesn’t align very well with the HTTP spec, which focuses on CRUD operations on entities. Let me address those concerns.

The Web does not fully conform to Fielding’s paper, but the reality is that the Web is RESTful enough. An implementation is never as pure as an idea (Look deeply the SOAP header processing model and the not-so-independent WS-* specs for another great example.). Yes, the modern Web uses cookies extensively, which don’t conform to Fielding’s ideas, but are extremely useful. More importantly, they do not sacrifice the notion of a stateless server, as long as there is a shared backend store. We could debate the merits of a stateless client vs. a stateless server, but the latter is how the Web really works, and pragmatically, that’s what I’m after, whether it conforms to Fielding or not.

About the model I’ve adopted not aligning with the HTTP spec, I disagree. The spec is written substantially in terms of CRUD operations on resources. The goal of the early Web was essentially distributed content publishing, the ancestor of the Wiki. The HTTP spec reflects that. We’ve moved well passed that problem domain to all sorts of other things, e.g., shopping at Amazon. It’s easy to think of the shopping process in terms of a protocol state machine. It’s harder to think of the content management problem that way, but you can if you try. In that problem space, the protocol state machine and the CRUD operations on entities converge into one model – in content management, the CRUD operations on the entities are the transitions in the protocol state machine.

I’ve been very careful in my last several posts not to get hung up on which HTTP methods are being used to transition between states. The HTTP spec positions the four main methods as equals, but I don’t look at it that way. POST is the core method. GET is an optimization that enables caching and, more importantly, makes it possible to bookmark protocol states as a single text string. PUT and DELETE made it in because of how the Web was first conceived. If it had started with the Amazon shopping cart, those verbs might not have been there. They might have been in an extension protocol, like WebDAV, but it didn’t happen that way. (Hopefully that isn’t too inflammatory.) I back up this position with the observation that while we encounter GET and POST all the time, PUT and DELETE are very rare. (Maybe APP will change that, but it hasn’t happened yet, and falls in the content management problem space for which those special verbs were defined anyway.)

But all of that is really moot. I don’t want to get lost in discussions or architectural styles, whether or the Web really conforms to Fielding (my answer is it conforms enough), whether REST and HTTP align, whether it is possible to be RESTian at all, the meaning of PUT and DELETE, or other distractions. I want to focus on digging into HTTP-based distributed state machines (a term I got from Sam) that leverage the pragmatic solutions for scalability, reliability, and security that we’ve developed on the Web.

So given all that, is REST the best term to use for this, or is there something better?

Three reasons that REST is not RPC

tim-ewald — Sat, 28 Apr 2007 23:01:00 GMT

I've gotten several comments saying that, at the end of the day, REST is just RPC. That's wrong, for at least 3 very reasons:

1) Each unique state in your protocol state machine has its own URI. That's different from an RPC endpoint that maintains a black-boxed state machine at a single endpoint. Being able to do state transition processing at disparate locations is hugely powerful. Watch the URLs you are navigating through as you browse, shop and checkout at Amazon. A single process can span machines offering differing levels of scalability, reliability and security.

RPC doesn't do that. You could conceivably build an RPC system that did do that, but if that happens it is a very rare occurence indeed. The more likely solution would be to have multiple RPC interfaces to the different machines involved in the process and then tie their protocols together. The application in the middle would have to understand how to mix the two protocols, passing data from one into the other, and back the other way. Yes, you can make it work, but it bakes a lot of protocol detail into the application mixing the calls, creating a much more complex solution.

2) With transition URIs embedded in the representations of states, it's easier to transition between states in the protocol. All of the URIs are accessed the same way, there is no separate interface per endpoint because the each endpoint represents the transition to an individual states, not all of the transitions required by a protocol. Again, technically this is doable with RPC, but nobody does it this way and solutions are more complex as a result.

3) The messages aren't interpreted as serialized call-stacks. The purpose of RPC is to copy a stack frame from one process to another, where it exists for the duration of a method call. That makes RPC very natural to anyone used to invoking a method, but it also makes it very hard to alter one side of the system without altering the other. While HTTP-based systems still use request/response messages, they aren't call-stacks. The fact that WS tools want to treat them that way is one of the reasons they have so much trouble handling large binary data, e.g., JPEG.

None of this is to say that RPC isn't useful. A couple people have mentioned callbacks to clients behind NAT based firewalls, a problem that RPC based on WCF duplex channels are very good at solving (protocols like XMPP and plain old TCP work well too). My system uses both RPC and REST. But it's a mistake to say that REST is RPC just because HTTP is request/response, we often use XML to represent state, and there are tools to map XML to and from callstacks. I made that mistake for a long time, until I realized what REST really is.

On programming models...

tim-ewald — Fri, 27 Apr 2007 19:04:00 GMT

Ittay commented on my REST post:

the thing is, when you write software, you use an RPC model. what bothers me about REST is that it is not only an API. it enforces you to change your programming model.

that is not to say i don't like it. i do, for its simplicity and self documentation (e.g., provide all moves as links), but there is a price you pay.

When you write software, you use a programming model that works. And sometimes you have to change models. We changed them for the Web: we moved to the notion of pages. It wasn't RPC, it wasn't even objects (at least from most developers perspectives originally). But it was simple and did what it was supposed to do. I've done RPC, CORBA, DCOM, Remoting, RMI, and Web services. All of those technologies have their place. But they all struggle in a loosely-coupled, massively distributed world. I'll happily change my programming model to solve that.

An example might help...

tim-ewald — Fri, 27 Apr 2007 18:56:00 GMT

I got a lot of great comments on last nights post, including a couple about REST being no different from xml-based RPC. I used to think so too, which is why my recent epiphany was so eye opening.

Consider a protocol for finding and reserving a flight between two cities. The client is in one of these states:

<ready>
- searched
- retrieved details
- reserved

These states map to URIs:

<none>
- http://quuxTravel.com/searched
- ??? depends on previous state
- ??? depends on previous state

A client begins by navigating to the searched state by GETting http://quuxTravel.com/searched?src=London&dest=NYC. The client gets back some XML like this:

The client is now in the searched state. It scans the list of itineraries to find the one with the lowest price. If the client wanted some other criteria that isn't surfaced in this state, e.g., total flight time, it could transition to the retrieved details state by GETting the URI stored in the itinerary's getDetails/@uri attribute. It would then return to the searched state (either by an explicit back-link or a history a la' the browser). The system would return an XML representation of that state that contained flight info.

When the client has chosen a flight, it transitions to the reserved state by POSTing to the URI stored in the itinerary's reserve/@uri attribute. It gets back an XML document confirming the reservation. At this point the protocol is complete. The client can begin again if desired, or go do something else.

Now, why is this different from RPC? Imagine the following interface for implementing this same protocol:

interface IFlightSystem
{
    Itineraries Search(string src, string dest);
    Details GetDetails(int itineraryId);
    Confirmation Reserve(itineraryId);
}

This interface exposes the same protocol, with more or less the same requirements on the client to know what the data being sent and received means. The difference is that in this case, the client is talking to one endpoint and mapping request/response payloads to call stacks. In the previous case neither of those things were true.

The REST model opens the door for the protocol to be implemented across different endpoints. This is useful for scalability, partitioning and data-directed routing, integration with external systems (note that the transition to the reserved state uses a URI at a partner company). In other word, it's actually a web of endpoints. Further, those URIs are dynamically constructed, so you can change them based on user, time of day, the data they're interested in, locale they're from, state of your data center, or whatever. That is hugely powerful. Because the documents being sent around are not mapped to call stacks, and may not even be mapped to objects, it's easier to stream data, add extra stuff over time, etc.

In one of his comments, Ittay asked what the REST model for the a method “string Foo(string, int, bool)” would be. In response, I described a simple protocol with one state, “FooInvoked”. To get to that state, you'd access a URI for /foo, passing a string, int and bool as query string parameters or in the request body. You'd get back a result state that contained a string. He countered that that felt just like a function call, which isn't surprising because that's where we started. The key, for me, is to look at problems not from the perspective of methods, as Ittay did, or entities, as Joe describes, but as states and the transitions between them. Then it really starts to make sense. And the power of it is real.

I finally get REST. Wow.

tim-ewald — Thu, 26 Apr 2007 18:39:00 GMT

Yeah, I'm alive. And I remember the password to my blog. I've been away for a bit, working on something very cool involving the TV. If all goes well, you'll hear about it in a big way. Anyway, I'm still having a ball out here in reality. Building something real has a way of focusing your decisions about technology. My app is a distributed system, some of which runs in a cable plant head-end or telco office (whatever's on the other end of the wire in your living room), and some of which runs elsewhere. We also connect to some things on the Web. These connections have to be extremely flexible and the bar to adoption has to be low. The thing I finally realized (some of you will say “Duh!”) is that Web services are not a good way to do this.

It's depressing to think that SOAP started just about 10 years ago and that now that everything is said and done, we built RPC again. I know SOAP is really an XML messaging protocol, you can do oneway async stuff, etc, etc, but let's face it. The tools make the technology and the tools (and the examples and the advice you get) point at RPC. And we know what the problems with RPC are. If you want to build something that is genuinely loosely-coupled, RPC is a pretty hard path to take.

That realization would have gotten me down if not for the fact that something else jazzed me up an hour or so later. I was in the process of considering the alternatives when I finally understood REST. And wow, it was eye-opening. REST is often positioned as CRUD operations against entities identified by URIs. Then it is dismissed as to simplistic to be useful. You can't build with just CRUD, the reasoning goes, just think about why we write sprocs. I've been down that path any number of times and always ended up in the same place. But I had it all wrong.

I skimmed Fielding's thesis a while back, but it wasn't until I read Sam Ruby's recent posts that it really sank in. Here's what I came to understand. Every communication protocol has a state machine. For some protocols they are very simple, for others they are more complex. When you implement a protocol via RPC, you build methods that modify the state of the communication. That state is maintained as a black box at the endpoint. Because the protocol state is hidden, it is easy to get things wrong. For instance, you might call Process before calling Init. People have been looking for ways to avoid these problems by annotating interface type information for a long time, but I'm not aware of any mainstream solutions. The fact that the state of the protocol is encapsulated behind method invocations that modify that state in non-obvious ways also makes versioning interesting.

The essence of REST is to make the states of the protocol explicit and addressible by URIs. The current state of the protocol state machine is represented by the URI you just operated on and the state representation you retrieved. You change state by operating on the URI of the state you're moving to, making that your new state. A state's representation includes the links (arcs in the graph) to the other states that you can move to from the current state. This is exactly how browser based apps work, and there is no reason that your app's protocol can't work that way too. (The ATOM Publishing protocol is the canonical example, though its easy to think that its about entities, not a state machine.)

The “state machine as node graph traversed via URI” view of the world has really interesting implications for being able to suspend and resume a protocol. Because links to other states are embedded in a state's representation there are interesting ways to solve dynamic load-balancing, data-directed-routing, versioning and other problems using normal Web infrastructure. And because it's HTTP based, you get all the features that protocol supplies, including streaming and support for non-XML MIME types (a huge concern when you're doing TV stuff). The one thing that's really missing here is a simple framework for implementing a URI graph on top of an HTTP handler (similar to what Marc's been working on for Java). I'm building my own now.

The thing I love about this model is that, as Sam says, it is of the Web, not over the Web. That doesn't mean I'll use it for everything. I use TDS to get to SQL Server. I use WCF for RPC-style communication between distributed components within major systems. I'll use this model when I cross major system boundaries, especially when I don't own both sides. I'll let you know how it turns out.

Are Excel Services the way to bring developers and business people together?

tim-ewald — Wed, 14 Jun 2006 10:58:00 GMT

I heard about Excel Services a while ago, but hadn't had any time to look at them even briefly until now. Basically, it's a server-side system that lets you access data and calculations in Excel spreadsheets via Web services. Think about how much business data and calculation is done with Excel. Now imagine being able to leverage the directly. Want to change the algorithm you use to compute some key financial data? Let the analyst modify the spreadsheet and copy the update to your server and you're done. Now *this* is the way to align technology and business. Of course, that assumes it all actually works well - I haven't done anything yet. But still, it has *tons* of potential. Very cool idea, definitely something to spend more time with.

MSDN Content Web Services online...

tim-ewald — Mon, 12 Jun 2006 16:58:00 GMT

One of the last things I did before leaving the MSDN team was to prototype a Web service for retrieving content programmatically. It's been a while since then, but a production version is now live. Craig provides an excellent introduction here. The prototype client we always talked about was msdnman - a command line tool like the Unix man command. Craig was the lucky one - he got to build it. It will be interesting to see what people do with these services. Off the top of my head, I can imagine including actual docs in a technology site, integrating with tools (a la' Reflector), and building a team level doc repository on TFS that allows you to annotate, add your own docs, etc. Anyway, congrats to the MTPS team for a job well done!

Two articles, one good and one bad...

tim-ewald — Fri, 19 May 2006 18:30:00 GMT

I ran into two Web service related articles recently. One really resonated with me: Enable the Service Oriented Enterprise, in the MS Architecture Journal. It presents the Enterprise Service Orientation Maturity Model, or ESOMM. Okay, I know what you're thinking: eeeeeeeewwww, a maturity model! But it's a lot more interesting and useful than you think (and they distance themselves from that other MM in a sidebar). Lots of developers and some architects think about service orientation in terms of the famous four tenets. They are good guiding principles and very useful, but not what a lot of people mean when they talk about SOA, all caps. Sure, there's a lot of hype around SOA, but there is a real point there too. Many companies are trying to redesign their software infrastructure into a portfolio of coarse-grained, reusable services. To do that successfully, you have go way past the four tenets for individual services and think about how you're going to organize the whole thing. I spent a lot of time thinking about that problem when I was at Mindreef. This article really summarized a lot of what I'd thought about, and a bunch of stuff I hadn't, in a nice, easy to understand way. If you work at a company doing “big SOA”, look at ESOMM.

The other article, Avoid XML Schema Wildcards For Web Services Interfaces, appeared in Internet Computing, but can't be downloaded without purchasing (which is pretty crumby, guys!). I got my copy forwarded by an interested reader who wanted my opinion on the position it took. I agree with the beginning of the article, which was disagreeing with the schema techniques described by Dave Orchard (and reiterated with slight variation by Dare) that were embraced by the W3C TAG. This approach is just too complicated in practice. The second part of the article described a versioning model that supported backward compatibility (old sender, new receiver) but did not address forward compatibility (new sender, old receiver). The problem with this one-way compatibility is that it just doesn't work. Imagine a typical request/response message exchange between an old client and a new service. The request message must support backward compatibility so that the old sender (the client) can communicate with the new receiver (the service). But the response message must support forward compatibility so that the new sender (the service) can communicate with the old receiver (the client). Having one without the other is essentially useless. Supporting both was big part of what lead me to my own versioning model. I'm not saying there are no other approaches to versioning, but if they don't support both backward AND forward compatiblity, then they're not useful in the context of most Web services today.

Versioning and semantic changes

tim-ewald — Fri, 05 May 2006 18:27:00 GMT

I got an email a little while ago from someone who read about my preferred XSD versioning strategy. They felt I had glossed over the issue of introducing a change that effects semantics and might be ignored by a receiver. Consider this example:

Now suppose I evolved the schema by adding this optional element:

The new Currency element is optional so that client who don't send it still work. They use the default currency that was used with the original version of the Deposit message. This works fine as long as you only never have new clients talking to old services. In that case, the client could send a Deposit with a Currency, but the old service would skip that unknown element (as per the rules I described in my original post).

The solution to this problem is to determine whether you will ever have the new client / old service situation (if you're services are deployed as single instance or always updated at the same time before any clients are released, then you never will). If you do, then exercise your judgement about changes. If you are going to change an XSD in a way that alters semantics, make sure you differentiate new instances somehow. For instance, you could add this to your schema:

This way you are versioning your schema, but not the Deposit element. This avoids the problem without introducing too much extra complexity.

Attempting to address Rajesh's concerns about optionality...

tim-ewald — Fri, 05 May 2006 18:16:00 GMT

My last post on optional content triggered a conversation between Rajesh and Jon. Rajesh's main concern is that making almost everything in your schema optional may give you flexibility, but if occurence requirements aren't captured somewhere, then its hard to consume a service (or other XML). That's true - you need to know what's required. The problem is that if you put that info in your schema, and then you need to change it and that requires changing your namespace, then you have a problem. I can see two possible solutions. One is to include occurence info in your schema but make it clear to everyone that you feel free to change it without changing your target namespace. The big issue here is that if you have an XSD used by multiple services that have different occurence constraints, what do you do? You could combine all the requirements and express the weakest, but keeping that up to date over time would be hard. You could have two different copies of the schema, each with different occurence requirements, but then how does the consumer generate code from both in a meaningful way? You could solve this problem by using a common schema with everything optional, and then offering a per-service redefinition of the schema with specific occurence constraints. xsd:redefine would handle that nicely, but you need the tools to know that they're supposed to generate code from the core schema referenced by the redefine. Since none of the mainstream code-gen tools handle redefine, that seems like a dead-end (which is too bad because it *does* capture the semantics nicely). No, I still think you're better off focusing your schema on describing the shape of data, if it is present, and captureing the current rules for what must be present some other way, e.g., documentation. Increasingly, I think of this as a sort of hybrid of static and dynamic typing. At development time, you get enough info to statically compile your code and not be way off base (as you could be with Process(string xml)). But the system is dynamic enough to allow wiggle-room at run-time to compensate for service evolution.

Initial code for version-aware schema validation

tim-ewald — Tue, 25 Apr 2006 18:22:00 GMT

When I wrote my initial post on my new approach to XSD versioning, I promised that I'd post code. Here's the first cut:

class Program
{
static void OriginalMain(string[] args)
{
    FileStream xml = new FileStream(args[0], FileMode.Open);
    FileStream xsd = new FileStream(args[1], FileMode.Open);

    XmlReader reader = null;
    XmlReaderSettings settings = new XmlReaderSettings();
    settings.IgnoreWhitespace = true;
    settings.ValidationFlags = XmlSchemaValidationFlags.None; //ReportValidationWarnings;
    settings.ValidationType = ValidationType.Schema;
    settings.Schemas.Add(XmlSchema.Read(xsd, null));
    settings.Schemas.Compile();

    // wire-up anonymous callback delegate
    int badDepth = -1;
    bool nextNodeInvalid = false;
    settings.ValidationEventHandler += delegate(object sender, ValidationEventArgs ea)
    {
      Console.WriteLine("Event -- {0}: {1}", ea.Severity, ea.Message);
      Console.WriteLine("Event -- {0}\t{1}\t{2}\t{3}\t{4}", reader.NodeType, reader.Name, reader.Value, reader.SchemaInfo == null ? "<none>" : reader.SchemaInfo.Validity.ToString(), reader.Depth);

      if (reader.NodeType == XmlNodeType.Element &&
          reader.SchemaInfo.Validity == XmlSchemaValidity.NotKnown &&
          nextNodeInvalid == false)
     {
        Console.WriteLine("Filtering out unexpected stuff now...");
        nextNodeInvalid = true;
        badDepth = reader.Depth;
      }
      else if (reader.NodeType == XmlNodeType.EndElement &&
                  reader.SchemaInfo.Validity == XmlSchemaValidity.Valid)
      {
          Console.WriteLine("Other stuff expected, ignoring...");
      }
  };

  reader = XmlReader.Create(xml, settings);
  while (reader.Read())
  {
     if (nextNodeInvalid)
     {
       int targetDepth = badDepth - 1;
       while (reader.Depth > targetDepth)
       {
         reader.Read();
         Console.WriteLine("Filtering -- {0}\t{1}\t{2}\t{3}\t{4}", reader.NodeType, reader.Name, reader.Value, reader.SchemaInfo == null ? "<none>" : reader.SchemaInfo.Validity.ToString(), reader.Depth);
       }
       nextNodeInvalid = false;
       badDepth = -1;
     }
     Console.WriteLine("Main -- {0}\t{1}\t{2}\t{3}\t{4}", reader.NodeType, reader.Name, reader.Value, reader.SchemaInfo == null ? "<none>" : reader.SchemaInfo.Validity.ToString(), reader.Depth);
}
}
}

What this code does is pretty simple. It catches XSD errors and if they occur at the beginning of an element, which indicates unexpected stuff is present, they ignore that error and any others that occur for that depth in the document. This version also happens to catch XSD errors if they occur on the close tag of an element, which indicates that expected stuff is absent. With optioanl extenstions, this is never a problem, but it was interesting to make it work with extensions that aren't marked minOccurs=”0”. Anyway, when it detects extra unexpected stuff, it indicates that that content is “filtered” - meaning that it could be removed from the XML stream and the resulting document could be assumed to be valid as per your current schema (you'd really want to do another validation pass to be absolutely sure, but I wouldn't bother in general). My next step is to wrap this in an XmlReader implementation so that it can be piped into a serializer more simply. It needs its own reader because it relies on control of the Read loop to filter stuff out. Anyway, I'll post that when it's working correctly. BTW, one thing to note about this code is that it only works with the sequence compositor. With all compositors, you can't assume that an unknown element is the beginning of content from a later version. I don't have any problem with this limitation, but it has to be mentioned.

Slogging

tim-ewald — Tue, 25 Apr 2006 12:07:00 GMT

Now matter which way you tackle contract evolution, you need to have a system in place to notify your service consumers. I envision a system based on “service blogs”, or “slogs”. A slog conveys information about the state of a service. As a consumer, I want to subscribe to a slog for each service I use. My aggregator becomes a “dependency dashboard” that tells me about upcoming service news. One thing I’d catch this way is upcoming service revisions. Optimally, the revision notification would include a pointer to a test environment and a time frame for testing. I would run my system against the new endpoint in order to test the functionality. If it works, great, I’m done. If it doesn’t, I need to figure out what I have to change to make things work again and I need to start a conversation with the service team to coordinate changes on my end and/or their end, as well as plans for when/how I migrate. As long as I’m on this topic, I should include the need to see basic ops stats bubbling up this way to so that I have a good sense that the services I’m using are meeting their SLAs. This whole part of the picture, which involves inter-team communication, seems pretty much ignored today. Mindreef is the only company I know that’s addressing it, and they’re in the early stages.

Making everything optional

tim-ewald — Thu, 20 Apr 2006 18:51:00 GMT

DJ commented on my post addressing the problem Raimond raised with my versioning strategy. He wondered if he'd missed an earlier post where I argued that you not use XSD to validate your data because if you make content optional, you can't use it to check what has to be there. Since I haven't written about that yet, I figured I'd start to address it now.

When people build a schema for a single service, they tend to make it reflect the precise requirements of that system at that moment in time. Then, when those requirements change, they revise the schema. The result is a system that tends to be very brittle. If you take the same approach when you design a schema for use by multiple systems, describing a corporate level model for customer data for instance, things are even worse. Some systems won't have all the required data. They have to decide whether to (a) collect the data, (b) make up bogus data, or (c) not adopt the common model. None of these are good approaches.

To solve both these problems, I've started thinking about my schema not as the definition of what this system needs right now but as the definition of what the data should look like if it's present instead. I move the actual checking for what has to be present inside the system (either client or service) and implement it using either code or a narrowed schema that is duplicate of the contract schema with more constraints in place.

The advantage of this model is that I can change the “has” requirements without changing the shape of the data. Yes, I'm changing my contract and I need to make sure that clients test against the new version before it all goes live. There's a chance, though, that some or even many clients won't have to change. If I revise the schema itself in a way that forces a namespace change, then I have to support parallel contracts or migrate all clients at the same time. Neither is a good option. In other words, I'm trying to minimize schema changes and use testing to pinpoint required code changes.