More on making everything optional

My last post generated a lot of great comments. I think it's important not to confuse your schema with your contract. A client and a service have to agree on all sorts of things, only some of which are captured in your WSDL/XSD(/Policy). My goal in proposing that almost everything in your XSD be optional is to find the sweet-spot between easy coding and flexibility for evolution.

In general, I want to provide an XSD so that consumers can build an object model and get intellisense if they want to. But once I commit something to XSD, changing it (as opposed to extending it) typically means changing the XSD namespace. Most OX mappers bake the target namespace into code. So if I change that, I'm forcing a lot of changes into clients. I'm also forcing my service to work with two essentially identical type models if I want to support multiple versions. That leads to loads of boiler-plate object-to-object mapping code. If a client is using the same types to talk to more than one service as well, changing schema namespace may introduce the same sort of mapping there as well. So my goal is to maximize support for evolution without changing target namespaces. This is the heart of my versioning model, which breaks the world into three sorts of changes: additions, extensions and changes. I can handle the first two without changing namespaces. I want to see what changes I can handle without changing namespaces. The most obvious (are they also the most common?) change is occurance requirements.

Consider a simple example. I have a system that receives Customer data. In V1, I define customer like this:

<complexType name=”CustomerType”>
  <sequence>
    <element name=”FirstName” type=”string” />
    <element name=”LastName” type=”string” />
    <element name=”Address” type=”tn:AddressType” minOccurs=”0” />
  </sequence>
</complexType>

FirstName and LastName are required, Address is optional. Then, in V2, I decide that Address is actually required. I can implement this change two ways:

  1. Revise the XSD to capture this requirement by marking the Address element minOccurs=“1“, change the XSD target namespace because this is a breaking change, and then use schema validation or some other approach to enforce the requirement.

  2. Leave the XSD alone and modify my service logic to enforce the requirement using an internal XSD for schema validation or some other approach to enforce the requirement.

The advantage of (2) is that all systems that were already sending the optional Address data simply continue to work, and I don't have to write lots of object model to object model mapping code. It's only those that do not work that way which need to be modified (but if Address is a real requirement, they needed to be modified anyway). The benefit for those systems is that the modification does not require moving to a new schema definition, which means more work on their part (if they’re using the same types to talk to other systems that aren’t moving to the V2 namespace at the same time, this saves writing lots of boilerplate object-to-object mapping code).

You might argue that this approach makes it so the client developer doesn't or can't learn what the real requirements are for your service. I disagree. That information just has to be conveyed some other way. Further, this encourages the client developer to understand that your service will evolve and that they should expect that they may need to move to keep up.

It's worth noting that if you are designing an XSD for use by lots of different services, you probably don't know what occurance constraints they will require. Making content optional makes loads of sense in that case too.


Posted Apr 24 2006, 03:04 PM by tim-ewald

Comments

Jon Fancey wrote re: More on making everything optional
on 04-25-2006 6:28 AM
The comments to your last post definitely fell into two categories where I think people were arguing for different things. the first category were those agreeing with you on flexible contracts, the second category were those who thought that XSD=Contract. I wish that were true, or even a reasonable position to hope for, but we are years away from that. There is so much out of band communication required between consumers and publishers to get (nontrivial)services working that I'm not sure will ever be moved to a metadata format flexible enough to capture everything.

The slogging idea is an interesting step though - have you actually tried to implement such a thing? Can UDDI help here?
Tim wrote re: More on making everything optional
on 04-25-2006 7:25 AM
Jon,

I agree that people were in two camps. I just don't see the XSD == contract thing ever really working. The problem is that what you put in your XSD gets baked into code. The more you put in, the more you have to change your code and the more you force others to change theirs. Increasingly I think you want just enough in XSD to do reasonable code-gen to ease consumption. But that's it.

Tim-
Jon Fancey wrote re: More on making everything optional
on 04-25-2006 11:24 AM
Tim, all this reminds me of your call a couple of years back to eschew XSD. One of the things I **really** hate about OX mappers is the lock-in and it is on many levels. So why bother with it? Is it too hard to program against the message?
Rajesh Jain wrote re: More on making everything optional
on 04-25-2006 12:44 PM
I basically don't see anthing new that Tim has said in this post. I attempt to present a rebuttal on my blog. Please navigate to the URL (sorry for the link).

Regards
Jon Fancey wrote re: More on making everything optional
on 04-25-2006 1:03 PM
Rajesh,

I'm interested in whether you can give me a use case or example for the kind of systems that you're building. Being flexible in message processing is always extra effort - and I agree there are many situations that just aren't worth it. But I'd like to understand (if you don't mind Tim) what type of stuff you need to build.
Rajesh Jain wrote re: More on making everything optional
on 04-25-2006 2:10 PM
Jon,
** Tim, with your permission, I am submitting details about my case. I know you will appreciate the kind of lunacy I am dealing with ;-). **

I build systems in a large Financial Services company. We have a lot of internal "services" (Numbering 100s) that my team has to consume (e.g. Pricing or Underwriting). We also write some services for our internal and external clients and Partners.

I have always tried to publish strongly typed contracts using the code-first approach (its just simpler for me; maybe because I didn't face many challenges similar to what Craig eluded to) for the services I have written. However, our middle tier (which kinda predates Web Services) doesn't support strongly typed contracts and the in-out messaging structure is just defined as "String Process (String)". The middle tier essentially uses XML-Over-HTTP and has an "envelop" similar in concept to soap.

Consuming some of the services that are published on the middle-tier, becomes a VERY non-trivial task because the contract and datatypes are not defined properly. Most of the time the documentation either doesn't exist, or exists in the developer's head, or is outdated (again I can understand because when there is a pressing business problem like a new product launch etc., focus is on getting the development done. Documentation is a secondary thought. I wish there was discipline around this, but the fact is there isn't). Integration consequently becomes a HUGE excercise (as you can imagine).

On the other hand, some of my services are consumed by external partners. I have noticed that bringing an external partner onboard is relatively simpler (and provides failure short circuiting if they attempt to send incorrect data. I wish I could do more with something like schematron but...). I must also admit that we get requests from partners to send them detailed documentation about the message. In that case, we just use Altova Xml Spy->Schema DEsign->Schema Documentation ( ;-) ). Most of them request XSDs because it is much easier to program against (maybe because there are tools that generate native object types simlar to XSD.EXE in every major XML stack. Sometimes the Object-XML-object does cause some fidelity problems but we try to restrain our design to XML primitives.) None of them have ever asked for WSDLs! (You won't believe it but some partners hand-code (like string concats) the whole soap message with the envelop et-all. For them, we published a document about what a soap message looks like in XML!)

I hope his helps.

Rajesh
Jon Fancey wrote re: More on making everything optional
on 04-26-2006 1:03 AM
One of my beliefs is that you should only try to control the things you can control. Anything else is a lesson in futility. That's why I delineated between internal and external cases and inter/intra organisational lines. It's all about the control/power/influence you have. Remember that integration is about more than technical issues but organisational, logistical and political too. When building intra org/dept systems .net remoting, asmx or similar are great. They provide type-based code-gen that's easy to program against. However, this does not work in the other case for all the reasons Tim has illustrated plus others here (and more). It probably never will simply because no vendor controls all the stacks and standards and as such cannot guarantee interop across space and time (that's what OASIS is trying to achieve, and it takes a lot of time, effort, money and commitment). This leaves us with a problem. If you can't use OX mappers and the like then what can you use? Tim's approach is completely reasonable, and although there are others, all the ones I know of have strong similarities in approach and implementation.

Your first comment is interesting. There is no way to infer a contract from a message (string or otherwise). It's always lossy because it contains no metadata and is almost always a subset of allowed possibilities. This is at the heart of OX mappers and XSD issues. The abstraction breaks. Breaking client/provider interaction due to a change is never a good thing. Supporting multiple "contracts" is a non-starter for the mapping hell Tim mentioned.

Your second comment comparing to scripting/compiled etc is a different argument as it is really about control. Your situation is maybe more common than you might think - this is why I asked for an example. The solution isn't to tighten everything up so nothing works any more and fix it with a big bang, here's my *new* contract, but to design with evolution in mind. If you are baulking at the extra effort involved I don't blame you. It's unavoidable. But hopefully it's upfront effort that you'll get back over the lifetime of your service.

If loose contracts aren't for you, that's fine. Provided that you realize you're making it harder for people to do business with you.
Jon Fancey wrote re: More on making everything optional
on 04-26-2006 1:04 AM
PS - I hope you're not calling me a lunatic :-)
Rajesh Jain wrote re: More on making everything optional
on 04-26-2006 9:37 AM
Jon,
Noooooo way I am calling you a lunatic. I was refering to the lunacy in my life as a manager of a development team which frequently has to deal with these integration challenges.

Your first comment is ABSOLUTELY valid. I agree that the ball game changes when you deal with external people.

Your second comment also makes sense but I disagree that you cannot infer a contract (I know you cannot infer precisely as in mathematical sense but you can glean SOME interesting information nontheless.) I could care less about OX mappers at this point when I (the human) am trying to understand the inputs and outputs (I know I know, way too simplistic ;-) )

I also agree that the argument is somewhat different but the "pattern" (loosely defined) is the same

I guess my fundamental issue is that we never communicate the corollary with a pattern and it kinda becomes the norm and used as a canonical example by people who don't understand the finer points (Trust me, I am not on the vendor side of the business and I know the kind of stupidity I have to deal with ;-) You guys are experts and know the ins and outs of a pattern. In some ways, people on my side of the fence look at you to set an example). Maybe, Tim should say "Make everything in XSDs except when.....(fill in your blanks)"

Rajesh Jain
XML Nation wrote Attempting to address Rajesh's concerns about optionality...
on 05-05-2006 12:16 PM
Dare Obasanjo aka Carnage4Life wrote Tim Ewald on Versioning XML Web Services with XSD
on 05-14-2006 4:53 PM
US Cities Real Estate wrote re: More on making everything optional
on 11-22-2006 4:04 AM
take a quick glance at "capture it" app bundled with resco picture viewer(your favorite and mine too :-) ). It is also a very nice app and the best feature about it is taking screenshots in PNG format(which no other product offers). It performs well in all resolutions too(did well on my universal(just sold) and alpine(just bought))
Ken Brubaker wrote Tim Ewald's solution for XML Schema versioning
on 11-28-2006 7:55 AM
Tim Ewald addresses the XML Schema versioning issue head on.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?