Plumbing 101: A bookshelf list

As I mentioned in a previous post, I'm essentially a plumber.  I work on infrastructure software, and have focused mostly on how to hook things together to get work done reliably.  I'm  a reasonably literate plumber, though.  I've worked, with moderate success, to stay up to date.  I personally think that this is key to being successful in the sorts of roles that I've had.

Like pretty much everyone else that I know, I've built up a reading list over time.  This varies from foundational books through to insightful and subversive papers.  So, let's imagine that you're starting on a team that builds transactional infrastructure.  What should be on your reading list?

First, there are the journeyman books that you probably already know about.  They'll tell you about the tools you have, and how to best use them to produce a product.  Aside from the your-favorite-language books, you'd undoubtedly want a couple like Code Complete and Writing Secure Code.

Next, you'll need at least one of the foundational texts.  My personal favorite is Gray & Reuter, Transaction Processing.  It's an imposing black book with a picture of the Rosetta Stone on the front.  Frankly, it weighs almost as much as the Rosetta Stone.  However, it is the first real 'how to' book for building a TP system.  It's almost 15 years old and still highly relevant.  I'd also recommend getting Weikum & Vossen, Transactional Information Systems.  It is also a truly excellent how to book, and newer.

[A brief aside: I started working on transactions some years before these came out, at a time when transaction management was a pretty active area of research.  Consequently, I got into the habit of reading the journals, and saving the ones that had some key insight.  I suspect I was typical.  I had my stack of papers and notes that I referred back to as needed.  It was all too reminiscent of Dr. Jones' grail diary.  When Gray & Reuter came out, much of the need to build your own grail diary disappeared.  Man did that make lists like this much simpler to construct!]

With these texts in hand, there are two old papers, both by Butler Lampson, as it happens, that I'd recommend as part of any basic library:

• Lampson, "Atomic Transactions", Distributed Systems-Architecture and Implementation.  This is a real gem of a paper.  It derives the two phase commit from a world where anything can fail, which is why I first read it.  Along the way gives you a great way of thinking about the edges of any design, which is why I recommend it.
   
• Lampson, "Hints for Computer System Design". This is a paper that anyone building system software simply must read.  And anyone building commercial software should read.

Then there are a couple of more recent items that you should understand.  These are:

• Lamport, "Paxos Made Simple".  If you're working on consensus protocols these days you should understand Paxos, and this is straightforward introduction to the topic.  You can also pick up Gray & Lamport, "Consensus on Transaction Commit", which relates Paxos and 2PC.

• Brewer, "Towards Robust Distributed Systems".  These are slides from the talk that described the "CAP conjecture" -- the inevitable tradeoff between consistency, availability, and tolerance to partitioning.  It's something that I think (hope) was reasonably widely recognized, but not clearly stated before this.

Finally, there are a couple of papers that I tend to recommend because they talk about ideas that are a little different.  They're good ideas, and they helped me flex my imagination when I read them:

• O'Neil, "The Escrow Transaction Method".  A clear example of where you can get to if you don't assume that locking implies CRUD.
   
• Haskin, et al, "Recovery Management in QuickSilver".  A pretty old paper, but one that asks what if transactions could be baked really deeply into your system.

Finally, these days, pretty much anything by Tim Harris on transactional memory.

Jim.