In praise of S3, the greatest cloud service of all time

25,000 #CloudMadness voters can’t be wrong: Amazon S3 is the greatest cloud service of all time.

Really, you never know how a popularity contest like that is going to go. In this case though, I think the voters made the right call. And I have reasons!

Depending on how you define it, the Simple Storage Service may not have been the first AWS offering - SQS technically launched first, but wasn’t available in production until later. Either way, a managed message queue alone was never going to convince most businesses to consider public cloud. A distributed storage solution that could scale as wide as your data with zero ops effort absolutely was.

As AWS’s Mike Deck put it on Twitter during the heat of voting: “I don’t think you’re appreciating what a revolution it was back in the late ‘00s to have virtually infinite, highly durable, pay-as-you-go storage that didn’t require you to manage any hardware.”

Remember when you had to cycle your backup drives every seven days, or drive a wagonload of tape to the offsite DR location? A generation that grew up with S3 doesn’t.

But S3 has become so much more than just a storage repository. As a static web server, S3 dishes up content for hundreds of thousands of websites including Netflix, Wikipedia, and the New York Times. In fact, the world has “standardized” on S3 APIs to such an extent that Google’s competing service just supports them out of the box.

That’s why, on the rare-ish occasions when S3 has an availability blip in one region or another, what seems like half the internet goes down. It’s hard to envision another cloud service -- except maybe a CDN such as the closely-related CloudFront -- having quite such an impact. S3 is fundamental internet infrastructure at this point, and it’s here to stay, because...

Distributed storage remains one of the hardest problems in computer science, especially at scale. History is littered with managed services that fell by the wayside because they couldn’t maintain integrity of customer data, their most precious resource.

With that in mind, S3’s durability guarantees -- eleven nines, are you kidding me? -- represent a jaw-dropping engineering feat. To put it in perspective, you’re more likely to personally get hit by a meteor than to lose just one of a million S3 objects… 400 times more likely. Check out this rather astonishing re:Invent keynote from S3 VP Mai-Lan Tomsen Bukovec, and try not to glaze over at the numbers: exabytes of data, tens of trillions of objects, more than 235 microservices distributing that data across who even knows how many physical facilities.

She’s talking about formal proof-of-correctness algorithms, checksums across loosely-coupled systems, complex actuarial models that anticipate when drives will fail. AWS has automated “durability auditors” that repeatedly crawl every byte of S3 data to verify that when you retrieve your stuff, it will be correct. And they constantly update these tools based on what they’ve learned from almost 15 years of running at unimaginable scale.

All that for the random code artifact I uploaded seven years ago by typing “s3 sync” at my command prompt. Makes me feel a bit unworthy, to be honest.

Sure, S3 has added tons of features over the years, some more specialized than others (S3 Access Points, anyone?). But the core value prop hasn’t changed: you drop in as many objects as you want, store them as long as you want, and they will never die. Drives fail, data centers go offline, but S3 remains. That’s why we developers take it for granted, an axiom you can build entire architectures around: like the sun rising, S3 will be there in the morning, barring a planetary extinction event. And that’s what makes it literally, unquestionably great.

Achieve greatness by developing cloud skills! Join A Cloud Guru and get access to all of our courses, labs, quizzes, and our new learning paths, which take you step-by-step from novice to guru in your chosen area of the cloud.

When most of us hear the word “serverless”, of course, our minds jump to a different service -- AWS Lambda, the original FaaS that launched a generation of stateless apps and HackerNews arguments. (It’s no accident that Lambda finished a close second in #CloudMadness polling.)

And yet Ben Kehoe, who’s been building serverless apps at iRobot for years, advocated hard for S3 in the polls. “S3 is the epitome of serverless,” he told me. “It's solving an incredibly hard problem that everybody needs solved, with a (relatively) straightforward API, scaling to whatever traffic you can throw at it, but costing you just what you're using. And its performance constantly gets better without any action needed from the user.”

Tim Allen Wagner, who invented Lambda, says Lambda “actually got started as an off-shoot of S3, not EC2. So there's another revolutionary thing that S3 gave the world!”

That’s right: the whole serverless revolution started as a way to build triggers for S3 events. Wagner remembers one of his career’s “scariest moments” as integrating Lambda with S3: “Back in those days it was like pointing a firehose at a dixie cup. Fortunately, S3 has an amazing architecture and team and Lambda grew into some (very big) shoes to join them at scale.”

“While S3 may be the greatest cloud service of all time,” Kehoe adds, “Lambda deserves credit for shifting the conversation from ‘these managed services are useful in and among my servers’ to ‘whoa, I can do *all* of this with services’. It was an evolutionary step in managed compute, but it revolutionized people's thinking.”

And that’s really the bottom line, isn’t it? S3, along with a few other foundational AWS services like EC2 and Elastic Load Balancer, established the primitives that allowed a Cambrian explosion of high-level innovation over the last ten years. While S3 may not be the shiny new thing anymore, it’s worth our time to step back and appreciate just what AWS has given us all these years with a storage storage optimization service that, under the hood, is anything but simple.

To see above the clouds, after all, it helps to stand on the shoulders of giants. And S3 is a giant.

Forrest Brazeal is an AWS Serverless Hero and enterprise architect who has led cloud adoption initiatives for companies ranging from startups to the Fortune 50.

Master modern tech skills, get certified, and level up your career. Whether you’re starting out or a seasoned pro, you can learn by doing and advance your career in cloud with ACG.