To modernize your legacy apps, first stop being ashamed of them

Over 15 years in software engineering, Marianne Bellotti has built data infrastructure for the United Nations and tackled some of the oldest and most complicated computer systems in the world as part of the United States Digital Service. At Auth0 she ran Platform Services, and currently she runs Identity and Access Control at Rebellion Defense. She can be found on most social networks under the handle bellmar.

We sat down with Marianne to talk about application modernization, the future of COBOL, and her book Kill It With Fire: Manage Aging Computer Systems (And Future Proof Modern Ones), now available for pre-order from No Starch Press.

Forrest Brazeal: Through your work with the US Digital Service, you’ve occupied a unique place at the intersection of decades-old critical infrastructure and modern application development (keeping in mind that in this context, “modern” might just mean "Java”!) What do you think most developers would find most surprising about this work? What has surprised you?

Marianne Bellotti: I think what’s most surprising is how resilient many of these old systems really are. The conventional narrative is ‘these systems are so old that at any moment they will collapse and it will be a disaster,' but most of the projects I worked on were not having performance issues.

It was far more common for them to be brittle, meaning that if you didn’t try to change or add anything they could probably keep going for decades more, no problem. But the level of complexity and loss of institutional knowledge was so great that changes had unpredictable impacts.

Old systems are successful systems. They tend to be really effective at what they do. If they weren’t then people would stop using them and it would be easier to just shut them off.

What surprises me is how many layers of abstraction computer systems can have. Consider what the architecture for software looked like in the advent of computing, then look at things thirty years later, then look at them today. In the lifetime of a single generation, we’ve gone from very primitive code close to hardware to code that draws from natural language patterns and produces the same results on any hardware. That’s pretty amazing.

Tell us a bit about “Kill It With Fire"! Why did you decide to write a book on modernizing legacy systems, and what do you hope readers will get out of it?

I joined USDS when it was barely a year old. Back then people fretted that by working on old systems their technical knowledge was going to atrophy. This is really something people would complain about— that they were not going to be hirable anymore because they would lose all their skills.

But I always felt I left USDS a dramatically better software engineer than I came in. It was important to me to show people that side of it. That the skills you learn working on old systems are also incredibly useful when building new systems.

What’s the #1 misconception today’s developers have about mainframes?

That they don’t exist anymore! And that they’re slow. Mainframes are not slow. A modern-day computer the size of a refrigerator is not going to be slow.

And yet there’s a prevailing narrative that a world running on COBOL is unsustainable because both the language and its developers are aging toward obsolescence. Is this actually true? Where do new COBOL developers come from?

The funny thing about COBOL programmers dying off is that people have actually been saying that for decades! And the number of COBOL programmers has stayed stable all this time. Truthfully, the problem isn’t that COBOL is dying out, the problem is that most of the remaining COBOL systems are large and extremely complex.

It takes a lot of time and in-depth experience to navigate a large, complex codebase, no matter what it’s written in. I often tell people about this cluster of Oracle databases I was working on once. It took the expert Oracle sent us over a year to figure out what all the stored procedures were doing. This person had devoted his whole career to Oracle and it took him a year to untangle this system!

The people who are good at sorting out that level of complexity tend to be older. So one shouldn’t look at the number of twenty-year-olds programming in COBOL and assume that suggests the number of future COBOL programmers. After all, I wrote my first line of COBOL at 36. Anywhere there is important, impactful work to be done that will pay a good salary, people are going to be available to do it.

I love the idea of “Chesterton’s Fence”, which is that you should take time to understand the purpose of an existing system (like a seemingly useless fence) before trying to improve it. Maybe we need to invent "Chesterton's Mainframe"?

Actually, my favorite Chesterton’s Fences are the systems that managed to do modern things without the benefit of modern technology. That Oracle database cluster I mentioned before? That system was an absolute mess, but the team that built it figured out how to do multi-master replication before any database supported that.

When I would onboard engineers onto this system, after they got through complaining how awful things were, I would ask them how they would build it the “right way” if they were in the shoes of the original team. Then I’d point out to them how many of their preferred technologies didn’t exist when this system was originally designed. Slowly you see them start to realize that the right way to build it was exactly the way it was built to begin with. We can do better now only because we have tools that fit the use case better.

But it’s worth pointing out that the newer tools do not always fit your use case better. Typically when the old system beats the new system, this is what has happened. The new tools are optimizing for conditions that are not relevant.

The classic example is cloud vs data center. Which will save you money? Well … that depends on your scale. In government, most agencies will see huge benefits from cloud. But an organization like NOAA, which collects 20 terabytes of data every day, is not going to benefit from paying Amazon huge fees instead of running their own centers.

I often hear people who've seen the industry move from thin clients to thick clients to cloud and SaaS saying  “the cloud is just mainframes all over again”. What nuances might they be overlooking here?

I think they have a point, but often not the one they think they are making. When people compare mainframe timesharing to how consumers rent time on cloud, it’s often as a way of saying "the old stuff is good enough, why bother changing?"

For me the lesson here is to really study the landscape you’re operating in. The reason we shifted from thin clients to fat clients back to thin clients is that the cost of hardware vs data transmission changed.

When we all used dumb terminals to connect remotely to mainframes to run jobs hardware was really big and really expensive. By comparison, bandwidth was cheap. Then the hardware got small enough for machines to fit on people’s desks at about the same time that our primitive networks were becoming congested. So fat clients—otherwise known as desktop applications—become popular. But we responded to that network congestion by investing heavily in improving connectivity so …. thin clients are back.

“The cloud is just mainframes all over again” makes it sound like we had it right the first time around and would have gotten to this future faster if we hadn’t taken a detour through the PC age. But the investment in the network that brought us to cloud would not have happened if PCs hadn’t dramatically expanded the market of people willing to pay for bandwidth.

Any final takeaways for today’s developers and leaders as they make application modernization decisions?

I think the best way to modernize legacy systems is to stop being ashamed of them first. If you can appreciate the value an old system brings to the table, you can identify how to reproduce that value in more modern tools and also how to improve upon it. If you’re embarrassed by your legacy systems, you’re much more likely to end up chasing fads.

This interview is part of ACG's Cloud Conversations series.