Analyzing the staying power of Python—and 3 ways the language can improve

There’s no debate that Python is one of the biggest programming languages in the world. But as technologists, teams and CTOs look to the future, they might be asking if Python is worth the investment.

To that end, let’s explore Python’s decade of exponential growth, the current challenges and common sticking points developers have with the language, and how Python can evolve to become even more accessible to new users.

On a personal level, my involvement in Python has grown from when I first used it for research projects in grad school to continually finding its way into my work as I’ve moved from job to job and project to project over 20 years. Now, it’s grown to the point where many of my projects are pure Python, so I’ve had a front-row seat to seeing it mature into a wonderful language.

When I take an even wider view than just my own personal experience with the language, it’s interesting to trace Python’s larger growth in popularity and maturity starting in around 2012.

The chart above shows the percentage of Stack Overflow queries for popular coding languages since 2009. As you can see, Python was near the bottom in the early 2010s, before taking a sharp uptick that turned into a meteoric rise toward the end of the last decade—a trajectory that hasn’t slowed down.

One conclusion you could make from this graph is that perhaps Python’s queries have increased because it’s a difficult language that requires people to ask a lot of questions about it. However, I don’t think that’s the case: As someone with a lot of C++ experience as well, I would consider that a much more difficult, question-worthy language, yet it hasn’t seen the same rise in Stack Overflow’s data.

I think what most people would probably conclude is that Python has simply become very popular.

To add additional insight, this chart shows the number of Python Enhancement Proposals (or PEPs) since 2000. A PEP is how a member of the Python community argues for a change to the features or standard library of Python, or how the language operates in general, and then that proposal is discussed, argued and voted to decide if it should go into effect; because of all this involvement, the number of PEPs can act as a rough gauge of how active the language is.

When you look back, you can see a lot of PEPs in the early 2000s, when the language was really being shaped into the Python we know today, followed by lots of peaks and valleys until around 2012, when the amount of PEPs reached a high and sustained level of activity through 2019.

Both charts show an uptick in Python community activity and popularity in 2012, which is when Python 3.3 came out. When Python made the switch to Python 3—going all-in on a version that was not backwards compatible to Python 2—it was a decades-long project and a big risk that has clearly paid off. In my experience, Python 3.3 was really when the language began to work reliably and powerfully without a lot of extra engineering required, and it could be used reliably on projects knowing that all the pieces you needed were there.

All this serves to create happy programmers, and happy programmers are excited to feed back into the language, write their own libraries, submit proposals and keep the ecosystem thriving. When that happens, the language continues to grow and, well, the rest is history.

With a sense of what Python looked like yesterday, you might be wondering: What about right now? Should I learn Python? Is it something I should invest time in? Can I bet on it being supported and having a solid developer pool for a while?

As I consider the two charts I previously shared, a couple truths about Python were made clear to me, and I feel that these truths can help answer some of these questions.

1. Python is immensely popular: It may seem trivial to reiterate this, but we must acknowledge the role a language’s popularity has in its staying power. If people aren’t using it, it’s likely going to die off. People are using (and enjoying using) Python.
2. Python’s level of popularity is self-sustaining: With that immense popularity comes the idea of a self-sustaining system; Python will continue to experience widespread use largely because it’s already there. It would be incredibly difficult to displace at this point.
3. Python’s approachability make it attractive: The innate qualities of Python, the way the language works and feels, will continue to draw developers to it—and perhaps more importantly, non-developers. Much of Python’s success has come from the fact that people without a CS degree, who don’t want to get into type systems or other esoteric programs can get into Python quite easily, because it’s pleasant to work with.
4. Python has an active developer base bringing in new ideas: With so many people contributing PEPs and committing to the standard library and ecosystem, Python is ready and able to pull in outside ideas in the future that can continue to make the language better. This active developer base is the bedrock of sustained growth for Python.

In short, I anticipate a Python in the future that’s a bit more modern and stately, and still a scrappy sense of community support and developer involvement, but is perhaps not as frenetic and rancorous as it has been in the past (those who lived through the the traumatic transition from Python 2 to Python 3 may know what I’m talking about.) 

To get to that phase of Python’s lifespan, I think we need to see several key potential problems addressed in a meaningful way. I say “potential” with intention, because I think all of the issues I’ll be bringing up hold just as much opportunity in them as they do pain points.

Potential problem #1: The tooling is confusing

In the beginning, there were distutils (the old way of building packages in Python), and they worked just OK. The more we pushed on it, the more we discovered places where it didn’t do what we needed it to. 

Now, what new Python developers are faced with is a lot of tools and technologies that end in “env” for some reason—Pyenv, Pyvenv, Virtualenv, Pipenv and so on, plus things like Twine, PIP, Flit and Poetry—that they have to wrap their heads around if they want to build anything but the simplest kinds of Python programs.

In truth, none of these are all that complicated; they all have a job and do them well. But coming in as a new programmer, you can read one blog post and get a particular path through these technologies, and read somewhere else and get a totally new path, until eventually you find you have to know what they all do before you can make a choice of what’s right for you. And that’s problematic.

It’s the same problem that plagues languages like JavaScript or Node: For the uninitiated, there’s a whole zoo of technologies to choose from, and it’s easy to get spun around.

A possible solution: 

Python knows this is an issue, so over the next few years, I believe we’ll see things settle down to some core tools to create a smoother experience for beginners. We’ll see these tools coalesce into something that provides a much nicer entry into the language.

Much in the same way that pytest has become the unofficial gold standard for doing testing in Python, we’re going to reach a similar point with our tooling around packaging and package uploading (and all the logistics there) as enterprising developers continue to bring in new ways to make the language more approachable to beginners by getting rid of that soup of terminology.

Potential problem #2: Bloat in the standard library

In the early days of Python, the language gained popularity for carrying a “batteries included” promise: There is a standard library of packages that you get with every Python installation, so no matter what you write, you are guaranteed to have those packages included. If you wanted to give something to your friend to run, they wouldn’t need any special installations—you just need Python, and it’ll work.

There’s now a growing notion that the idea of having such a large standard library has run its course, and is an albatross around the neck of the core Python developers tasked with maintaining it, especially when you consider that a bunch of stuff in the standard library traces back to 1980s-era protocols that are long out-of-date.

A possible solution: 

PEP 594 proposes killing these “dead batteries” from the standard library, which would allow for effort to be spent on more important things like runtime or really important libraries. Taking this a step further, I can also see a future where we have a decentralized standard library, so that if a developer needed to use a different version of datetime or pathlib, for example, they wouldn’t need to be stuck with a particular version in their python runtime.

Potential problem #3: The GIL

Many Python developers probably don’t know what the Global Interpreter Lock (or GIL) is, and they are lucky. To oversimplify it without getting too deep, it basically means that only one thing can be happening in your Python program at a time; even if you have multiple threads, for example, only one can make progress because you have to hold the GIL to make progress. For obvious reasons (like wanting to run more than one thing at a time), the GIL has been a thorn in the side of Python for a while.

A possible solution: 

While the GIL doesn’t really need to exist anymore from a theoretical standpoint, I don’t think it will go away any time soon. I do think it will continue to be discussed, because there’s always going to be a subset of Python nerds (which I say with love) who will pay really, really deep attention to and want to continue to work on the GIL because it’s deeply technical. They’re entranced by it. But the flip side is that because it’s so technical, it would be really difficult to remove the GIL because of the deep systemic thought required, which is what has stopped it from happening.

A pro of removing the GIL is that it could unlock some very powerful new Python “superpowers” in terms of the programs you could build, but it wouldn’t really be relevant for most Python users because they’re building programs that don’t depend on things running parallel. Many of the problems with the GIL could also be addressed in other ways, such as sub-interpreters or solutions like asyncio. 

For those interested in the future of Python, keeping an eye on these three problem areas (and the discussion around them in the community) will be really important.

If you want to know what the weather is going to be like tomorrow, you can use a barometer. But if you don’t have a barometer, the best way to tell what the weather will look like tomorrow is to look at the weather today. I think the same thing applies to Python: It’s immensely popular and widespread, and thus we’re going to see it in the same places it’s in today.

So, where are those places?

Cloud configuration and automation, application scripting, system administration

Python is huge in these areas for many of the same reasons it’s popular in general: It’s very approachable and hackable, and the people working in these areas like using it. I don’t have to go in and compile anything; I can just make a change to the script that’s doing my sysadmin work and it just takes place immediately. Programs that work like this tend to stick around a long time.

Web development

There’s a lot of Python technology running web servers in various places, and frameworks like Django, Pyramid and Flask are becoming popular, and I think we’ll continue to see that. This is because these tools are powerful and pleasant to use—and by having a Python technology I can use to make a web server, it makes it easy to take some other Python technology that’s doing domain-specific work and plug a web server on top of it so I can expose it to the web. These days, everything has to be web-ified, and this makes it easy to do with Python.

Microcontrollers 

There’s a surprising amount of Python being used in work that goes on in microcontrollers. Historically, this has been where C really shines, but now we have projects like MicroPython and CircuitPython that make it much more straightforward to work on microprocessors. We’re going to see more and more Python in this area, especially with how much embedded work people are doing all the time in Arduinos and Raspberry Pi as hobbies.

Scientific computing

Python is the standard in scientific computing these days, having displaced MATLAB to a large degree, with a large ecosystem of tools for doing all sorts of low-level, cross-discipline scientific work (such as NumPy, pandas and Numba) as well as domain-specific libraries (like Biopython and Astropy). A big reason Python has dominated (and will continue to dominate) the scientific space is that most researchers and scientists don’t want to become expert programmers; Python is a low friction, lightweight way to get enough into programming that they can do their work with programming becoming their job.

You can watch Austin's full webinar on the future of Python here, and improve your own Python skills with the Pluralsight learning path "Core Python."