How does language, memory and package size affect cold starts of AWS Lambda?

In a recent blog, we examined the performance difference between the runtimes of languages that AWS Lambda supports natively. Since that experiment was specifically interested in the runtime differences of a ‘warm’ function, the ‘cold start’ times were intentionally omitted.

A cold start occurs when an AWS Lambda function is invoked after not being used for an extended period of time resulting in increased invocation latency.

Since the cold start times of AWS Lambda is an important performance consideration, let’s take a closer look at some experiments designed to isolate the variables which may impact the first-time invocations of functions.

From my experience running Lambda functions in production environments, cold starts usually occurred when an AWS Lambda function is idle for longer than five minutes. More recently, some of my functions didn’t experience a cold start until after 30 minutes of idle time. Even if you keep your function warm, a cold start will occur about every 4 hours when the host virtual machine is recycled — just check out the metrics by IO Pipe.

For testing purposes, I needed a reliable method for consistently ensuring a cold start of an AWS Lambda function. The only surefire way to create a cold start is by deploying a new version of a function before invocation.

For the experiment, I created 45 variations of AWS Lambda function. Using the Serverless framework setup below, it was easy to create variants of the same function with different memory sizes.

I recursively deployed all 45 functions and invoked each of them programmatically using the simple script below.

The deployment and invocation loop took about three minutes. To collect a meaningful amount of data points, I ran the experiment for over 24 hours.

I based the hypothesis on my knowledge that the amount of CPU resources is proportional to the amount of memory allocated to an AWS Lambda function.

1. C# and Java would have higher cold start time
2. Memory size affects cold start time linearly
3. Code size affects cold start time linearly

Now it was time to see if the experiments supported my hypothesis.

To evaluate the the impact of memory on cold starts, I created 20 functions with 5 variants — using different memory sizes for each language runtime. The supported languages are C#, Java, Node.js, and Python.

After running the experiment for a little over 24 hours, I collected the following data — here’s the results:

Observation: C# and Java have much higher cold start time

The most obvious trend is that statically typed languages (C# and Java) have over 100 times higher cold start time. This clearly supports our hypothesis, although to a much greater extent than I originally anticipated.

Observation: Python has ridiculously low cold start time

I’m pleasantly surprised by how little cold start the Python runtime experiences. OK, there were some outlier data points that heavily influenced some of the 99 percentile and standard deviations — but you can’t argue with a 0.41ms cold start time at the 95 percentile of a 128MB function.

Observation: memory size improves cold start time linearly

The more memory allocate to your function, the smaller the cold start time — and the less standard deviation. This is most obvious with the C# and Java runtimes as the baseline (128MB) cold start time for both are very significant.

So far, the data from the first experiment supports the initial hypothesis.

To evaluate the the impact of memory and the package size on cold starts, I created 25 functions with various code and memory sizes. Node.js was the constant language for this experiment.

Here are the results from this experiment:

Observation: memory size improves cold start time linearly

As with the first experiment, the memory size improves the cold start time and standard deviation in a roughly linear fashion.

Observation #2 : code size improves cold start time

Interestingly the size of the deployment package does not increase the cold start time. I would have assumed that the bigger package would equate to more time to download & unzip. Instead, a larger deployment package seems to have a positive effect on decreasing the overall cold start time.

To see if the behaviour is consistent, I would love for someone else to repeat this experiment using a different language runtime. The source code used for these experiments can be found here, including the scripts used to calculate the stats and generate the plot.ly box charts.

theburningmonk/lambda-coldstart-comparison—
lambda-coldstart-comparison - Comparing the coldstart time of AWS Lambda functions using a variety of language runtime—github.com

Here are a few things I learned from these experiments:

• functions are no longer recycled after ~5 minutes of idleness, which makes cold starts far less punishing than before
• memory size improves cold start time linearly
• C# and Java runtimes experience ~100 times the cold start time of Python and also suffer from much higher standard deviation
• you should consider running your C#/Java Lambda functions with a higher memory allocation than you would Node.js/Python functions
• bigger deployment package size does not increase cold start time

update 04/07/2017: I posted a follow-up experiment to find out how long Lambda keeps idle functions around before recycling them.

Master modern tech skills, get certified, and level up your career. Whether you’re starting out or a seasoned pro, you can learn by doing and advance your career in cloud with ACG.