Behind Lambda's 1,000 Concurrent Executions - How the Firecracker Warm Pool and the Worker Manager Work
This article explains how Lambda manages microVMs within the default limit of 1,000 concurrent executions, the reuse strategy through the warm pool, the placement decisions of the worker manager, and the internal optimizations that lower the cold start rate.
The Lifecycle of the Lambda Execution Environment
When a Lambda function invocation arrives, the Lambda service allocates an execution environment. An execution environment is a sandbox that runs on a Firecracker microVM and contains the function's code, the runtime, and the configured environment variables. The lifecycle of an execution environment consists of 3 phases. In the INIT phase, the runtime starts and the code in the global scope outside the handler runs. In the INVOKE phase, the handler function is invoked. In the SHUTDOWN phase, the runtime's shutdown processing runs. The important point is that after the INVOKE phase completes, the execution environment is not destroyed immediately but is kept in a "frozen" state for a certain period. When the next request arrives, the frozen execution environment is "thawed" and reused. This is a warm start. An execution environment in the frozen state consumes no CPU time and occupies only memory. Lambda manages a pool of these frozen execution environments, and this is the "warm pool".
Warm Pool Management Strategy
Lambda's warm pool management is an optimization problem that balances minimizing the cold start rate against maximizing resource efficiency. If execution environments are kept in the warm pool for a long time, cold starts decrease, but because they keep occupying memory, the capacity of the physical hosts comes under pressure. Conversely, if they are destroyed right away, resources are released, but cold starts increase. Which algorithm strikes this balance is part of Lambda's internal implementation and is not public. The only two points that are officially documented are that an execution environment that has finished an invocation is kept in a frozen state and reused for subsequent invocations, and that Lambda terminates that execution environment at an arbitrary time. The retention time of the warm pool has not been published either, and it changes dynamically depending on the function's invocation pattern and the capacity situation of the physical hosts. Therefore, at design time, you need to write code on the assumption that it is fast if reused but still works correctly whenever it is destroyed. Even in an implementation that reuses connections and clients initialized outside the handler, it is safe to treat initialization equivalent to the first run as something that can happen at any time. Firecracker itself is published as open source as a lightweight virtualization foundation running in both Lambda and Fargate, and you can read about its design in a paper presented at NSDI 2020.
The Worker Manager - Which Physical Host to Place On
Lambda's worker manager is the component that decides on which physical host (worker) to place a new execution environment. This placement decision needs to satisfy multiple constraints at the same time. The first is that the physical host has spare capacity (CPU, memory). The second is that the execution environments of the same customer are distributed across multiple physical hosts (fault isolation). The third is that, for functions that require a VPC connection, an ENI can be secured in a subnet of the target VPC. Before the improvement in September 2019, VPC-connected Lambda functions had to wait for an ENI to be created and attached at every cold start. AWS described the issue at the time as "the time required to create and attach a network interface makes cold starts longer", and the number of seconds of added delay itself has not been published. In addition, because the number of ENIs increased in step with concurrency, these functions also faced 3 ceilings at the same time: subnet IP address consumption, the ENI limit per account, and the rate limit of the ENI creation API. In the current approach, ENIs are created in advance when a function is created or when its VPC settings are updated. Execution environments connect by tunneling to ENIs managed by Hyperplane (the AWS network function virtualization foundation that is also used by Network Load Balancer and NAT Gateway). Because ENIs are shared across execution environments, a few per combination of security group and subnet are enough, and an increase in concurrency no longer leads directly to an increase in ENIs. With this change, cold starts of VPC-connected Lambda functions became nearly equivalent to those of non-VPC functions.
Concurrency Management and Burst Limits
Lambda's default concurrency of 1,000 per account is shared by all functions in the Region. If function A is using 800 concurrent executions, function B can use only 200. On the other hand, the rate at which concurrency increases (the scaling rate) is not shared but per function. The official documentation defines it as "1,000 execution environment instances every 10 seconds (or 10,000 requests per second every 10 seconds) for each Region / each function". The old model, in which an initial burst allowance was set for each Region and concurrency grew only by a fixed amount per minute after that, was replaced in the November 2023 revision. Because the limit became per function, even if one function uses up its scaling rate, the ramp-up speed of another function in the same Region is not affected. Invocations that arrive beyond this rate are throttled, resulting in an error for synchronous invocations and in retries from the internal queue for asynchronous invocations. As a rough guide to the speed, you can reach the default of 1,000 in 10 seconds, and even when you grow from 3,000 to 10,000 after a limit increase, it is at a level of about 70 seconds. This limit is a design that secures time for the internal infrastructure, such as Firecracker microVM startup, ENI allocation, and physical host capacity allocation, to keep up with sudden increases in load.
Two Generations of Cold Start Countermeasures - Provisioned Concurrency and SnapStart
There are two generations of means for avoiding cold starts at ramp-up. Provisioned Concurrency is a feature that initializes a specified number of execution environments in advance and keeps them in a warm state, and it can respond in double-digit milliseconds without waiting on the scaling rate constraints. The other is Lambda SnapStart. With SnapStart, when you publish a function version, Lambda runs it through initialization, captures the memory and disk state of the initialized execution environment as a Firecracker microVM snapshot, and encrypts and caches it. At invocation and at scale-out, instead of running the INIT phase from scratch, Lambda resumes execution environments from the cached snapshot. This changes the role of the initialization of the execution environment itself, which is the main subject of this article. The supported runtimes are Java 11 or later, Python 3.12 or later, and .NET 8 or later, and it is effective only on published versions and the aliases that point to them. It does not support use together with Provisioned Concurrency, Amazon EFS, Amazon S3 Files, or ephemeral storage of more than 512 MB. Also, one snapshot is reused as the initial state of multiple execution environments. Therefore, you need to design on the assumption that the handler side recreates the uniqueness of IDs and random numbers generated at initialization and the state of network connections established at initialization.
What to Watch Out for When Execution Environments Are Reused
Reuse of execution environments (warm starts) is advantageous from a performance perspective, but it has side effects that developers should watch out for. First, the state of global variables is retained. Values of global variables set in the previous invocation remain in the next invocation. Using this to cache database connections (connection pooling) is a recommended pattern, but if you store request-specific data in global variables, a bug occurs in which the data of the previous request leaks into the next request. Second, files in the /tmp directory are retained. Because files written to /tmp in the previous invocation remain, you may use up the disk capacity (up to 10 GB). Third, background processes remain. If you start asynchronous processing in the handler function and return a response without waiting for it to complete, that asynchronous processing may still be running at the next invocation.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
If this page and the official documentation disagree, treat the official documentation as authoritative.