AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event sources

AWS Lambda now supports Apache Kafka queues with self-managed Kafka event source mapping (ESM). This allows customers to scale Kafka workloads independent of topic partitions, improving throughput during traffic spikes and processing resilience. Previously, Kafka ESM used the consumer group protocol, assigning each partition to a single consumer, requiring over-partitioning or moving traffic to another queuing system for higher parallelism, which added infrastructure and producer code maintenance overhead. Self-managed Kafka ESM queue support is based on Kafka shared groups, available in general availability with Apache Kafka 4.2. This release enables multiple event pollers to consume from the same partition simultaneously, with each poller supporting up to 10 concurrent invocations, decoupling parallelism from partition count without changing producers, topic settings, or function code. This improves throughput during traffic spikes and allows processing delayed or failed events without delaying other events on the same partition. As queue mode does not preserve partition order, it is suited for workloads where ordering is not required. This feature requires a shared group-enabled Kafka cluster (Apache Kafka 4.2 or later) and is available in all AWS commercial regions where self-managed Kafka ESM is offered. To get started, create a self-managed Kafka ESM with ConsumptionMode set to Queue via the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK. Standard Lambda pricing applies, and customers configuring provisioning modes for self-managed Kafka ESM also pay for event poller units (EPU). For more information, see the AWS Lambda developer guide and AWS Lambda pricing.

Why it matters

This update is useful for users building event-driven Kafka workloads for use cases such as image and video encoding, fraud detection, notification delivery, and telemetry ingestion.