Getting Started with Event Streaming on Amazon MSK Serverless - Building a Zero-Ops Kafka Environment
This article introduces a method for building a Kafka environment that needs no cluster management with IAM authentication, and simplifying event streaming through integration with EventBridge Pipes.
Features of MSK Serverless
MSK Serverless is a Kafka environment in which users do not need to handle cluster provisioning, scaling, or patching. When you create a cluster, a connection endpoint is provided, and you can start creating topics and sending and receiving data from Kafka clients. There is no need to specify broker instance types or the number of nodes. The throughput limits are 200 MBps per cluster and 5 MBps per partition for ingress, and 400 MBps per cluster and 10 MBps per partition for egress. With provisioned clusters you can customize Kafka configuration parameters (such as retention.ms and max.message.bytes) in detail, but in Serverless these settings are managed automatically.
IAM Authentication and Access Control
MSK Serverless uses IAM authentication as standard. Kafka clients connect to MSK with AWS credentials (IAM roles), and you set topic-level access control with IAM policies. Fine-grained control that allows a specific IAM role only to read and write specific topics is possible. The user name and password management required by SASL/SCRAM authentication on provisioned clusters is no longer needed, and IAM also handles credential rotation automatically. For connections from Lambda functions or ECS tasks, authentication is complete simply by granting the execution role access permissions to MSK.
Integration with EventBridge Pipes
EventBridge Pipes is a service that uses MSK topics as a source and connects them directly to targets such as Lambda, Step Functions, SQS, and Kinesis Data Streams. Previously you needed to develop and operate your own Kafka consumer applications, but with Pipes you can pass events to targets without writing consumer code. Filtering enables selective processing based on event conditions, and with enrichment you can also insert data transformation by a Lambda function. With the combination of MSK Serverless + EventBridge Pipes + Lambda, you can build an event streaming pipeline without managing servers.
MSK Serverless Pricing
MSK Serverless pricing consists of cluster hours, partition hours, storage, and data transfer. A cluster costs about $0.75 per hour, and partitions cost about $0.0015 per partition hour (as of August 2026). Storage costs about $0.10 per GB per month, and you check how data transfer charges are handled on the pricing page. In terms of cost, Serverless is a candidate for workloads with intermittent traffic, and provisioned is a candidate for environments that run at high throughput 24 hours a day.
Choosing Between Serverless and Provisioned
MSK has two forms, Serverless and provisioned, and you choose based on the nature of the workload. Choosing Serverless is a good fit when traffic is intermittent and demand is hard to predict, when you want to minimize operational effort, or when you want to follow sudden scaling automatically. On the other hand, choosing provisioned is a good fit when high throughput continues steadily 24 hours a day, when you want to control detailed Kafka settings, or when you want to adjust costs to match usage. There is also the approach of starting quickly with Serverless and considering a move to provisioned once the load characteristics become clear.
Topic Design and Partitions
Kafka performance and ordering guarantees are determined by the design of topics and partitions. Because message order is preserved within a partition, you group data where order matters into the same partition with an appropriate key. Increasing the number of partitions can raise throughput through parallel processing, but it needs to be balanced against the configuration on the consumer side. With consumer groups, multiple processes can share partitions and consume them in parallel. Because a skewed key design concentrates load on specific partitions, choosing keys that distribute data evenly is the prerequisite for stable processing.
Implementing Producers and Consumers
Because MSK is Kafka compatible, you can use existing Kafka client libraries as they are. In serverless-oriented configurations, you can subscribe to MSK topics directly with Lambda event source mappings and eliminate the operation of consumer applications. If you want to standardize the structure of messages across your organization, you combine it with the Glue Schema Registry and manage compatibility while keeping schemas consistent. On the producer side, you configure batch sending and compression of messages to raise throughput and efficiency. Because authentication is completed with IAM roles, managing connection information becomes simple.
Monitoring and Operations
You measure the health of event streaming by consumer lag (the amount of unprocessed messages that have built up). If lag keeps increasing, the consuming side is not keeping up with the producing side, so you review consumer parallelism and the number of partitions. You monitor throughput, partition status, and errors with CloudWatch metrics and detect anomalies with alerts. Because Serverless has limits on throughput and the number of partitions, you check at design time whether the expected load fits within them. If you also decide in advance on the message retention period and how to handle processing failures, operations become stable.
Summary
MSK Serverless is a service that leaves Kafka cluster management to AWS and lets you focus on building event streaming. IAM authentication simplifies credential management, and EventBridge Pipes reduces consumer development. While maintaining Kafka ecosystem compatibility, you can take a form of operations that does not manage servers.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon MSK official page
- Amazon EventBridge official page
- AWS Lambda official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.