Amazon MSK
A fully managed streaming service that provides Apache Kafka cluster provisioning and operations on AWS, with Kafka-compatible API support
Overview
Amazon MSK (Managed Streaming for Apache Kafka) is a streaming service that lets you run fully managed Apache Kafka broker clusters on AWS. It is compatible with the open-source Kafka API, allowing you to migrate existing Kafka producer and consumer applications without code changes. MSK Serverless automates broker provisioning and scaling, while MSK Connect enables managed execution of Kafka Connect connectors to simplify integration with databases and S3.
Key Considerations for Broker Design and Partition Strategy
MSK provisioned clusters offer two broker types, Standard brokers and Express brokers, and with Standard brokers you determine the instance type, count, and storage size yourself. The developer guide states that with Express brokers you don't need to provision or manage storage for your data. The developer guide states that you specify the number of brokers per Availability Zone (AZ), with a minimum of one broker per AZ. The partition count is a key parameter as it determines throughput parallelism, and the principle is to match it to the maximum parallelism within a consumer group. Partition counts can be increased later but cannot be decreased, so initial design requires balancing headroom for future traffic growth without over-provisioning. The best practices page for Standard brokers recommends a three-AZ cluster, a replication factor (RF) of at least 3, and min.insync.replicas (minISR) of at most RF minus 1, and states that a minISR of 2 allows three-way replicated topics to be available when one replica is offline. Standard broker storage uses EBS, and the quotas page lists a maximum of 16,384 GiB per broker. AWS states in the developer guide that MSK supports KRaft mode, which replaces ZooKeeper for metadata management, that existing ZooKeeper-based clusters on Kafka 3.9.x have an in-place migration path to KRaft, and that clusters on a Kafka version past its end-of-support date are upgraded automatically to the recommended version (as of October 2026).
Reducing Operational Overhead with MSK Serverless and MSK Connect
MSK Serverless is an option when you want to eliminate the effort of capacity planning and scaling brokers. It automatically scales based on topic write/read throughput, removing the need to think about broker instance types or storage capacity. However, it does not allow full freedom to tune all Kafka configuration parameters, and there are limits on partition counts and consumer group counts, which are listed on the quotas page. MSK Connect is a service that runs Kafka Connect connectors in a managed environment - you can set up CDC (Change Data Capture) with Debezium or data lake exports with S3 Sink Connector by registering the connector as a custom plugin. Azure offers Azure Event Hubs with a Kafka protocol-compatible endpoint, but Event Hubs has its own partition management model, so Kafka replication factor behavior and consumer group rebalancing mechanics differ.
Selection Criteria - MSK vs. Kinesis Data Streams
When building real-time streaming on AWS, choosing between MSK and Kinesis Data Streams is a common design decision. Kinesis is an AWS-native service integrated with Lambda and Firehose. MSK, on the other hand, lets you leverage the rich Kafka ecosystem of connectors and stream processing frameworks (Kafka Streams, ksqlDB) as-is, resulting in a lower learning curve for teams with existing Kafka expertise. The decision comes down to three main factors. First, if you have existing Kafka clients or connector assets, the developer guide states that existing applications, tooling, and plugins are supported on MSK without changes to application code. Second, Lambda supports both Kinesis and MSK as event sources, and serverless options exist on both sides, with Kinesis on-demand mode and MSK Serverless. Third, for retention, Kinesis Data Streams has a default of 24 hours and a maximum of 365 days, while the developer guide states that MSK retention is set within the available storage and that tiered storage scales to virtually unlimited storage (as of October 2026). The pricing models differ between the two and are described on each service's pricing page.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon Kinesis official page
- AWS Lambda official page
- Amazon S3 official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.