Building an Apache Kafka Streaming Platform with Amazon MSK - Cluster Design and Operations
This article covers designing managed Apache Kafka clusters, choosing between them and MSK Serverless, and data integration patterns with MSK Connect.
MSK Features and Choosing Between MSK and Kinesis
MSK is a managed service for Apache Kafka, and AWS manages broker provisioning, patching, and failure recovery. Because MSK runs the open source version of Apache Kafka, you can use existing Kafka producers, consumers, Kafka Streams applications, and Connect connectors as they are. To choose between MSK and Kinesis Data Streams, you select MSK when you need compatibility with an existing Kafka ecosystem, and Kinesis when you prioritize AWS native integration and simplicity. As for the billing unit, Kinesis uses pay-as-you-go billing per shard, while MSK uses hourly billing for broker instances.
Cluster Design and MSK Serverless
For a provisioned cluster, you specify the broker instance type (such as kafka.m5.large), the number of brokers, and the storage capacity. For production environments, a minimum of 3 brokers (one in each AZ) is recommended, and you ensure data durability with a replication factor of 3 and a min.insync.replicas setting of 2. MSK Serverless is a mode that became GA in 2022, and with it users do not have to manage cluster capacity or scaling themselves. You can focus only on creating topics and sending and receiving data, and throughput scales automatically per partition. It suits applications that want to use capacity that increases and decreases automatically. However, provisioned clusters are more flexible for customizing Kafka configuration parameters.
Data Integration with MSK Connect
MSK Connect is a service that runs Kafka Connect connectors in a managed way. With the S3 Sink Connector, you can automatically write topic data out to S3 and analyze it with Redshift or Athena. It is also possible to update tables in real time with the DynamoDB Sink Connector, and to capture table changes in RDS with the Debezium Source Connector and stream them to Kafka topics. You can configure connector scaling as automatic or manual, and you adjust throughput by increasing or decreasing the number of workers. Because connector plugins are created from JAR files uploaded to S3, you can use connectors from Confluent Hub and the open source community.
MSK Pricing
Pricing for a provisioned cluster consists of hourly billing for broker instances and storage. kafka.m5.large costs about $0.21 per hour (about $151 per month), and a minimum configuration of 3 brokers costs about $453 per month (as of August 2026). Storage costs about $0.10 per GB per month. MSK Serverless is billed by cluster hours (about $0.75/hour) and partition hours, a pricing model where you pay only for what you use. Compared with Kinesis Data Streams (about $0.015 per shard hour), you select MSK when you need Kafka ecosystem compatibility, and Kinesis when you value AWS native integration.
Designing for High Availability and Data Durability
In a production Kafka platform, you design it so that broker failures do not cause data loss. You distribute brokers across multiple Availability Zones and replicate the data of each topic to multiple brokers. With a setting that defines the minimum number of replicas a write must reach before it is considered successful, you can maintain data consistency even if some brokers go down. If you place replicas across zones, you can also withstand the failure of an entire zone. You balance durability, availability, and performance by combining the number of replicas with the write acknowledgment threshold.
Security Through Authentication and Encryption
MSK provides multiple authentication and encryption methods. Communication between clients and brokers is encrypted with TLS to prevent eavesdropping. For authentication, you can choose a method that uses AWS IAM, in addition to a method based on user names and passwords. With IAM authentication, you can control read and write permissions per topic with IAM policies and integrate with the AWS role-based mechanism. Data at rest is also encrypted, and the keys are managed. You design who can write to and read from which topics with least privilege, narrowing access to the data flowing through the streaming platform. You select the authentication method according to your requirements.
Cost Optimization with Tiered Storage
Kafka retains data for a set period, but the longer the retention period, the more it strains broker storage and the more costs increase. With Tiered Storage, new data stays in fast broker storage and older data is moved automatically to a low-cost tier. This keeps down the cost of retaining data for long periods, and you can use the data for reprocessing that goes back into the past. Because you do not have to keep adding broker storage, capacity planning also becomes easier. You separate frequently accessed recent data and rarely referenced historical data into tiers to balance retention and costs.
Monitoring and Scaling
You continuously monitor the health of a Kafka cluster with multiple metrics. With CloudWatch metrics, you track broker load, disk usage, and consumer lag, and detect anomalies. If you need more detailed Kafka-specific metrics, you enable Prometheus-compatible monitoring to collect them. As load grows, you scale by adding brokers to expand processing capacity or by expanding storage. To keep free disk space from running out, you set up usage alerts in CloudWatch and prepare with automatic expansion, adding storage, and shortening the retention period.
Summary
MSK is a service that maintains Kafka ecosystem compatibility while delegating the operational management of Apache Kafka to AWS. With MSK Serverless, you leave cluster capacity management to AWS, and with MSK Connect, you run Kafka Connect connectors in a managed way. With Tiered Storage, cold data is moved automatically to a low-cost storage tier, which keeps broker storage costs down.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon MSK official page
- Amazon Kinesis official page
- Amazon S3 official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.