Analytics and Search

Managed Kafka Streaming - Building Large-Scale Real-Time Data Pipelines with Amazon MSK

Learn how to build a fully managed Kafka cluster with Amazon MSK (Managed Streaming for Apache Kafka) and when to choose it over Kinesis. This article covers design patterns for large-scale real-time data streaming infrastructure.

約 6 分で読めます

Apache Kafka and Amazon MSK

Apache Kafka is the de facto standard for large-scale real-time data streaming, adopted by companies worldwide. It excels in use cases that demand processing millions of events per second, such as log aggregation, event sourcing, metrics collection, and stream processing. Amazon MSK is a fully managed service for Apache Kafka that automates cluster provisioning, configuration, patching, and monitoring. Running a Kafka cluster on-premises involves complex operational tasks including ZooKeeper management, broker scaling, partition rebalancing, disk capacity monitoring, and security patching. MSK handles all of these in a managed fashion while maintaining full compatibility with the Apache Kafka API, allowing you to migrate existing Kafka applications without code changes.

Building and Operating an Amazon MSK Cluster

MSK clusters are created within a VPC, with brokers distributed across multiple Availability Zones for high availability. MSK Serverless is a provisioning-free serverless option that automatically scales with traffic and charges only for what you use. MSK Provisioned lets you explicitly specify broker instance types and storage for predictable performance. MSK Connect is a managed implementation of Apache Kafka Connect that lets you deploy connectors to automatically stream data between AWS services such as S3, DynamoDB, OpenSearch, and RDS. It supports multiple authentication methods including IAM authentication, SASL/SCRAM, and mutual TLS authentication, with topic-level access control for fine-grained security. CloudWatch metrics and Prometheus-compatible open monitoring provide comprehensive cluster health visibility. To create an MSK Serverless cluster via CLI: aws kafka create-cluster-v2 --cluster-name streaming-cluster --serverless "{"clientAuthentication":{"sasl":{"iam":{"enabled":true}}},"vpcConfigs":[{"subnetIds":["subnet-abc","subnet-def"],"securityGroupIds":["sg-123"]}]}" creates a serverless Kafka cluster.

Choosing Between Amazon MSK and Kinesis Data Streams

MSK and Kinesis Data Streams are both real-time streaming services, but they differ in design philosophy. Kinesis is an AWS-native serverless streaming service with easy integration with Lambda, Firehose, and Data Analytics. It requires no provisioning, scales by adjusting shard counts, and its seamless integration with AWS services is its greatest advantage. MSK, on the other hand, provides full compatibility with the Apache Kafka ecosystem, letting you use existing Kafka applications, Kafka Streams, ksqlDB, Schema Registry, and other tools as-is. MSK is the best choice when you want to leverage Kafka's rich community ecosystem or when migrating from an on-premises Kafka cluster. MSK also supports unlimited data retention (dependent on storage capacity), compared to Kinesis's maximum of 365 days, making it suitable for use cases requiring long-term retention. To broaden your data analytics knowledge, specialized books on Amazon can also be useful.

Stream Processing Architecture Design Patterns

A stream processing architecture centered on MSK uses a publish/subscribe model where producers publish events to Kafka topics and consumers process them in real time. The Kafka Streams library lets you perform stream joins, aggregations, and window processing within your application. Using MSK Connect, you can build event-driven architectures that stream change data capture (CDC) from databases to Kafka topics and propagate changes to downstream microservices in real time. For data lake integration with S3, the S3 Sink Connector in MSK Connect automatically archives data in Parquet or Avro format for analysis with Athena or Redshift Spectrum. AWS Glue Schema Registry helps manage schema evolution and maintain data contracts between producers and consumers.

MSK Pricing

A provisioned cluster with kafka.m5.large costs approximately $151/month per broker, with a minimum 3-broker configuration costing approximately $453/month. Storage costs approximately $0.10 per GB/month. MSK Serverless is billed by cluster hours (approximately $0.75/hour) and partition hours. Compared to Kinesis Data Streams (approximately $0.015 per shard hour), choose MSK when Kafka ecosystem compatibility is needed, and Kinesis when you prioritize AWS-native integration.

Summary - Choosing a Managed Kafka Streaming Platform

Amazon MSK provides a large-scale real-time data streaming platform as a fully managed Apache Kafka service. Full compatibility with the Kafka API makes it easy to migrate existing applications, and MSK Serverless enables serverless operation. For new development that prioritizes AWS-native integration, choose Kinesis; for leveraging the Kafka ecosystem or migrating from existing Kafka, MSK is the optimal strategy. Combining MSK Connect for external system integration with Kafka Streams for stream processing lets you build end-to-end real-time data pipelines.

Related Services

Amazon MSKA fully managed Apache Kafka service for real-time streaming data processingAmazon KinesisA service for collecting, processing, and analyzing real-time streaming dataAWS GlueA serverless ETL service that automates data extraction, transformation, and loading

Related Articles

Designing Streaming Data Processing - Building Real-Time Data Pipelines with KinesisLearn streaming data processing design techniques with Amazon Kinesis, including building real-time data pipelines with Data Streams, Data Firehose, and Lambda integration.Real-Time Data Streaming - Instant Data Processing with Amazon KinesisLearn design patterns for real-time data pipelines combining Kinesis Data Streams and Data Firehose. This article covers shard design, buffering, Lambda transformations, and delivery to S3, Redshift, and OpenSearch.Event-Driven Architecture - Building Loosely Coupled Systems with Amazon EventBridgeLearn how to build event-driven architectures using Amazon EventBridge.

More on This Topic

Practical Use Cases for Amazon Quick - Department-Specific Scenarios and Workflow Automation Design PatternsExplore concrete use cases for sales, IT, and finance departments, along with design patterns for notifications, approvals, and multi-step workflows using Quick Flows.BI Dashboard Visualization - Building a Data-Driven Decision Platform with Amazon QuickSightExplains how to build interactive BI dashboards with Amazon QuickSight and a serverless data analytics platform with Athena integration. Covers high-speed visualization with the SPICE engine and practical methods for sharing insights across the organization.Building Blockchain Networks - Leveraging Distributed Ledgers with Amazon Managed Blockchain and QLDBExplains how to build blockchain networks with Amazon Managed Blockchain and use Amazon QLDB as a verifiable ledger database. Covers practical use cases such as supply chain management and ensuring transparency in financial transactions.Privacy-Preserving Data Collaboration with AWS Clean RoomsRun joint analysis across multiple companies without sharing or copying data. Learn about aggregation rules for preventing individual identification and Cryptographic Computing for encrypted analysis.Customer Identity Unification - Resolving Scattered Customer Data with AWS Entity ResolutionLearn how to perform entity resolution (record matching) on customer data using AWS Entity Resolution. This article covers ML-based matching, rule-based matching, privacy protection, and integration with Clean Rooms.Leveraging Third-Party Data with AWS Data Exchange - Data Procurement and Subscription ManagementProcure third-party data products via Marketplace and build automated delivery pipelines to S3. This article also covers how to productize and monetize your own data.Building a Data Lake with Amazon S3 and Lake Formation - Design Patterns and GovernanceExplore data lake design patterns using S3 as the storage foundation and Lake Formation for fine-grained access control. This article also covers ETL pipelines and cost optimization.Data Lake Governance - Centralized Access Control with AWS Lake FormationLearn about building, access control, and governance for data lakes using AWS Lake Formation. This article covers fine-grained column-level and row-level permission management for S3-based data lakes, along with Glue and Athena integration.

Similar Articles and Services

Building an Apache Kafka Streaming Platform with Amazon MSK - Cluster Design and OperationsDesign a managed Apache Kafka cluster, learn when to choose MSK Serverless, and explore data integration patterns with MSK Connect.Amazon MSKA fully managed streaming service that provides Apache Kafka cluster provisioning and operations on AWS, with full Kafka-compatible API supportGetting Started with Event Streaming on Amazon MSK Serverless - Building a Zero-Ops Kafka EnvironmentLearn how to build a cluster-free Kafka environment with IAM authentication and simplify event streaming through integration with EventBridge Pipes.Designing Streaming Data Processing - Building Real-Time Data Pipelines with KinesisLearn streaming data processing design techniques with Amazon Kinesis, including building real-time data pipelines with Data Streams, Data Firehose, and Lambda integration.