Real-Time Data Streaming - Achieving Instant Data Processing with Amazon Kinesis
Explains design patterns for real-time data pipelines that combine Kinesis Data Streams and Data Firehose. Introduces shard design, buffering, Lambda transformations, and delivery to S3 / Redshift / OpenSearch.
The Importance of Real-Time Data Streaming
In modern business, large volumes of data are generated in real time, such as sensor data from IoT devices, clickstreams from web applications, and transaction logs from financial transactions. By processing / analyzing this data immediately, you can create value such as anomaly detection, real-time dashboards, and personalized recommendations. Amazon Kinesis is a fully managed service that collects / processes / analyzes streaming data in real time. Building an equivalent streaming platform on-premises commonly involves operating an Apache Kafka cluster, but this brings operational tasks such as managing brokers and rebalancing partitions. Because Kinesis takes on this operational work on the service side, it becomes easier to focus on the business logic of data streaming.
The Composition of the Kinesis Family
The main streaming services related to Amazon Kinesis are the following four, and the ones that currently carry the Kinesis name are Data Streams and Video Streams. Kinesis Data Streams is the foundation for real-time processing by custom applications, and it controls throughput in units of shards. Amazon Data Firehose (formerly Kinesis Data Firehose) is a delivery service that automatically delivers streaming data to destinations such as S3, Redshift, and OpenSearch. Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) analyzes streaming data in real time with Apache Flink (the SQL application version was discontinued in January 2026). Kinesis Video Streams specializes in ingesting and processing video streams. An example of creating an on-demand Kinesis Data Streams stream with the CLI: the stream is created with aws kinesis create-stream --stream-name click-stream --stream-mode-details StreamMode=ON_DEMAND. Then the Lambda event source mapping is set up: aws lambda create-event-source-mapping --function-name process-clicks --event-source-arn arn:aws:kinesis:ap-northeast-1:123456789012:stream/click-stream --starting-position LATEST --batch-size 100 --parallelization-factor 2 enables parallel processing.
Serverless Stream Processing Through Integration with Lambda
The integration of Kinesis Data Streams and Lambda is a representative pattern for serverless real-time data processing. Lambda automatically polls records from Kinesis shards and invokes the processing function in batches. With Enhanced Fan-Out, dedicated throughput (2 MB/sec per shard) is reserved for each consumer, and multiple consumers can process in parallel without affecting one another. With Lambda's event source mapping, you can finely control the batch size, batch window, parallelization factor, and more, and you can optimize the latency and throughput of processing. Error handling features such as retries when errors occur, forwarding to a dead-letter queue, and batch splitting by bisect are also provided as standard. With on-premises Kafka + consumer applications, you need to implement these features yourself, which increases the development and operational burden.
Scalability and Cost Efficiency
Kinesis Data Streams scales in units of shards, and each shard provides a throughput of 1 MB/sec for writes and 2 MB/sec for reads. In on-demand mode, the number of shards is adjusted automatically according to traffic. Amazon Data Firehose is pay-as-you-go. There are no minimum fees or setup costs. Data compression and transformation can be performed within Firehose. The data retention period is 24 hours by default and can be extended up to 365 days, which also supports reprocessing and replay use cases.
Typical Use Cases and Deciding When to Apply It
Real-time streaming delivers value in situations where data needs to be processed / judged at a timing close to the moment it is generated. Typical examples include real-time monitoring that immediately visualizes access conditions and system status, mechanisms that detect fraudulent transactions on the spot, and recommendations that respond immediately to user behavior. On the other hand, for daily report creation or processing where aggregating in bulk is sufficient, batch processing is simpler and more economical. The required immediacy becomes the criterion for deciding whether to use streaming. Determining whether real-time capability is truly necessary and applying streaming only to the parts that need it leads to a design that avoids excessive complexity.
Summary - Choosing a Real-Time Streaming Platform
As a fully managed platform for real-time data streaming, Amazon Kinesis can handle the work from collection through processing, analysis, and delivery. With automatic scaling through on-demand mode and pay-as-you-go pricing, it can support a range from small PoCs to large-scale production workloads. For organizations considering building a real-time data processing platform, Kinesis is one of the candidates.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
If this page and the official documentation disagree, treat the official documentation as authoritative.