Building a Real-Time Data Pipeline with Amazon Kinesis - Choosing Between Data Streams and Data Firehose
Real-time data is ingested with Data Streams and delivered automatically to S3 / Redshift / OpenSearch with Data Firehose. This article explains how to build a streaming pipeline through shard design and choosing when to use on-demand mode.
Overview of Kinesis
Kinesis is a family of services that provides collection / processing / analysis of real-time streaming data. Data Streams provides real-time processing with custom consumers, Data Firehose provides automatic delivery to S3 and Redshift, and Managed Apache Flink provides SQL/Flink processing of stream data.
Choosing Between Data Streams and Firehose
Data Streams manages throughput in units of shards, and you implement custom processing with Lambda or KCL (Kinesis Client Library). It is suited to real-time alerts that require sub-second latency and to cases where multiple consumers read the same stream. Data Firehose automatically buffers data from producers and delivers it to S3, Redshift, OpenSearch, and Splunk. You can apply data transformation (Lambda) and format conversion (Parquet) before delivery, and no consumer implementation is required. For aggregating logs and delivering them to analytics platforms, choosing Firehose is a good fit.
Data Streams Design Patterns
The number of shards in Data Streams is determined by throughput requirements. One shard provides 1 MB/second (1,000 records/second) for writes and 2 MB/second for reads. In on-demand mode, the number of shards is adjusted automatically, and in provisioned mode, you set it manually. Partition key design controls how data is distributed and prevents hot shards (skew toward specific shards). When you enable enhanced fan-out, dedicated read throughput of 2 MB/second is allocated to each consumer, and multiple consumers can read the same stream in parallel. KCL (Kinesis Client Library) automatically manages shard rebalancing and checkpoints and simplifies the development of consumer applications.
Optimizing Kinesis Pricing
Provisioned mode in Data Streams is billed by shard hours (about $0.015/hour) and PUT payload units (25KB units, about $0.014 per 1 million units) (as of August 2026). On-demand mode is billed by the volume of data ingested (about $0.08 per 1 GB) and the volume read, and is suited to workloads with fluctuating traffic. Data Firehose is billed only on the volume of data ingested (about $0.029 per 1 GB), and no shard management is required. Aggregating records by combining multiple small records into one PUT reduces the number of PUT payload units and lets you keep Data Streams costs down. The minimum data retention period is 24 hours, and extending it incurs additional charges, so you decide it according to your replay requirements.
Lambda Integration and Error Handling
When you process Data Streams with Lambda, the event source mapping manages retrieving records from shards and invoking batches. Failed batches are retried, but if processing keeps getting stuck on a specific record, subsequent records are held up. For this reason, you isolate only the problematic records by specifying a setting that splits the failed portion by binary search, a maximum number of retries, and a destination for failed records. Raising the number of concurrent executions per shard increases throughput, and order within the same shard is still preserved. Configuring these in advance lets you shorten the time that subsequent processing is stopped by the failure of a single record.
Firehose Transformation and Delivery Design
Data Firehose can process data before delivery. If you transform records with Lambda and store JSON in a columnar format such as Parquet, it becomes easier to make downstream analytic queries fast and inexpensive. With dynamic partitioning, you can route records to S3 destination paths according to their content and automatically build a data lake organized by date or attribute. If you balance latency and file size by tuning the buffer size and time, and move records that failed transformation or delivery to a dedicated error output destination, you can reprocess the failed records later.
Ordering Guarantees / Reprocessing / Retention Period
Because Data Streams preserves order per shard, data where order matters is consolidated into the same shard through partition key design. Data is retained in the stream for a certain period, and within the retention period, consumers can go back and reprocess past data (replay). Its advantage is that even if a consumer stops because of a failure, it can reread from the point where it missed data after recovery. However, because extending the retention period adds cost, you set a period that matches your reprocessing requirements. Designing processing to be idempotent on the assumption that duplicate processing can occur lets you avoid side effects during reprocessing.
Key Points for Monitoring and Operations
In a streaming platform, you monitor the metric that indicates consumer lag (the age of the retrieved records). If it keeps increasing, it is a sign that processing capacity is not keeping up with the ingestion volume, so you review the number of shards and the consumer parallelism. You also check whether write or read throttling is occurring, and if there are signs of hot shards, you improve the partition key design. By continuously tracking these metrics with dashboards and alarms, you can take action before delays and missed data become serious.
Summary
Kinesis is a processing platform for real-time streaming data. You implement custom real-time processing with Data Streams. With enhanced fan-out, multiple consumers can read the same stream in parallel. You automate delivery to S3 and Redshift with Data Firehose and keep costs down by reducing the number of PUT payload units through record aggregation. In on-demand mode, you can reduce the effort of managing the number of shards yourself.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon Kinesis official page
- AWS Lambda official page
- Amazon S3 official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.