Amazon Kinesis
A family of fully managed services for collecting, processing, and analyzing large volumes of streaming data in real time
Overview
Amazon Kinesis is a family of services for processing real-time streaming data at scale. Kinesis Data Streams is the core service, ingesting millions of records per second with low latency and allowing multiple consumers to process data in parallel. Kinesis Data Firehose builds code-free ETL pipelines that automatically deliver ingested data to S3, Redshift, OpenSearch, and other destinations. It's adopted for use cases where batch processing can't keep up, such as IoT device telemetry, web application clickstreams, and real-time financial transaction analysis.
Choosing Between Data Streams and Data Firehose
Kinesis Data Streams is a primitive streaming platform where throughput is controlled per shard, with each shard providing up to 1 MB/sec write and 2 MB/sec read capacity according to the AWS developer guide (as of October 2026). You implement consumer-side logic freely, making it suitable for real-time processing with Lambda or custom applications. Amazon Data Firehose, on the other hand, is a managed service that automates data transformation (via Lambda) and delivery to destinations like S3, Redshift, and OpenSearch, with no shard management required. AWS announced in February 2024 that it renamed Amazon Kinesis Data Firehose to Amazon Data Firehose. As a rule of thumb, choose Data Streams when you need fine-grained control over processing logic, and Firehose when the goal is simply delivering data to a destination. Data Streams also supports multiple consumers reading from the same stream independently, which Firehose does not offer.
Shard Design and Data Retention Considerations
In provisioned mode, you specify the number of shards, and the stream's capacity is the sum of the capacities of its shards. On-demand mode scales automatically with traffic, and the developer guide describes provisioned mode as suited for predictable traffic. A poorly chosen partition key can create hot shards where one shard receives disproportionate traffic while others sit idle. A high-cardinality key (such as a device ID or user ID) helps spread traffic across shards. Data retention defaults to 24 hours and can be extended up to 365 days, longer than the 7-day maximum of the Azure Event Hubs Standard tier in Microsoft's documentation (as of October 2026). Enhanced Fan-Out gives each consumer dedicated read throughput per shard, with separate charges for data retrieval and consumer-shard hours.
Building a Real-Time Dashboard
One Kinesis pattern is building real-time dashboards. Web application access logs are sent to Kinesis Data Streams, aggregated in real time by Lambda, written to DynamoDB, and served to the frontend via API Gateway. For the ingestion layer, the Kinesis Producer Library (KPL) aggregates and batches records to improve throughput, while the Kinesis Client Library (KCL) handles consumer-side concerns like shard assignment and checkpointing. An alternative approach uses Data Firehose to deliver raw data to S3, then queries it with Athena for near-real-time analytics without managing any compute infrastructure. When choosing between these patterns, consider whether you need true real-time updates (Data Streams + Lambda) or whether near-real-time analysis with Firehose + Athena is sufficient for your use case.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS Lambda official page
- Amazon S3 official page
- Amazon Redshift official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.