Amazon Kinesis
A family of fully managed services for collecting, processing, and analyzing large volumes of streaming data in real time
Overview
Amazon Kinesis is a family of services for processing real-time streaming data at scale. Kinesis Data Streams is the core service, ingesting millions of records per second with low latency and allowing multiple consumers to process data in parallel. Kinesis Data Firehose builds code-free ETL pipelines that automatically deliver ingested data to S3, Redshift, OpenSearch, and other destinations. It's adopted for use cases where batch processing can't keep up, such as IoT device telemetry, web application clickstreams, and real-time financial transaction analysis.
Choosing Between Data Streams and Data Firehose
Kinesis Data Streams is a primitive streaming platform where throughput is controlled per shard, with each shard providing 1 MB/sec write and 2 MB/sec read capacity. You implement consumer-side logic freely, making it suitable for real-time processing with Lambda or custom applications. Data Firehose, on the other hand, is a managed service that automates data transformation (via Lambda) and delivery to destinations like S3, Redshift, and OpenSearch, with no shard management required. The decision criteria is straightforward: choose Data Streams when sub-200ms latency is required and you need fine-grained control over processing logic, and Firehose when delays of tens of seconds to minutes are acceptable and the goal is simply delivering data to a destination. Data Streams also supports multiple consumers reading from the same stream independently, which Firehose does not offer.
Shard Design and Data Retention Considerations
Shard count directly determines a stream's throughput capacity and cost. Each shard handles 1 MB/sec ingest and 2 MB/sec read, so a workload producing 10 MB/sec of data needs at least 10 shards. On-demand mode automatically adjusts shard count based on traffic but charges based on peak throughput, making provisioned mode 50% or more cheaper for stable workloads. Partition key design is critical - a poorly chosen key can create hot shards where one shard receives disproportionate traffic while others sit idle. Using a high-cardinality key (such as a device ID or user ID) ensures even distribution. Data retention defaults to 24 hours but can be extended up to 365 days, which is a significant advantage over Azure Event Hubs where Standard tier retention maxes out at 7 days. Data analytics books on Amazon provide further reading on stream processing architecture. Enhanced Fan-Out gives each consumer a dedicated 2 MB/sec read throughput but incurs per-consumer charges, so evaluate cost-effectiveness when you have three or more consumers.
Building a Real-Time Dashboard
A classic Kinesis pattern is building real-time dashboards. Web application access logs are sent to Kinesis Data Streams, aggregated in real time by Lambda, written to DynamoDB, and served to the frontend via API Gateway. This architecture delivers sub-second visibility into metrics like active users, error rates, and transaction volumes. For the ingestion layer, the Kinesis Producer Library (KPL) batches records to maximize throughput efficiency, while the Kinesis Client Library (KCL) handles consumer-side concerns like shard assignment and checkpointing. An alternative approach uses Data Firehose to deliver raw data to S3, then queries it with Athena for near-real-time analytics without managing any compute infrastructure. When choosing between these patterns, consider whether you need true real-time updates (Data Streams + Lambda) or whether minute-level freshness (Firehose + Athena) is sufficient for your use case.