Amazon Kinesis のアイコン

Amazon Kinesis Popular2013年〜

A service for collecting, processing, and analyzing real-time streaming data

About 2 min read

What It Does

Amazon Kinesis is a family of services for collecting, processing, and analyzing real-time streaming data at scale. Kinesis Data Streams handles stream data collection and processing, Kinesis Data Firehose delivers stream data to S3, Redshift, and other destinations, and Kinesis Data Analytics provides SQL/Flink-based analysis of stream data.

Use Cases

Used for real-time processing of sensor data from IoT devices, website clickstream analysis, real-time log aggregation and analysis, real-time financial transaction monitoring, and game player behavior analysis.

Everyday Analogy

Think of it like a conveyor belt. Products (data) flowing continuously from a factory (data source) are carried on a conveyor belt (stream), where they undergo inspection (processing), sorting (analysis), and packaging (storage) in an assembly-line fashion.

What Is Kinesis?

Amazon Kinesis is a family of services for real-time data processing. While batch processing 'accumulates data and processes it all at once,' Kinesis enables stream processing that 'processes data the moment it's generated.' It has the scalability to handle millions of records per second.

Kinesis Data Streams and Firehose

Kinesis Data Streams is a service for processing stream data with custom applications. You control throughput by adjusting the number of shards and process data with Lambda or custom consumers. Kinesis Data Firehose is a service that automatically delivers stream data to S3, Redshift, OpenSearch, Splunk, and more. You can configure data transformation and delivery without writing code. To deepen your practical knowledge of Kinesis Data Streams and Firehose, technical books on Amazon are helpful.

Getting Started

Select 'Create data stream' in the Kinesis console and specify a stream name and capacity mode (on-demand or provisioned). Send data using the AWS SDK's PutRecord API and configure Lambda as a consumer to complete your real-time processing pipeline. With Firehose, you can accumulate data in S3 without writing any code.

Things to Watch Out For

  • Data Streams retention period defaults to 24 hours and can be extended up to 365 days, but costs increase with longer retention
  • Choose Firehose for simple data delivery, and Data Streams when custom processing is needed
共有するXB!

Related Services

AWS Lambda iconAWS LambdaA serverless compute service that runs code without server managementAmazon S3 iconAmazon S3A scalable object storage serviceAmazon Redshift iconAmazon RedshiftA cloud data warehouse service for high-speed analytics on petabyte-scale dataAmazon Managed Service for Apache Flink iconAmazon Managed Service for Apache FlinkA managed stream processing service for running Apache Flink applications

Related Articles

Building Blockchain Networks - Leveraging Distributed Ledgers with Amazon Managed Blockchain and QLDBExplains how to build blockchain networks with Amazon Managed Blockchain and use Amazon QLDB as a verifiable ledger database. Covers practical use cases such as supply chain management and ensuring transparency in financial transactions.Building a Real-Time Data Pipeline with Amazon Kinesis - Choosing Between Data Streams and Data FirehoseIngest real-time data with Data Streams and automatically deliver it to S3, Redshift, and OpenSearch with Data Firehose. This article explains how to build a streaming pipeline with shard design and on-demand mode selection.Real-Time Stream Processing with Amazon Managed Service for Apache Flink - Stateful Processing and Window AggregationRun real-time stream processing with SQL or Java/Python applications in a fully managed Apache Flink environment. This article covers design patterns for window aggregation, pattern detection, and Kinesis/MSK integration.Managed Kafka Streaming - Building Large-Scale Real-Time Data Pipelines with Amazon MSKLearn how to build a fully managed Kafka cluster with Amazon MSK (Managed Streaming for Apache Kafka) and when to choose it over Kinesis. This article covers design patterns for large-scale real-time data streaming infrastructure.Marketing Communication - Multi-Channel Customer Engagement with Amazon PinpointLearn how to build multi-channel marketing with Amazon Pinpoint and optimize email delivery with SES integration. This article covers practical approaches to customer engagement strategies that unify push notifications, SMS, email, and voice.

More in This Category

Amazon Athena iconAmazon Athena PopularA serverless query service that lets you analyze data in S3 using standard SQLAWS Clean Rooms iconAWS Clean Rooms NewA privacy-preserving data collaboration service that enables multiple organizations to jointly analyze data without sharing itAWS Data Exchange iconAWS Data Exchange SpecializedA data marketplace that makes it easy to find, subscribe to, and use third-party datasetsAmazon DataZone iconAmazon DataZone NewA data management service that unifies data discovery, sharing, and governanceAmazon EMR iconAmazon EMR SpecializedA managed service for running big data frameworks like Apache Spark, Hive, and PrestoAmazon EMR Serverless iconAmazon EMR Serverless SpecializedA serverless service for running Spark and Hive jobs without cluster managementAWS Entity Resolution iconAWS Entity Resolution NewA data matching service that reconciles and unifies records across different data sourcesAWS Glue iconAWS Glue PopularA serverless ETL service that automates data extraction, transformation, and loading

Similar Articles and Services

Real-Time Data Streaming - Instant Data Processing with Amazon KinesisLearn design patterns for real-time data pipelines combining Kinesis Data Streams and Data Firehose. This article covers shard design, buffering, Lambda transformations, and delivery to S3, Redshift, and OpenSearch.Designing Streaming Data Processing - Building Real-Time Data Pipelines with KinesisLearn streaming data processing design techniques with Amazon Kinesis, including building real-time data pipelines with Data Streams, Data Firehose, and Lambda integration.Amazon KinesisA family of fully managed services for collecting, processing, and analyzing large volumes of streaming data in real timeBuilding a Real-Time Data Pipeline with Amazon Kinesis - Choosing Between Data Streams and Data FirehoseIngest real-time data with Data Streams and automatically deliver it to S3, Redshift, and OpenSearch with Data Firehose. This article explains how to build a streaming pipeline with shard design and on-demand mode selection.Real-Time Stream Processing with Amazon Managed Service for Apache Flink - Stateful Processing and Window AggregationRun real-time stream processing with SQL or Java/Python applications in a fully managed Apache Flink environment. This article covers design patterns for window aggregation, pattern detection, and Kinesis/MSK integration.