AWS First Steps
LanguageEnglish

Amazon Managed Service for Apache Flink

A fully managed service for running Apache Flink applications that processes and analyzes real-time streaming data from Kinesis Data Streams and MSK with low latency

About 4 min read

Overview

Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) is a fully managed stream processing service for running Apache Flink applications. Simply deploy Flink applications written in Java, Scala, or Python, and cluster provisioning, scaling, patching, and checkpoint management are all automated. It uses Kinesis Data Streams, Amazon MSK, S3, and other services as sources and sinks to process events.

Flink Application Deployment and Scaling

AWS announced in August 2023 that it renamed Amazon Kinesis Data Analytics to Amazon Managed Service for Apache Flink. To deploy an application to Managed Flink, upload a Flink application JAR file or ZIP archive to S3 and specify the entry point class and runtime environment in the application configuration. Apache Flink Studio also allows interactive SQL query execution on Apache Zeppelin notebooks for real-time exploration and analysis of streaming data. Scaling is performed in KPU (Kinesis Processing Unit) increments, and the developer guide states that a single KPU provides 1 vCPU and 4 GB of memory (as of October 2026). With automatic scaling, parallelism goes up or down based on the application's CPU usage, and the KPU count changes with it. When scaling, an application snapshot is taken and state is restored at the new parallelism level. While automatic scaling updates parallelism, the application experiences downtime. The Parallelism and ParallelismPerKPU settings should be aligned with the number of source shards or partitions.

Checkpointing and State Management

In Flink's stateful processing, intermediate state from window aggregations and session management is maintained within the application. Managed Flink periodically persists this state as checkpoints to S3, and upon failure, restores state from the checkpoint to resume processing. With the default configuration, the checkpoint interval is 60 seconds (as of October 2026), and setting the configuration type to CUSTOM lets you change it, though shorter intervals increase checkpoint I/O overhead. RocksDB (disk-based) is one state backend, and it keeps state that does not fit in memory on disk. The state backend type is among the Flink settings that you ask to modify through a support case. Savepoints are user-initiated snapshots used to carry over state during application upgrades or logic changes. In Managed Flink, you take a savepoint via the Snapshot API and start the new application version from that savepoint, upgrading stateful applications while carrying over their state.

Source and Sink Integration with Kinesis and MSK

Input sources for Managed Flink include Kinesis Data Streams and Amazon MSK (Managed Streaming for Apache Kafka). With a Kinesis source, Flink's Kinesis Consumer reads data in parallel per shard, and setting Flink's parallelism to match the shard count helps spread reads in parallel. With an MSK source, Flink's Kafka Consumer reads data from topic partitions. IAM authentication for MSK is supported, eliminating the need for access key management. Using Apache Flink sink connectors, applications can write to sinks (output destinations) such as S3, Kinesis Data Streams, MSK, and DynamoDB. The S3 sink uses Flink's bucketing feature to partition files by time or key, arranging data so that downstream Athena queries benefit from partition pruning. Watermark strategy configuration directly impacts event-time processing accuracy - failing to properly set the allowed lateness for late-arriving data will produce inaccurate window computation results.

References (Official AWS Resources)

The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.

If this page and the official documentation disagree, treat the official documentation as authoritative.

Share

Related Terms

Amazon KinesisAmazon MSKAmazon EMRAmazon OpenSearch ServiceAmazon S3

Related Services

Amazon KinesisAmazon MSKAmazon S3Amazon OpenSearch Service

Related Articles

Building an Apache Kafka Streaming Platform with Amazon MSK - Cluster Design and OperationsThis article covers designing managed Apache Kafka clusters, choosing between them and MSK Serverless, and data integration patterns with MSK Connect.Building RAG Applications with Amazon Bedrock Knowledge Bases - Implementing Retrieval-Augmented GenerationAutomatically index documents on S3 and unify search and generation with the RetrieveAndGenerate API. Covers chunking strategy selection and safety enforcement with Guardrails.Data Search and Analytics in Practice - Building Full-Text Search and Visualization with OpenSearchLearn about data search and analytics design with Amazon OpenSearch Service, including how to build an analytics foundation with full-text search, log analysis, and dashboard visualization.Building a Real-Time Data Pipeline with Amazon Kinesis - Choosing Between Data Streams and Data FirehoseReal-time data is ingested with Data Streams and delivered automatically to S3 / Redshift / OpenSearch with Data Firehose. This article explains how to build a streaming pipeline through shard design and choosing when to use on-demand mode.Real-Time Stream Processing with Amazon Managed Service for Apache Flink - Stateful Processing and Window AggregationReal-time stream processing with SQL or Java/Python applications runs in a fully managed Apache Flink environment. This article explains design patterns for window aggregation, pattern detection, and Kinesis/MSK integration.

Similar Terms and Articles

Real-Time Stream Processing with Amazon Managed Service for Apache Flink - Stateful Processing and Window AggregationReal-time stream processing with SQL or Java/Python applications runs in a fully managed Apache Flink environment. This article explains design patterns for window aggregation, pattern detection, and Kinesis/MSK integration.Amazon Managed Service for Apache FlinkA managed stream processing service for running Apache Flink applicationsReal-Time Data Streaming - Achieving Instant Data Processing with Amazon KinesisExplains design patterns for real-time data pipelines that combine Kinesis Data Streams and Data Firehose. Introduces shard design, buffering, Lambda transformations, and delivery to S3 / Redshift / OpenSearch.Designing Streaming Data Processing - Building Real-Time Data Pipelines with KinesisLearn streaming data processing design techniques with Amazon Kinesis, including building real-time data pipelines with Data Streams, Data Firehose, and Lambda integration.Managed Kafka Streaming - Building Large-Scale Real-Time Data Pipelines with Amazon MSKLearn how to build a fully managed Kafka cluster with Amazon MSK (Managed Streaming for Apache Kafka) and when to choose it over Kinesis. This article covers design patterns for large-scale real-time data streaming infrastructure.