Amazon EMR Serverless
A serverless big data processing service that runs Apache Spark and Hive jobs without cluster management, charging only for the vCPU and memory time consumed during processing
Overview
Amazon EMR Serverless is a serverless big data processing service that runs Apache Spark and Hive jobs. It eliminates the need for cluster provisioning, scaling, and patching - you simply submit jobs and the required resources are automatically provisioned. After processing completes, resources are released and you are charged only for the vCPU-hours and memory GB-hours actually consumed. EMR Runtime for Apache Spark delivers up to 3.5x faster performance compared to open-source Spark at no additional cost.
Application and Job Execution Lifecycle
In EMR Serverless, you first create an application. An application is a container that defines a Spark or Hive runtime environment, specifying an EMR release version (emr-6.x, emr-7.x). Application states include CREATING, CREATED, STARTING, STARTED, STOPPING, STOPPED, and TERMINATED, and applications accept jobs only in the STARTED state. Jobs are submitted via the StartJobRun API, specifying the S3 path to a JAR file or Python script along with the entry point for Spark jobs. Once submitted, EMR Serverless automatically provisions workers and begins processing. Configuring pre-initialized capacity keeps the driver and workers initialized and ready, so jobs start in seconds. Pre-initialized workers are billed even while the application is idle. After job completion, workers beyond the pre-initialized capacity are released. Auto-stop is enabled by default (as of October 2026): after 15 minutes of idle time, the application itself transitions to STOPPED and releases its pre-initialized capacity.
Worker Configuration and Cost Control
EMR Serverless workers are defined by vCPU and memory combinations that can be customized to match job characteristics. Per the user guide defaults (as of October 2026), Spark drivers and executors each get 4 cores and 14 GB of memory, but memory-intensive workloads can be tuned to configurations like 4 vCPU / 30 GB. Setting a maximum worker count caps resource consumption per job. The key to cost control is the application-level maximum resource setting. The maxCapacity parameter specifies vCPU, memory, and disk ceilings, ensuring total resources across all jobs never exceed these limits. Charges are based on the aggregate vCPU, memory, and storage used from the time workers are ready to run until they stop, rounded up to the nearest second with a 1-minute minimum (pricing page, as of October 2026). By default, job logs are stored in EMR managed storage for up to 30 days, and S3 buckets or CloudWatch are also available as log destinations. Application UIs such as the Spark UI are available when managed storage logging is enabled.
S3 Data Lake Integration Patterns
One EMR Serverless architecture uses S3 as a data lake with Spark jobs performing ETL processing. Input data resides in S3 as Parquet, ORC, CSV, or JSON files, with Glue Data Catalog serving as the metastore. Spark SQL can directly reference Glue Data Catalog tables, with partition pruning and predicate pushdown applied automatically. Apache Iceberg table read/write support enables building data pipelines with ACID transactions, schema evolution, and time-travel queries. Combining EMR Serverless with Step Functions lets you orchestrate daily ETL pipelines. A Step Functions task state calls the StartJobRun API, waits for job completion, then proceeds to the next step. Adding an EventBridge Scheduler for daily triggers gives you a scheduled batch processing pipeline built on serverless services.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
If this page and the official documentation disagree, treat the official documentation as authoritative.