Amazon EMR
A managed big data platform that runs open-source frameworks like Apache Spark, Hive, and Presto to process and analyze petabyte-scale data
Overview
Amazon EMR (Elastic MapReduce) is a fully managed big data processing service that runs open-source frameworks including Apache Spark, Hadoop, Hive, Presto, and HBase. In addition to the traditional model of building clusters on EC2 instances, it offers three deployment options: EMR on EKS for running on existing EKS clusters, and EMR Serverless for serverless execution. By leveraging S3 as a data lake and separating compute from storage, you can terminate clusters when processing is complete to minimize costs.
Three Deployment Models and Selection Criteria
EMR on EC2 offers fine-grained control over instance types, cluster configuration, and bootstrap actions. It's ideal when you need specific Spark or Hive versions, custom AMIs, or local storage via HDFS. EMR on EKS runs Spark jobs on existing EKS clusters, leveraging Kubernetes resource management and multi-tenancy capabilities. It suits organizations where multiple teams share a single cluster while isolating jobs through namespaces. EMR Serverless, which became generally available in 2022, removes the need to build and manage clusters. You simply submit jobs and the required resources are automatically provisioned and released upon completion. As a rule of thumb: choose EC2 when you need fine-tuned configuration, EKS when you need Kubernetes integration, and Serverless when you want to minimize operational overhead.
Spark Job Optimization and Cost Control
When running Spark on EMR, instance configuration and partition design are factors that affect performance and cost. Spot Instances, which you can use for worker nodes, are available at up to a 90% discount compared to On-Demand prices (as of October 2026). Spark's shuffle partition count (spark.sql.shuffle.partitions) should be tuned from the default of 200 based on data volume - targeting 128-256 MB per partition strikes a good balance between memory efficiency and task parallelism. For reads from S3, using columnar formats like Parquet or ORC and partitioning data by date or category to enable partition pruning is critical. EMR Runtime for Apache Spark is described as running up to 4.5x faster than open-source Spark (product page, as of October 2026).
Data Lake Architecture and When to Choose Glue Instead
EMR and Glue can both execute Spark-based ETL processing, but they serve different roles. Glue is serverless with low operational overhead and integrates the Glue Data Catalog for metadata management and crawlers for automatic schema detection. It's well-suited for routine ETL pipelines and use cases where Data Catalog integration is important. EMR, on the other hand, supports diverse frameworks beyond Spark - including Hive, Presto, HBase, and Flink - and offers greater flexibility in cluster configuration, making it better for complex analytical processing and interactive queries. For example, you can handle daily routine ETL with Glue while using EMR for ad-hoc large-scale analytics and machine learning preprocessing. For building a data lake, one architecture is to store raw data in S3, detect schemas with Glue crawlers and register them in the Data Catalog, then query from EMR or Athena. Using table formats like Apache Iceberg or Delta Lake on EMR enables ACID transactions and time-travel queries.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS Glue official page
- Amazon Athena official page
- Amazon S3 official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.