Running Spark Jobs Serverlessly with Amazon EMR Serverless - Big Data Processing Without Cluster Management
Learn about running Spark and Hive jobs with EMR Serverless, job run design, and cost optimization strategies.
EMR Serverless Overview
EMR Serverless is a big data processing service that runs Spark and Hive jobs serverlessly, automatically scaling workers within the maximum capacity configured for the application. While EMR on EC2 requires configuring instance types, node counts, and Auto Scaling settings, EMR Serverless automatically provisions resources when you submit a job. Cluster patching and version upgrades are also unnecessary, allowing you to focus operational effort entirely on job development.
Auto-Scaling and Pay-Per-Use Model
By default, an EMR Serverless application starts automatically when a job is submitted and stops automatically after 15 minutes of idle time. When submitting a job, you specify vCPU and memory limits for drivers and executors, but during execution the number of executors increases and decreases based on data volume. Billing depends on the amount of resources, such as vCPU and memory, that jobs use. This mechanism eliminates the waste of keeping clusters running for batch ETL jobs that run only once per day, making it easier to keep costs down for low-utilization workloads compared to maintaining always-on clusters with EMR on EC2.
Application Design and Hive Integration
EMR Serverless applications are created by selecting either a Spark or Hive runtime. For Spark applications, you place PySpark scripts in S3 and execute them via job runs. For Hive applications, you write ETL processing in HiveQL scripts and use the Glue Data Catalog as the metastore. Configuring pre-initialized workers avoids cold starts at job launch. You can individually specify vCPU and memory for job run drivers and executors, allowing resource allocation tailored to job characteristics. Storing S3 data in Parquet format with partitioning can reduce the amount of data that queries read.
Choosing Between EMR on EC2 and Glue
Similar options to EMR Serverless include EMR on EC2 and Glue, and the choice depends on workload characteristics. EMR on EC2 is advantageous when features not supported by Serverless are needed (GPU instances, Presto/Trino clusters) or when cluster utilization is high enough to leverage Reserved Instance discounts. Glue ETL allows building ETL pipelines with a visual editor and excels at Data Catalog integration and job bookmarks (resume capability), but offers limited access to Spark tuning parameters. As a guideline: use EMR Serverless when jobs can be completed with standard Spark/Hive features and cluster utilization is low; use EMR on EC2 when frameworks other than Spark (Flink, HBase) are needed; choose Glue when no-code visual ETL is the priority.
Design Best Practices and Pitfalls
Three key design points ensure stable operation with EMR Serverless. First, set the maximum capacity of the application to prevent cost increases from unexpected scaling. Without a maximum capacity for the application, a job that keeps generating massive executors due to data skew can lead to unexpected charges. Second, pre-initialized workers suit cases where job frequency is high. Pre-initialized capacity is released when the application stops. Third, plan a compaction strategy when using Iceberg tables. Accumulated small files increase Spark task counts, making jobs take longer to start. Incorporating periodic compaction into job pipelines maintains query performance.
EMR Serverless Pricing
EMR Serverless uses pay-per-use pricing based on the amount of resources used. For sporadic batch processing, it makes it easier to keep costs down than always-on clusters with EMR on EC2. Pre-initialized workers incur charges even when idle, so enable or disable them based on job frequency. Set resource limits on job runs to control runaway job costs, and use timeouts for automatic termination.
Summary
EMR Serverless is a service that runs Spark and Hive jobs without cluster management. Pay-per-use pricing and automatic stopping when idle keep costs down, and pre-initialized workers avoid cold starts. Using the Glue Data Catalog as a metastore enables ETL processing on Parquet data in S3. Setting resource limits and managing pre-initialized workers are the main points for keeping costs down.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon EMR Serverless official page
- Amazon S3 official page
- AWS Glue official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.