Running Spark Jobs Serverlessly with Amazon EMR Serverless - Big Data Processing Without Cluster Management
Learn about running Spark and Hive jobs with EMR Serverless, job run design, and cost optimization strategies.
EMR Serverless Overview
EMR Serverless is a big data processing service that runs Spark and Hive jobs serverlessly, auto-scaling up to 400 vCPU of worker resources. While EMR on EC2 requires configuring instance types, node counts, and Auto Scaling settings, EMR Serverless automatically provisions resources when you simply submit a job. Cluster patching and version upgrades are also unnecessary, allowing you to focus operational effort entirely on job development.
Auto-Scaling and Pay-Per-Use Model
The core value of EMR Serverless is that resources exist only during job execution and scale to zero upon completion. When submitting a job, you specify vCPU and memory limits for drivers and executors, but during execution the number of executors increases and decreases at per-second granularity based on data volume. Billing is calculated from actual vCPU-seconds and memory GB-seconds consumed, and cost drops to zero when the job finishes. This mechanism eliminates the waste of keeping clusters running for batch ETL jobs that run only once per day, achieving significant cost savings for low-utilization workloads compared to maintaining always-on clusters with EMR on EC2.
Application Design and Hive Integration
EMR Serverless applications are created by selecting either a Spark or Hive runtime. For Spark applications, you place PySpark scripts in S3 and execute them via job runs. For Hive applications, you write ETL processing in HiveQL scripts and use the Glue Data Catalog as the metastore. Configuring pre-initialized workers avoids cold starts at job launch, enabling jobs to begin within seconds. You can individually specify vCPU and memory for job run drivers and executors, allowing resource allocation tailored to job characteristics. Storing S3 data in Parquet format with partitioning optimizes query performance. For Spark use cases and practical insights, related books (Amazon) are a helpful reference.
Choosing Between EMR on EC2 and Glue
Similar options to EMR Serverless include EMR on EC2 and Glue, and the choice depends on workload characteristics. EMR on EC2 is advantageous when features not supported by Serverless are needed (GPU instances, custom AMIs, Presto/Trino clusters) or when cluster utilization is high enough to leverage Reserved Instance discounts. Glue ETL allows building ETL pipelines with a visual editor and excels at Data Catalog integration and job bookmarks (resume capability), but offers limited access to Spark tuning parameters, making EMR Serverless more flexible for large-scale Spark SQL analytics workloads. As a guideline: use EMR Serverless when jobs can be completed with standard Spark/Hive features and cluster utilization is low; use EMR on EC2 when frameworks other than Spark (Flink, HBase) are needed; choose Glue when no-code visual ETL is the priority.
Design Best Practices and Pitfalls
Three key design points ensure stable operation with EMR Serverless. First, always set maximum resource limits on job runs to prevent cost runaway from unbounded scaling. A job without limits that generates massive executors due to data skew can lead to unexpected charges. Second, enable pre-initialized workers only when job frequency is high (multiple times per hour). Since they incur charges while idle, enabling them for daily batch jobs results in 23 hours of idle costs that negate Serverless benefits. Third, plan a compaction strategy when using Iceberg tables. Accumulated small files cause Spark task counts to explode, prolonging job startup times. Incorporating periodic OPTIMIZE commands into job pipelines maintains query performance.
EMR Serverless Pricing
EMR Serverless uses pay-per-use pricing based on vCPU-hours and memory GB-hours. vCPU costs approximately 0.052 USD per hour, and memory costs approximately 0.0057 USD per GB-hour. Since there are no charges when jobs are not running, cost efficiency significantly improves over EMR on EC2 for sporadic batch processing. Pre-initialized workers incur charges even when idle, so enable or disable them based on job frequency. Set resource limits on job runs to control runaway job costs, and use timeouts for automatic termination. The break-even point with EMR on EC2 tends to favor Serverless when cluster utilization falls below approximately 30%.
Summary
EMR Serverless is a service that runs Spark and Hive jobs without cluster management. Pay-per-use pricing eliminates idle costs, and pre-initialized workers avoid cold starts. Using the Glue Data Catalog as a metastore enables efficient ETL processing on Parquet data in S3. It is more cost-effective than EMR on EC2 in environments where cluster utilization falls below 30%, and proper management of resource limits and pre-initialized workers is key to cost optimization.