Amazon EMR Serverless now supports terabyte-scale shuffle

Amazon EMR Serverless now raises the per-job shuffle limit from 200 GB to 1 TB, supporting terabyte-scale shuffle operations. This allows data engineers and scientists to run open-source big data analytics frameworks without configuring, managing, or scaling clusters or servers, benefiting enterprise customers performing complex shuffle-requiring operations like large-scale table joins across terabyte datasets and complex aggregations on high-cardinality data. Added spill support offloads memory-intensive operations to disk, improving job reliability and success rates for demanding analytics workloads. The feature is available in Amazon emr-7.14 and emr-spark-8.1 or later, across the 18 AWS regions where Amazon EMR Serverless is offered.

Why it matters

This update impacts data engineers and data scientists running big data analytics frameworks in production, especially those needing to process terabyte-scale datasets.