Amazon EMR now supports native data quality checks in Apache Spark jobs

Amazon EMR provides native data quality validation within Apache Spark jobs, allowing data engineering teams to catch bad data inline before it reaches downstream systems without building or maintaining separate validation infrastructure. Based on AWS Glue and available at no extra cost, it offers a consistent experience across both engines using the same declarative rule language and APIs, and integrates with Apache Iceberg, Apache Hudi, and Delta Lake in EC2, EKS, and serverless deployment modes. Users can create validation rules in Spark scripts or reference rules stored in the AWS Glue Data Catalog, applying them post-load, during transformation, or post-processing. With over 30 rule types-including dynamic rules and ML-based anomaly detection for tracking drift over time without fixed thresholds-users can choose actions such as pipeline failure, filtering invalid records, or routing them to quarantine. Results are generated at the dataset, rule, and row levels, making it easy to identify records or rules causing quality issues. Amazon EMR Data Quality is available on all AWS commercial, AWS GovCloud (US), and AWS China regions via AWS Runtime for Apache Spark (emr-spark-8.1.0). To get started, access the Amazon EMR Data Quality documentation and add DQDL rules to your EMR workload Spark scripts.

Why it matters

This update benefits data engineering teams that need to effectively manage data quality. Amazon EMR's data quality capabilities simplify in-pipeline validation, helping ensure data quality without requiring separate infrastructure