AWS Glue Data Quality now supports distribution statistics for data profiling
AWS Glue Data Quality adds a new Distribution Analyzer to generate frequency distribution profiles, integrating directly with DQDL rulesets for histograms and value distributions, stored in S3 for future querying.
AWS Glue Data Quality now supports a new Distribution Analyzer that generates frequency distribution profiles for data. Using this analyzer in the Data Quality Definition Language (DQDL), users can generate histograms for numeric columns and value distributions for categorical, date, and boolean columns. With support for custom bin counts, users can explore the shape and patterns of their data at the granularity that matters most to their use case. Understanding data distribution is foundational for building reliable data pipelines. Distribution statistics help quickly identify skewness, outliers, and unexpected patterns across datasets without writing custom code. The capability integrates directly with existing DQDL rulesets, allowing distribution profiling alongside current data quality checks in a single evaluation run. Distribution statistics are stored in Amazon S3 for future querying through services like Amazon Athena and are also exposed through APIs, making it easy to integrate distribution insights into monitoring workflows and visualization tools, including SageMaker Unified Studio. AWS Glue Data Quality distribution statistics are available in all AWS commercial regions and AWS GovCloud (US) regions.