AWS First Steps
Region expansion

AWS Glue 5.1 is now available in AWS European Sovereign Cloud Region

AWS Glue upgrades to Apache Spark 3.5.6, Python 3.11, and Scala 2.12.18, enhancing performance and security

AWS Glue 5.1 is now available in the AWS European Sovereign Cloud region. This version upgrades the core engine to Apache Spark 3.5.6, Python 3.11 and Scala 2.12.18, enhancing performance and security. Support for open table format libraries such as Apache Hudi 1.0.2, Apache Iceberg 1.10.0 and Delta Lake 3.3.2 has also been updated. Additionally, Apache Iceberg format version 3.0 support has been added, including new features like default column values and delete vectors. Fine-grained access control from AWS Lake Formation has been extended to write operations (DML and DDL) for Spark DataFrames and Spark SQL. Full table access control for Apache Hudi and Delta Lake tables has also been added, expanding data security options.

Why it matters

AWS Glue is a serverless, scalable data integration service that simplifies discovering, preparing, moving, and integrating data from multiple sources. This update enhances performance and security, improving data management and integration capabilities.

Read the original AWS announcement

Related articles and terms

Amazon EMRA managed big data platform that runs open-source frameworks like Apache Spark, Hive, and Presto to process and analyze petabyte-scale dataAWS Lake FormationA service that centralizes data lake construction, management, and security, providing column-level and row-level fine-grained access control for data on S3Data Lake Governance - Centralized Access Control with AWS Lake FormationLearn about building, access control, and governance for data lakes using AWS Lake Formation. This article covers fine-grained column-level and row-level permission management for S3-based data lakes, along with Glue and Athena integration.Amazon EMR ServerlessA serverless big data processing service that runs Apache Spark and Hive jobs without cluster management, charging only for the vCPU and memory time consumed during processingBuilding a Data Lake with Amazon S3 and Lake Formation - Design Patterns and GovernanceExplore data lake design patterns using S3 as the storage foundation and Lake Formation for fine-grained access control. This article also covers ETL pipelines and cost optimization.