Aurora PostgreSQL now supports querying of Apache Iceberg and Parquet data
Aurora PostgreSQL now allows direct querying of Apache Iceberg and Parquet data in data lakes using existing PostgreSQL applications and tools, eliminating the need for ETL pipelines or data duplication.
Starting today, Aurora PostgreSQL allows querying data stored in Apache Iceberg and Parquet formats in data lakes directly using existing PostgreSQL applications or tools, eliminating the need for ETL pipelines or data duplication. You can create PostgreSQL external tables that reference Iceberg or Parquet data stored in Amazon S3, Amazon S3 Tables, or the AWS Glue Data Catalog. When querying these external tables, Aurora uses the high-performance DuckDB query engine embedded in PostgreSQL to run queries against the underlying Iceberg and Parquet data. Existing applications and BI tools continue to use the same PostgreSQL interface. External Iceberg REST Catalog (IRC) compatible catalogs can be federated through AWS Glue Data Catalog for query access without data movement or duplication. For latency-sensitive workloads, standard SQL statements can materialize Iceberg or Parquet data into native Aurora PostgreSQL tables, removing the need for ETL pipelines. This feature is generally available in Aurora PostgreSQL versions 17.11 and later, 18.6 and above, across all AWS commercial and GovCloud (US) regions at no additional cost. To get started, use the Amazon RDS console or any PostgreSQL client. For more details, see the blog post or documentation.
Why it matters
This update is useful for application developers and data engineers who need to access data from data lakes. It eliminates the need for data copying or ETL pipelines from data lakes to Aurora, reducing costs and engineering effort.