Building a Data Lake with Amazon S3 and Lake Formation - Design Patterns and Governance
This article introduces data lake design patterns that use S3 as the storage foundation and achieve fine-grained access control with Lake Formation. It also explains ETL pipelines and cost optimization.
Data Lake Design Patterns
A data lake uses S3, which has 99.999999999% (eleven nines) durability, as its storage foundation and manages everything step by step, from ingesting raw data to transforming it into an analyzable state. The landing zone (Raw) stores ingested raw data as is, and the staging zone (Processed) performs type conversion and cleansing with Glue jobs. The curation zone (Curated) holds analytical data with business logic applied, in Parquet format. In S3 prefix design, adopting a year/month/day partition structure reduces the amount of data scanned in Athena queries through partition pruning.
Governance with Lake Formation
Lake Formation is a service that centrally manages access control for a data lake. Previously, you had to configure S3 bucket policies, IAM policies, and Glue catalog policies individually, but with Lake Formation you can perform GRANT/REVOKE-based permission management at each of the database / table / column / row levels. With tag-based access control (LF-TBAC), you can assign classification tags to data and automatically apply access rights based on the tags. With cross-account sharing, you can grant other accounts within Organizations access at the table level, which also helps with data mesh configurations.
ETL Pipeline Design
A data lake's ETL pipeline is built with Glue jobs. Glue crawlers detect the schema of raw data in the landing zone and register it in the Data Catalog. Glue jobs perform type conversion, missing value handling, and deduplication, and output the results in Parquet format to the curation zone. You set partition keys (date, region) to optimize Athena query performance. Glue workflows define the dependencies among multiple jobs and control the order of crawler → ETL job → data quality check. EventBridge automatically starts the ETL pipeline triggered by data arriving in S3, and updates proceed without delay after the data arrives.
Data Lake Cost Optimization
You optimize data lake costs by making use of S3 storage classes. You store raw data in the landing zone in S3 Standard and set a lifecycle rule that transitions it to S3 Intelligent-Tiering after 30 days. Curated data is kept in Standard because it is queried frequently, and data in the archive zone is transitioned to Glacier Instant Retrieval. Athena query costs can be reduced with the Parquet format and appropriate partition design, and in some cases the amount of data scanned is reduced compared with CSV. You set the number of DPUs for Glue jobs appropriately and avoid excessive resource allocation. You visualize the cost breakdown per bucket with S3 Storage Lens and regularly delete unnecessary data and review storage classes.
Layer Design
A data lake is often designed by dividing it into layers according to the processing stage of the data. You divide it into stages, such as a layer that holds raw data as ingested, a layer with cleansing and formatting applied, and a layer aggregated / processed into a form that is easy to analyze. By keeping the raw data, you can reprocess it later for different purposes and also handle redoing the processing. Clearly separating each layer makes it easier to trace data lineage and also easier to manage quality. Users refer to the data in the appropriate layer for their purpose. A staged layer structure becomes the foundation of a flexible, highly reusable data lake.
File Formats and Partitioning
The performance and cost of a data lake are greatly influenced by the file storage format and how the data is divided. When you store data in a columnar format, analytical queries can read only the columns they need, so the amount of data scanned decreases, queries become faster, and costs become easier to keep down. If you partition data by date or attribute, queries can read only the range they need. On the other hand, a large number of small files degrades query performance, so you need to take measures to combine them into files of moderate size. Appropriately designing the storage format / partitioning / file size is an important factor in analyzing large-scale data efficiently.
Metadata and Catalog Management
Because a wide variety of data accumulates in a data lake, you need a mechanism to manage what is where. You register table definitions in a data catalog and centrally manage the structure and location of the data. Crawlers automatically read the structure of the data, which lets you keep the catalog up to date. Because the structure of ingested data changes over time, you make the design able to handle schema evolution. With a well-maintained catalog, users can find the data they need more easily and can access it consistently from multiple analytics tools. Metadata management determines how easy a data lake is to use.
From Data Lake to Lakehouse
Traditional data lakes can store large amounts of data inexpensively, but they have been weak at managing updates and consistency. What compensates for this is the use of new table formats that enable updates / deletes and consistent operations on top of a data lake. With open table formats, you can gain reliable data operations like a data warehouse and the ability to reference data as of a past point in time, while keeping the cost efficiency of a data lake. You can handle the same table consistently from multiple analytics engines. This approach, which combines the advantages of storage and warehouses, is broadening the choices for data platforms.
Summary
A data lake that combines S3 and Lake Formation is a configuration that refines data step by step with a three-tier zone design and enforces governance with Lake Formation's fine-grained access control. Automatic schema detection with Glue crawlers and the adoption of columnar formats let you reduce operational effort while also improving query performance.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon S3 official page
- AWS Lake Formation official page
- AWS Glue official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.