Building a Data Lake with AWS Lake Formation - Fine-Grained Access Control and the Data Catalog
Fine-grained column- and row-level access control and tag-based management establish data lake governance. This article introduces the integration with Glue Data Catalog and cross-account access.
Overview of Lake Formation
Lake Formation is a service that simplifies building, managing, and securing data lakes on S3. Traditionally, access control for data lakes on S3 was managed with a combination of S3 bucket policies and IAM policies, but column-level and row-level control was difficult. Lake Formation manages permissions at 4 levels, namely database, table, column, and row, and supports data lakes at the scale of thousands of tables. It applies these permissions uniformly to queries from Athena, Redshift Spectrum, and EMR.
Access Control and Tag-Based Management
In the Lake Formation permission model, you grant SELECT, INSERT, and DELETE permissions on databases, tables, and columns to principals (IAM users and roles). Row-level filters can restrict access to only the rows that match specific conditions, which makes it possible to let each department view only its own data. With LF-TBAC, you assign tags to data (for example, sensitivity=high) and set tag-based permissions for principals. Even when new tables are added, permissions are applied automatically if the tags match.
Cross-Account Access and Auditing
Cross-account sharing in Lake Formation grants other accounts within Organizations access permissions to tables and databases. You use RAM (Resource Access Manager) to create resource links and display the shared tables in the Glue catalog of the consumer account. Data filters define row-level and cell-level access control, so you can expose different data ranges to each department even from the same table. Through the integration with CloudTrail, audit logs record who accessed which column of which table. Lake Formation tag-based access control (LF-TBAC), in which you assign tags to data assets and permissions are granted automatically by matching them against principal tags, is an efficient way to operate in large-scale environments.
Lake Formation Pricing and Operations
Lake Formation itself incurs no additional charges. Costs depend on the usage charges of the underlying S3 storage, Glue crawlers and jobs, and Athena queries. When you use the Lake Formation transaction feature (Governed Tables), additional charges apply for transaction API calls and storage. By introducing LF-TBAC, you are freed from managing individual table and column permissions, and the operational cost of permission management can be reduced significantly. We recommend that you place S3 paths under Lake Formation management through data location registration and fully migrate away from hybrid mode, avoiding dual management of IAM policies and Lake Formation permissions.
The Flow of Building a Data Lake
Building a data lake with Lake Formation involves several stages. First, you register S3 paths with Lake Formation as data storage locations and place them under its management. Next, a Glue crawler reads the structure of the data and creates table definitions in the Data Catalog. This lets you treat a collection of raw files as queryable tables. Once you set up an ingestion mechanism, you can reflect continuously arriving data in the catalog. By putting this whole flow in place, data that used to be scattered begins to function as a well-governed analytics foundation.
Relationship with the Data Catalog
Lake Formation permission management is built on the Glue Data Catalog. The catalog holds metadata such as the structure and location of tables, and Lake Formation layers on top of it a permission layer that defines who can access which databases, tables, and columns. Through the catalog, you can centrally apply fine-grained column-level and row-level control that was difficult with S3 bucket policies and IAM alone. Because you can manage the data itself separately from how it is viewed and who can access it, operations such as securely sharing the same data with each department in a different scope become possible.
Integration with Query Engines
The permissions you set in Lake Formation are applied uniformly to multiple analytics services. Whether access comes from ad hoc queries in Athena, references from Redshift Spectrum, or large-scale processing in EMR, the same permission rules take effect. This removes the need to build access control separately for each tool you use, and you can maintain consistent governance. Controls such as masking columns to hide sensitive information and filtering rows to show only the assigned range also work consistently through any engine. Analysts can choose the tools they are familiar with while handling data securely under the organization's governance.
Governance Best Practices
In a large-scale data lake, setting individual permissions for each table tends to make management break down. If you adopt tag-based access control, which tags data and grants permissions based on those tags, even as new tables are added, appropriate permissions are applied automatically if the tags match, and operations become much easier. You apply masking to sensitive columns and enforce least privilege. You keep records of access as audit logs so that you can track who viewed what. Unifying S3 paths under Lake Formation management and avoiding dual management with IAM is also important for preventing confusion.
Summary
Lake Formation is a service that provides fine-grained access control for S3 data lakes. Column- and row-level permission management and LF-TBAC (tag-based access control) make permission management in large-scale environments more efficient, and cross-account sharing enables secure data sharing within Organizations. CloudTrail integration records audit logs of data access and builds the foundation of data governance.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS Lake Formation official page
- AWS Glue official page
- Amazon Athena official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.