AWS Lake Formation Specialized2019年〜
A service that simplifies building, managing, and securing data lakes
What It Does
AWS Lake Formation is a service for building data lakes on S3 and centrally managing data ingestion, cataloging, and security. It provides fine-grained access control at the table and column level, and uniformly manages access from analytics services like Athena, Redshift, and EMR.
Use Cases
Used for building enterprise data lakes, integrating and analyzing multiple data sources, implementing data governance, and managing data sharing and access control across departments.
Everyday Analogy
Think of it like a library catalog management system. It classifies and registers (catalogs) books (data) arriving from various publishers (data sources), and controls which books each patron (analyst) can access (access control).
What Is Lake Formation?
AWS Lake Formation is a service that simplifies the setup and management of data lakes. A data lake is an S3-based repository that centrally stores structured, semi-structured, and unstructured data. Lake Formation provides data ingestion, integration with the Glue Data Catalog, and fine-grained access control.
Access Control and Data Sharing
Lake Formation's standout feature is table-, column-, and row-level access control. Fine-grained data access control that's difficult to achieve with IAM policies alone can be uniformly managed through Lake Formation's permission model. The data sharing feature also enables secure data sharing across different AWS accounts. To gain a deeper understanding of access control and data sharing, reference books on Amazon are also worth checking out.
Getting Started
Set up a data lake administrator in the Lake Formation console and register an S3 bucket as the data lake location. Catalog your data with Glue crawlers and configure access permissions through Lake Formation permissions. When you run queries from Athena or Redshift Spectrum, Lake Formation access controls are automatically applied.
Things to Watch Out For
- Both Lake Formation access controls and IAM policies apply, which can make permission design complex
- Lake Formation itself is free. Separate charges apply for Glue, S3, Athena, and other services
