Building a Security Data Lake with Amazon Security Lake - Unified Analysis in OCSF Format
Learn about Security Lake's automatic aggregation of CloudTrail, VPC Flow Logs, and Route 53 logs, OCSF normalization, and integration with subscribers.
Overview of Security Lake
Security Lake is a service that automatically aggregates and normalizes security data from AWS and third-party sources. Previously, security analysis required collecting and transforming CloudTrail logs, VPC Flow Logs, and GuardDuty findings individually, but Security Lake automatically converts these into OCSF (Open Cybersecurity Schema Framework) format and aggregates them into an S3-based data lake. Data is stored in the S3-based data lake and can be queried with SQL from Athena. Through Organizations integration, a delegated administrator account can manage logs across all member accounts centrally, and data from a multi-account environment converges into a single data lake.
Data Sources and OCSF Normalization
Security Lake automatically collects AWS data sources such as CloudTrail, VPC Flow Logs, and Route 53 Resolver logs. Third-party data sources (CrowdStrike, Palo Alto Networks, Cisco Security, etc.) can also be added as custom sources. OCSF is an open framework that converts security events from different sources into a unified schema, enabling queries using the same column names and data types regardless of source. Through OCSF normalization, for example, CloudTrail's sourceIPAddress and VPC Flow Logs' srcaddr are both mapped to a unified src_endpoint.ip, so a single query can analyze data across sources. Partitioning is automatically applied by region, account ID, and event date, reducing Athena query scan volumes.
Subscribers and Analysis
Subscribers are consumers that access data in the data lake. Data access subscribers are notified of new objects as data is written to the S3 bucket, through an HTTPS endpoint that they provide or an SQS queue. Query access subscribers directly query Lake Formation tables in the S3 bucket with services like Athena. SIEM tools such as Splunk and Datadog can be configured as subscribers to integrate Security Lake data into existing security operations tools.
Design Best Practices and Pitfalls
When deploying Security Lake, you can designate a dedicated log archive AWS account as the delegated administrator, separate from workload accounts. If you designate the Region where you create a subscriber as the rollup aggregation region, data from other regions can be consolidated there, but note that cross-region data transfer charges apply. A common pitfall is that enabling CloudTrail data events (S3 object-level operations) can greatly increase ingestion volumes, so measure log volume in a sample account beforehand. When adding custom sources, failing to pre-validate OCSF schema mapping can cause parse failures and data loss, so always validate with a test custom source before production deployment.
Comparison with CloudWatch Logs Aggregation and Custom ETL
Traditional approaches without Security Lake include aggregating logs into CloudWatch Logs with Logs Insights queries, or building a custom data lake with Amazon Data Firehose and Glue ETL. CloudWatch Logs is simple to configure but lacks unified schemas across log sources, making cross-source queries difficult, and storage costs for long-retention logs are significantly higher than S3. Custom ETL offers schema design flexibility but incurs substantial development and operational costs for normalization pipelines, and without OCSF-like open standard compliance, integrating with third-party SIEMs requires custom transformations. Security Lake's advantage lies in managed normalization, partitioning, and subscriber management, plus OCSF compliance that enables integration with security tools that support OCSF. However, Security Lake is purpose-built for security logs; application logs and business metrics aggregation remains better suited to CloudWatch Logs or custom ETL.
Security Lake Pricing
Security Lake pricing depends on factors such as the volume of data ingested. S3 storage charges apply separately. Querying with Athena incurs Athena charges separately. Set data retention periods per region and tier older data to Glacier automatically to reduce storage costs. When enabling across an entire Organization, you can roll out in phases, starting with high-log-volume accounts and monitoring costs along the way. When enabling CloudTrail data events and VPC Flow Logs, measure costs for one account over one week first, estimate the monthly amount, then roll out to all accounts.
Summary
Security Lake is a data lake service that automatically aggregates AWS security data in OCSF format. Organizations integration centralizes security data across the entire organization, enabling cross-cutting analysis with Athena and SIEM tools. Key points for deployment include isolating the delegated administrator account, pre-validating costs when enabling data events, and accounting for cross-region transfer costs with rollup aggregation regions.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon Security Lake official page
- Amazon Athena official page
- AWS Security Hub official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.