Building a Security Data Lake with Amazon Security Lake - Unified Analysis in OCSF Format
Learn about Security Lake's automatic aggregation of CloudTrail, VPC Flow Logs, and Route 53 logs, OCSF normalization, and integration with subscribers.
Overview of Security Lake
Security Lake is a service that automatically aggregates and normalizes security data from AWS and third-party sources. Previously, security analysis required collecting and transforming CloudTrail logs, VPC Flow Logs, and GuardDuty findings individually, but Security Lake automatically converts these into OCSF (Open Cybersecurity Schema Framework) format and aggregates them into an S3-based data lake. Data is stored in Apache Iceberg table format and can be queried directly with SQL from Athena. Through Organizations integration, a delegated administrator account can manage logs across all member accounts centrally, and data from a multi-account environment converges into a single data lake—a key differentiator from CloudWatch Logs or per-account S3 aggregation approaches.
Data Sources and OCSF Normalization
Security Lake automatically collects 8 types of AWS-native data sources: CloudTrail management events, CloudTrail data events, VPC Flow Logs, Route 53 Resolver logs, Security Hub, Lambda execution logs, EKS audit logs, and WAF logs. Third-party data sources (CrowdStrike, Palo Alto Networks, Cisco Security, etc.) can also be added as custom sources. OCSF is an open framework that converts security events from different sources into a unified schema, enabling queries using the same column names and data types regardless of source. Through OCSF normalization, for example, CloudTrail's sourceIPAddress and VPC Flow Logs' srcaddr are both mapped to a unified src_endpoint.ip, enabling cross-source correlation analysis in a single query. Partitioning is automatically applied by region, account ID, and event date, significantly reducing Athena query scan volumes.
Subscribers and Analysis
Subscribers are consumers that access data in the data lake. Data access subscribers can query data directly on S3, analyzing it with Athena or Redshift Spectrum. Query access subscribers receive SQS notifications when new data arrives, enabling real-time analysis pipelines. SIEM tools such as Splunk and Datadog can be configured as subscribers to integrate Security Lake data into existing security operations tools. Subscribers are automatically granted cross-account access via RAM (Resource Access Manager), eliminating the need for individual IAM policy configurations. To deepen your understanding of Security Lake, specialized books on Amazon can also be helpful.
Design Best Practices and Pitfalls
When deploying Security Lake, designating a dedicated log archive AWS account as the delegated administrator—fully isolated from workload accounts—is the recommended design. Configuring a rollup aggregation region lets you consolidate data from all regions into one for centralized querying, but note that cross-region data transfer charges apply. A common pitfall is that enabling CloudTrail data events (S3 object-level operations) can generate ingestion volumes tens of times greater than management events; measuring log volume in a sample account beforehand is essential. When adding custom sources, failing to pre-validate OCSF schema mapping can cause parse failures and data loss, so always validate with a test custom source before production deployment. Iceberg table compaction (merging small files) runs automatically, but query performance may temporarily degrade during peak ingestion from high-frequency data sources; designing operations to avoid analysis peak hours is also advisable.
Comparison with CloudWatch Logs Aggregation and Custom ETL
Traditional approaches without Security Lake include aggregating logs into CloudWatch Logs with Logs Insights queries, or building a custom data lake with Kinesis Data Firehose and Glue ETL. CloudWatch Logs is simple to configure but lacks unified schemas across log sources, making cross-source queries difficult, and storage costs for long-retention logs are significantly higher than S3. Custom ETL offers schema design flexibility but incurs substantial development and operational costs for normalization pipelines, and without OCSF-like open standard compliance, integrating with third-party SIEMs requires custom transformations. Security Lake's advantage lies in managed normalization, partitioning, and subscriber management, plus OCSF compliance that avoids vendor lock-in while enabling broad security tool integration. However, Security Lake is purpose-built for security logs; application logs and business metrics aggregation remains better suited to CloudWatch Logs or custom ETL.
Security Lake Pricing
Security Lake pricing consists of data ingestion volume and storage volume. Data ingestion from AWS-native sources costs approximately $0.75 per GB, with S3 storage charges applied separately. Since data is stored in Apache Iceberg format, Athena query costs are based on S3 scan volume (approximately $5 per TB). Set data retention periods per region and tier older data to Glacier automatically to reduce storage costs. When enabling across an entire Organization, a phased rollout starting with high-log-volume accounts and monitoring costs along the way is recommended. Note that CloudTrail data events and VPC Flow Logs generate very large volumes; when enabling these, measure costs for one account over one week first, extrapolate to monthly estimates, then roll out to all accounts to avoid budget overruns.
Summary
Security Lake is a data lake service that automatically aggregates AWS security data in OCSF format. Organizations integration centralizes security data across the entire organization, enabling cross-cutting analysis with Athena and SIEM tools. Key success factors for deployment include isolating the delegated administrator account, pre-validating costs when enabling data events, and accounting for cross-region transfer costs with rollup aggregation regions.