API Audit Logging with AWS CloudTrail - Trail Design and Security Analysis
Record all API activity and run advanced analysis with SQL queries in CloudTrail Lake. This article covers automatic anomaly detection with Insights and real-time detection through EventBridge integration.
Overview of CloudTrail
CloudTrail is a service that records API activity in your AWS account. Every API call is recorded, including EC2 instance launches, S3 bucket creation, and IAM policy changes. By default, management events from the past 90 days can be viewed for free, and creating a trail stores them indefinitely in an S3 bucket. Each event includes the calling IAM entity, source IP address, request parameters, and response elements, enabling precise tracking of who did what, when, and from where. Event delivery latency is typically 5 to 15 minutes; real-time monitoring requires EventBridge integration.
Trail Design and CloudTrail Lake
An organization trail applied to all regions is recommended. Events are stored in JSON format in an S3 bucket and encrypted with SSE-KMS. Organization trails are created from the AWS Organizations management account and consolidate events from all member accounts into a single S3 bucket. The bucket policy should allow writes only from the CloudTrail service principal, and object lock should be enabled to prevent tampering. CloudTrail Lake retains events in an event data store for up to 7 years (up to 2,555 days with extended retention) and allows searching and analysis with SQL queries. Unlike Athena, it requires no Glue table definitions or partition design upfront - indexes are automatically generated at ingestion time, resulting in fast query responses. You can instantly run queries such as "events where an IAM user was created in the past 30 days" or "API calls from a specific IP address."
Data Events and Insights
Data events record S3 object-level operations (GetObject, PutObject), Lambda function invocations, and DynamoDB table operations. Since the volume of data events is significantly higher than management events, you should narrow down the target resources before enabling them. CloudTrail Insights automatically detects anomalous patterns in management events, such as sudden spikes in API call counts or rising error rates, highlighting activity that deviates from normal behavior. For example, if RunInstances, which is normally called about 10 times a day, is suddenly called 1,000 times, an Insights event is generated. Integration with EventBridge enables real-time detection of specific API calls (such as DeleteBucket or StopLogging), triggering alerts via Lambda. For a comprehensive guide from basics to advanced usage of CloudTrail, check out books on Amazon.
Design Best Practices and Pitfalls
Place the trail's S3 bucket in a dedicated account (log archive account) so that production account administrators cannot delete or tamper with logs. The bucket policy should allow only PutObject and explicitly Deny delete APIs. Enabling log file integrity validation (digest files) provides cryptographic detection of post-delivery log tampering. A common pitfall is leaving individual account trails active after creating an organization trail, resulting in double billing. Additionally, ingesting the same events multiple times into a CloudTrail Lake event data store doubles storage costs, so it is important to avoid overlapping ingestion sources (organization trail vs. per-account trails). Delivery failures to S3 occur silently, so monitoring CloudWatch metrics (such as InsufficientS3BucketPolicy) and setting up early detection of log gaps is essential.
CloudTrail Lake vs. Athena
There are two approaches for analyzing CloudTrail logs stored in S3: CloudTrail Lake and Athena + S3. CloudTrail Lake is easy to set up and lets you query indexed data immediately, billed at approximately $0.005/GB scanned. Athena scans raw logs on S3 directly, billed at approximately $5/TB scanned (reducible with Parquet conversion). For frequent small investigation queries, CloudTrail Lake is more economical; for cross-analyzing large volumes of historical logs or applying custom partitioning strategies, Athena is more flexible. The two are not mutually exclusive - a practical configuration uses Lake for recent investigations and Athena for long-term trend analysis. Lake query results can be saved as dashboards, which is convenient for recurring routine investigations.
CloudTrail Cost Optimization
The first trail for management events is free, and additional trails cost approximately $2 per 100,000 events. Data events cost approximately $0.10 per 100,000 events, and costs can surge rapidly in environments with heavy S3 read operations. CloudTrail Lake queries are billed based on the amount of data scanned, at approximately $0.005 per GB. To manage costs, limit data event targets to sensitive buckets and critical Lambda functions, and avoid blanket application across all resources. Use S3 lifecycle rules to manage the retention period of stored logs and migrate older logs to Glacier to reduce storage costs. CloudTrail Lake event data store storage charges accumulate as the retention period grows longer, so select an appropriate retention period (1 year, 3 years, or 7 years) based on your compliance requirements.
Summary
CloudTrail is an audit logging service that records all API activity in your AWS account. Use CloudTrail Lake for advanced analysis with SQL queries, and Insights for automatic detection of anomalous API call patterns. Centrally manage logs across all accounts with organization trails, and achieve real-time detection of security events through EventBridge integration.