API Audit Logging with AWS CloudTrail - Trail Design and Security Analysis
Record all API activity and run advanced analysis with SQL queries in CloudTrail Lake. This article covers automatic anomaly detection with Insights and real-time detection through EventBridge integration.
Overview of CloudTrail
CloudTrail is a service that records API activity in your AWS account. Every API call is recorded, including EC2 instance launches, S3 bucket creation, and IAM policy changes. By default, management events from the past 90 days can be viewed for free, and creating a trail stores them indefinitely in an S3 bucket.
Trail Design and CloudTrail Lake
An organization trail applied to all regions is recommended. Events are stored in JSON format in an S3 bucket and encrypted with SSE-KMS. CloudTrail Lake retains events in an event data store for up to 7 years and allows searching and analysis with SQL queries. You can instantly run queries such as "events where an IAM user was created in the past 30 days" or "API calls from a specific IP address." Insights uses ML to learn the baseline of API calls and detects anomalous patterns, such as a 10x spike in API call volume.
Data Events and Insights
Data events record S3 object-level operations (GetObject, PutObject), Lambda function invocations, and DynamoDB table operations. Since the volume of data events is significantly higher than management events, you should narrow down the target resources before enabling them. CloudTrail Insights automatically detects anomalous patterns in management events, such as sudden spikes in API call counts or rising error rates, highlighting activity that deviates from normal behavior. For example, if RunInstances, which is normally called about 10 times a day, is suddenly called 1,000 times, an Insights event is generated. Integration with EventBridge enables real-time detection of specific API calls (such as DeleteBucket or StopLogging), triggering alerts via Lambda. For a comprehensive guide from basics to advanced usage of CloudTrail, check out books on Amazon.
CloudTrail Cost Optimization
The first trail for management events is free, and additional trails cost approximately $2 per 100,000 events. Data events cost approximately $0.10 per 100,000 events, and costs can surge rapidly in environments with heavy S3 read operations. CloudTrail Lake queries are billed based on the amount of data scanned, at approximately $0.005 per GB. To manage costs, limit data event targets to sensitive buckets and critical Lambda functions, and avoid blanket application across all resources. Use S3 lifecycle rules to manage the retention period of stored logs and migrate older logs to Glacier to reduce storage costs.
Summary
CloudTrail is an audit logging service that records all API activity in your AWS account. Use CloudTrail Lake for advanced analysis with SQL queries, and Insights for automatic detection of anomalous API call patterns. Centrally manage logs across all accounts with organization trails, and achieve real-time detection of security events through EventBridge integration.