Why CloudWatch Logs Costs More Than You Expect - The Cost Structure of Ingestion, Storage, and Analysis
This article explains the pricing structure where CloudWatch Logs ingestion costs over 20 times more than storage, the cost explosions caused by VPC Flow Logs and Lambda logs, and optimization strategies using the Infrequent Access class and S3 export.
Ingestion Costs Dominate the Cost Structure
CloudWatch Logs pricing consists of three components. Ingestion charges apply when sending log data to CloudWatch Logs, at $0.76 per GB in ap-northeast-1. Storage charges are billed monthly for stored log data, at $0.033 per GB per month. Analysis (Logs Insights) charges apply based on the amount of data scanned by queries, at $0.0076 per GB. The key takeaway is that ingestion costs are approximately 23 times higher than storage costs. Ingesting 1 GB of logs costs $0.76, while storing that same log for one month costs only $0.033. In other words, the majority of CloudWatch Logs costs come from ingestion. Shortening log retention periods has limited cost-reduction impact; reducing the volume of ingested logs is the most effective cost-saving measure.
Three Patterns That Cause Cost Explosions
There are three patterns where CloudWatch Logs costs become unexpectedly high. First, VPC Flow Logs. Sending VPC Flow Logs to CloudWatch Logs generates logs proportional to network traffic volume. In high-traffic VPCs, tens of gigabytes of flow logs can be generated per day. Ingesting 1 TB of flow logs per month costs $760 in ingestion charges alone. Sending VPC Flow Logs directly to S3 is cheaper, as there are no ingestion charges and you only pay S3 storage costs ($0.025/GB). Second, Lambda function logs. Lambda automatically sends START, END, and REPORT logs to CloudWatch Logs for each execution. A function that runs 1 million times per day generates a significant volume of logs from this alone. If the function heavily uses console.log, log volume increases further. Third, forgotten debug logs. Deploying to production with DEBUG-level logging still enabled can increase log volume by 10 to 100 times the normal amount.
Infrequent Access Class - Cutting Ingestion Costs in Half
The CloudWatch Logs Infrequent Access (IA) class, introduced in 2023, can reduce ingestion costs by approximately 50%. The IA class ingestion rate is $0.38 per GB (half of Standard), while the storage rate remains the same at $0.033/GB. The limitation of the IA class is that Logs Insights queries are not available. You must use Live Tail or filter patterns to search logs. The recommended approach is to use the Standard class for logs that are frequently queried for analysis, and the IA class for logs where storage is the primary purpose and analysis frequency is low. The class is specified when creating a log group. The class of an existing log group cannot be changed, so migrating to the IA class requires creating a new log group and redirecting the log destination. Lambda logs are automatically sent to a log group named /aws/lambda/function-name, but since you can change the log group in the Lambda configuration, switching to an IA class log group is possible.
S3 Export and Firehose - The Optimal Solution for Long-Term Storage
For long-term log storage, S3 is significantly cheaper than CloudWatch Logs. CloudWatch Logs storage costs $0.033/GB, while S3 Standard costs $0.025/GB and S3 Glacier Instant Retrieval costs $0.005/GB. There are two ways to export logs from CloudWatch Logs to S3. The CreateExportTask API batch-exports logs for a specified time period to S3. However, only one export task can run at a time per region, which can be time-consuming in environments with many log groups. Subscription filters using Kinesis Data Firehose deliver logs to S3 in real time. While Firehose data processing charges ($0.036/GB) apply, this is inexpensive compared to CloudWatch Logs ingestion charges ($0.76/GB). The most efficient architecture is to send logs to both CloudWatch Logs and S3. Store logs in CloudWatch Logs for a short period (7 to 30 days) for real-time monitoring and alarms, and store them in S3 for long-term retention and analysis with Athena.
Practical Techniques for Reducing Log Volume
The most effective way to reduce costs is to reduce the volume of ingested logs. First, set appropriate log levels. In production environments, set the log level to INFO or WARN and avoid outputting DEBUG-level logs. Controlling the log level via environment variables allows you to switch to DEBUG only when debugging is needed. Second, use sampling. If you do not need to log every request, record only a certain percentage. For example, logging only 10% of requests reduces log volume to one-tenth. Third, use structured logging. Using JSON-formatted structured logs and including only necessary fields reduces the data volume per line. Instead of including the full stack trace in logs, recording only the error message and the first few lines is also effective. Fourth, use metric filters. Extracting metrics from logs and storing them as CloudWatch metrics allows you to shorten the retention period of the logs themselves. For a systematic approach to log management cost optimization, specialized books on Amazon are a helpful reference.