EventBridge Pipes Design Best Practices - Optimizing Filtering and Batch Processing
Explains practical best practices for EventBridge Pipes, covering filtering design, batch size optimization, error handling strategies, and improving cost efficiency.
Principles of Filtering Design
Filtering in EventBridge Pipes is the first line of defense for keeping unnecessary events from reaching the target. Events excluded by filtering are not billed, so appropriate filter design leads directly to cost optimization. Filter patterns use the EventBridge event pattern syntax, and you can specify conditions such as exact match, prefix match, numeric comparison, and existence checks against JSON fields. As a best practice, write your filter conditions as specifically as possible. For example, when DynamoDB Streams is the source, setting a filter on the eventName field that lets only INSERT through and excludes MODIFY and REMOVE can greatly reduce unnecessary event processing. You can also combine multiple filter patterns with OR conditions, and you can specify up to 5 patterns.
Optimizing Batch Processing
Pipes retrieves events from the source in batches and delivers them to the target. The batch size and batch window settings tie directly into the trade-off between throughput and latency. Increasing the batch size lets you retrieve many events in a single poll and improves throughput, but latency increases because Pipes waits until the batch fills. If you set a short batch window (maximum wait time), delivery starts at the specified time even if the batch is not full. For use cases that require real-time responsiveness, you set a batch size of 1 and a batch window of 0 seconds. For throughput-focused batch processing, you use settings such as a batch size of 10 and a batch window of 60 seconds. For an SQS source, you can specify a batch size of up to 10,000 messages. For a Kinesis source, the maximum is 10,000 records.
Error Handling Strategies
Error handling in Pipes behaves differently depending on the type of source. For an SQS source, messages that fail processing return to the queue after the visibility timeout and are reprocessed. Messages that exceed the maximum receive count move to a dead-letter queue (DLQ). For Kinesis or DynamoDB Streams sources, the shard iterator does not advance, so a failed record becomes a blocker and subsequent records are not processed either. To avoid this problem, you set OnPartialBatchItemFailure and enable partial batch responses that retry only the failed records. If you use Lambda for enrichment, implement appropriate error handling inside the Lambda function and build in logic that retries on transient failures and skips or sends to the DLQ on permanent failures.
Key Points for Cost Optimization
Pipes pricing is based on the number of requests ($0.40 per 1 million requests in 64KB units) (as of August 2026). There are three main points for optimizing costs. First is making use of filtering. Events excluded by filters are not billed, so excluding unnecessary events early is the most effective cost reduction measure. Second is reducing the payload size with input transformation (Input Transformer). If, instead of passing all fields of the source event to the target, you extract and forward only the fields you need, the number of events that fit within the 64KB billing unit increases. Third is reconsidering whether enrichment is necessary. When you use Lambda for enrichment, Lambda execution charges are incurred in addition to the Pipes charges. If the target side can perform the supplementary processing, you can skip enrichment and reduce costs.
Observability and Testing in Practice
To operate Pipes stably, it is important to make sure you can continuously observe how it behaves. You monitor execution counts, failure counts, and the time taken at each stage with metrics, and detect anomalies with alerts. You also use the time taken to determine whether the processing used for enrichment has become a bottleneck. Before applying configuration changes to production, you test them in a staging environment and confirm that filter conditions and transformation rules work as intended. You send test events through and check whether the data arrives at the expected target in the expected form. Putting observation and testing mechanisms in place lets you prevent unexpected changes in behavior caused by configuration changes and maintain a reliable pipeline.
Summary
When operating EventBridge Pipes in production, apply best practices from four perspectives: filtering precision, tuning of batch settings, robustness of error handling, and cost efficiency. Filtering in particular is the most important setting, as it affects both cost and performance. We recommend first checking the behavior in a development environment while changing the parameters of each setting, and putting in place a system for monitoring processing volume and error rates with CloudWatch metrics before applying the settings to production.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon EventBridge Pipes official page
- Amazon EventBridge official page
- Amazon SQS official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.