Event Integration with Amazon EventBridge Pipes - Connection Patterns for Sources and Targets
This article introduces design patterns that connect event sources such as SQS, DynamoDB Streams, and Kinesis directly to targets, with filtering and enrichment in between.
Pipes Connection Patterns
EventBridge Pipes is a service that connects event sources and targets point to point. Previously, passing messages from an SQS queue to Step Functions required creating a Lambda function, but with Pipes you can connect sources and targets directly. Supported sources are SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MSK, and self-managed Kafka, and targets include many services such as EventBridge buses, Step Functions, Lambda, API Gateway, SNS, and SQS. You control throughput with the batch size and concurrency settings.
Filtering and Enrichment
Filtering uses the EventBridge event pattern syntax to pass only events that match specific conditions into the pipeline. You can apply controls such as extracting only INSERT events from DynamoDB Streams, or processing only SQS messages in which a specific field exceeds a threshold. In the enrichment step, filtered events are sent to a Lambda function or an API Gateway endpoint to add external data or convert formats. The input transformer specifies JSON paths in the event to map it to the structure the target expects, providing data transformation without code.
Combining Sources and Targets
Pipes supports SQS, Kinesis Data Streams, DynamoDB Streams, MSK, MQ, and self-managed Kafka as sources. Targets cover 15 or more services, including Step Functions, Lambda, ECS tasks, EventBridge buses, API Gateway, SNS, and SQS. The enrichment step calls Lambda or API Gateway to add external information to the event data before delivering it to the target. The input transformer converts the event structure and shapes it into the format the target expects.
Pipes Pricing
EventBridge Pipes is billed by the number of requests, at about $0.40 per 1 million requests per 64 KB chunk (as of August 2026). Narrowing down the events delivered to the target through filtering can reduce processing costs on the target side (the number of Lambda invocations and Step Functions state transitions). Lambda invocations for enrichment incur standard Lambda charges separately. Compared with writing glue code in Lambda without Pipes, you can reduce Lambda execution costs and maintenance costs.
Error Handling and Retries
Pipes provides error handling that depends on the type of source. With SQS, Kinesis, and DynamoDB Streams, messages that fail processing are retried, and those that exceed the specified number of attempts can be moved to a dead-letter queue (DLQ). As a result, even if some events fail, the pipeline as a whole does not stall, and you can investigate or reprocess only the problematic events later. When processing in batches, you can also configure partial retries of only the failed items, which lets you avoid processing successful events twice. You monitor the events that accumulate in the DLQ and use them to detect root causes.
Ordering Guarantees and Throughput Control
When Kinesis Data Streams or DynamoDB Streams is the source, order is preserved per shard. Because order within the same shard is maintained while you raise concurrency to gain throughput, you can scale safely even for workloads where order matters. Adjusting the batch size and the batching window lets you balance latency and throughput. You configure these settings to match the processing capacity of the target and avoid placing excessive load on downstream services. For sources that do not need ordering guarantees, such as SQS, you can raise concurrency and prioritize efficiency.
Combining Pipes with Step Functions
When you specify Step Functions as the target of a pipe, you can start a multi-step workflow triggered by an event. Previously you had to write Lambda code to bridge source polling and workflow startup, but that glue code is no longer needed. You choose synchronous state machine execution for processing that completes quickly and asynchronous execution for processing that takes time, using each according to the use case. If filtering narrows the events down to only those that should start the workflow, you can keep the number of state transitions down and optimize costs. You can build event-driven business processing simply.
Key Points for Monitoring and Operations
You track the health of Pipes with CloudWatch metrics. You monitor the number of executions, the number of failures, the duration of each stage, and so on, and send notifications through alarms when anomalies occur. If the latency of the Lambda function or API used for enrichment becomes a bottleneck, the throughput of the entire pipeline hits a ceiling, so you look at the duration of each stage to identify where to improve. If you manage the configuration as code and review changes to filter patterns and transformation rules through CI/CD before applying them, you can prevent unexpected behavior changes in production. You keep logs so that you can trace which events were processed and how.
Summary
EventBridge Pipes reduces glue-code Lambda functions by connecting event sources and targets directly. It connects six types of sources, such as SQS, Kinesis, and DynamoDB Streams, to 15 or more targets, and uses filtering and enrichment to select and process events. The input transformer converts events into the format the target expects, greatly reducing the maintenance cost of integration code.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- Amazon EventBridge official page
- Amazon SQS official page
- AWS Lambda official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.