Event Integration with Amazon EventBridge Pipes - Connection Patterns for Sources and Targets
Learn how to connect event sources like SQS, DynamoDB Streams, and Kinesis directly to targets, with filtering and enrichment steps in between.
Pipes Connection Patterns
EventBridge Pipes is a service that connects event sources to targets in a point-to-point fashion. Previously, passing messages from an SQS queue to Step Functions required creating a Lambda function, but Pipes lets you connect sources and targets directly. Supported sources include SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MSK, and self-managed Kafka. Targets include EventBridge buses, Step Functions, Lambda, API Gateway, SNS, SQS, and many other services. You can control throughput by configuring batch size and concurrency settings.
Filtering and Enrichment
Filtering uses EventBridge event pattern syntax to pass only events matching specific conditions through the pipeline. You can extract only INSERT events from DynamoDB Streams or process only SQS messages where a specific field exceeds a threshold. The enrichment step sends filtered events to a Lambda function or API Gateway endpoint for adding external data or transforming formats. Input transformers map JSON paths from events to the structure expected by the target, enabling code-free data transformation.
Source and Target Combinations
Pipes supports SQS, Kinesis Data Streams, DynamoDB Streams, MSK, MQ, and self-managed Kafka as sources. Targets include Step Functions, Lambda, ECS tasks, EventBridge buses, API Gateway, SNS, SQS, and over 15 other services. The enrichment step can call Lambda or API Gateway to add external information to event data before delivering it to the target. Input transformers convert event structures into the format expected by the target. For a comprehensive look at event integration architecture, check out technical books (Amazon).
Pipes Pricing
EventBridge Pipes is billed per request, at approximately $0.40 per million requests per 64 KB chunk. Filtering to narrow down events delivered to the target reduces processing costs on the target side (Lambda invocations, Step Functions state transitions). Lambda invocations for enrichment are billed separately at standard Lambda rates. Compared to writing glue code in Lambda, Pipes can reduce both Lambda execution costs and maintenance overhead.
Summary
EventBridge Pipes reduces glue-code Lambda functions by directly connecting event sources to targets. It connects 6 types of sources, including SQS, Kinesis, and DynamoDB Streams, to over 15 targets, with filtering and enrichment for event selection and transformation. Input transformers convert events into the format expected by the target, significantly reducing the maintenance cost of integration code.