Building Pub/Sub Messaging with Amazon SNS - Fan-Out Patterns and Filtering
Learn about topic-based messaging with SNS, subscription filters, and the SQS fan-out pattern for scalable event distribution.
Overview of SNS
SNS is a Pub/Sub messaging service that supports up to 12.5 million subscriptions per topic, with a maximum message size of 256 KB. Publishers send messages to a topic, and those messages are delivered to all subscribers of that topic. While SQS provides point-to-point queuing, SNS enables one-to-many fan-out. It offers two types of topics, Standard and FIFO, and uses subscription filters for efficient message routing.
Fan-Out and Filtering
In the SNS + SQS fan-out pattern, multiple SQS queues subscribe to an SNS topic, allowing a single message to be delivered to multiple consumers in parallel. A typical example is distributing an order event simultaneously to an inventory management queue, a shipping queue, and a notification queue. Subscription filter policies enable filtering based on message attributes (e.g., order_type = "premium"), so each subscriber receives only the messages it needs. FIFO topics provide message ordering guarantees and deduplication, with message group IDs controlling the order of delivery.
Dead-Letter Queues and Retries
Configuring a dead-letter queue (DLQ) on an SNS subscription routes failed delivery messages to an SQS queue for later inspection. For HTTP/S endpoint deliveries, retry policies (immediate retry, backoff retry) control the number of retries and intervals. Lambda subscriptions follow Lambda's asynchronous invocation retry policy. Delivery status logs can be sent to CloudWatch Logs for analyzing delivery success rates and error causes. Cross-account topic access is controlled through resource-based policies. For a deeper understanding of SNS, related books on Amazon can be a helpful resource.
Managing SNS Costs
SNS costs are determined by the number of publishes and the delivery protocol. Deliveries to SQS are free, making the SNS to SQS to Lambda pattern potentially more cost-effective than direct SNS to Lambda delivery. Message filtering reduces unnecessary deliveries and optimizes Lambda invocation counts. For high-volume messaging, batching multiple events into a single message can reduce the total number of publishes.
Summary
SNS is a service that enables fan-out patterns through Pub/Sub messaging. Subscription filters provide efficient attribute-based message routing, and DLQs reliably capture failed delivery messages. Since deliveries to SQS are free, the SNS to SQS to Lambda pattern offers a cost-efficient approach to building asynchronous processing pipelines. Cross-account topic access is managed through resource-based policies.