Building Pub/Sub Messaging with Amazon SNS - Fan-Out Patterns and Filtering
Learn about topic-based messaging with SNS, subscription filters, and the SQS fan-out pattern for scalable event distribution.
Overview of SNS
SNS is a Pub/Sub messaging service that supports up to 12.5 million subscriptions per topic, with a maximum message size of 256 KB. Publishers send messages to a topic, and those messages are delivered to all subscribers of that topic. While SQS provides point-to-point queuing, SNS enables one-to-many fan-out. Standard topics deliver with best-effort ordering at nearly unlimited throughput, while FIFO topics provide strict ordering guarantees and deduplication at 300 messages per second (3,000 with batching). Subscription protocols include SQS, Lambda, HTTP/S, Email, SMS, and Kinesis Data Firehose, allowing simultaneous delivery to multiple subscribers using different protocols from a single topic.
Fan-Out and Filtering
In the SNS + SQS fan-out pattern, multiple SQS queues subscribe to an SNS topic, allowing a single message to be delivered to multiple consumers in parallel. A typical example is distributing an order event simultaneously to an inventory management queue, a shipping queue, and a notification queue. Subscription filter policies enable filtering based on message attributes, so each subscriber receives only the messages it needs. Filter policies can be scoped to MessageAttributes (default) or MessageBody, and support compound conditions combining exact string match, prefix match, numeric range, and the exists operator. FIFO topics control ordering via message group IDs - messages within the same group are delivered in strict FIFO order. Different groups are delivered independently in parallel, enabling designs that guarantee order only where needed while maintaining throughput.
Dead-Letter Queues and Retries
Configuring a dead-letter queue (DLQ) on an SNS subscription routes failed delivery messages to an SQS queue for later inspection. For HTTP/S endpoint deliveries, retry policies control the number of retries and intervals. HTTP/S delivery retries occur in 3 phases: immediate retry (1 attempt), backoff phase (10 attempts with exponential backoff), and post-backoff phase (38 attempts at 20-second intervals) for a total of 49 retries. Lambda subscriptions follow Lambda's asynchronous invocation retry policy. Delivery status logs can be sent to CloudWatch Logs for analyzing delivery success rates and error causes. Delivery status logs support separate sampling rates for success and failure - it is common practice to keep success log sampling at around 10% for cost management. Cross-account topic access is controlled through resource-based policies. For a deeper understanding of SNS, related books on Amazon can be a helpful resource.
Design Best Practices and Pitfalls
Message attribute design directly impacts filtering efficiency. Include attributes used for filtering in MessageAttributes so subscribers don't need to parse the payload body. Standard topics may deliver messages more than once (at-least-once), so consumers must ensure idempotency. For large message payloads (over 256 KB), use the Extended Client Library pattern: store the payload in S3 and include only the S3 reference in the SNS message. For FIFO topics, MessageDeduplicationId design is critical - enabling content-based deduplication uses a body hash for automatic detection, but be aware of unintended deduplication. Two encryption options are available: SSE-SNS (AWS managed key) and SSE-KMS (CMK). For cross-account scenarios, the CMK key policy must grant decryption permissions to the destination account.
Choosing Between SNS and EventBridge
SNS and EventBridge are both foundations for event-driven architectures but serve different domains. SNS is suited for high-throughput scenarios (nearly unlimited with Standard topics) where simple attribute filtering is sufficient. EventBridge is suited for scenarios requiring advanced pattern matching on event structure (arbitrary fields in JSON body), schedule triggers, or SaaS integrations (Shopify, Zendesk, etc.). In terms of pricing, SNS delivery to SQS is free while EventBridge charges per event, making SNS lower-cost for pure Pub/Sub workloads that fan out the same event to many targets. Conversely, for cases with a few rules performing complex routing, EventBridge's rule-based design offers better maintainability. Combining both - routing with EventBridge then fanning out with SNS - is also an effective architecture.
Managing SNS Costs
SNS costs are determined by the number of publishes and the delivery protocol. Deliveries to SQS are free, making the SNS to SQS to Lambda pattern potentially more cost-effective than direct SNS to Lambda delivery. HTTP/S deliveries cost $0.06 per 100,000, and SMS costs vary significantly by destination country ($0.00645/message for the US, $0.07086/message for Japan). Message filtering reduces unnecessary deliveries and optimizes Lambda invocation counts. For high-volume messaging, batching multiple events into a single message can reduce the total number of publishes. FIFO topics cost twice as much as Standard ($0.50 per million requests), so use Standard topics for parts that don't require ordering guarantees to control costs.
Summary
SNS is a service that enables fan-out patterns through Pub/Sub messaging. Subscription filters provide efficient attribute-based message routing, and DLQs reliably capture failed delivery messages. Since deliveries to SQS are free, the SNS to SQS to Lambda pattern offers cost-efficient asynchronous processing, with FIFO topics applying ordering guarantees only where needed. By choosing appropriately between SNS and EventBridge based on throughput requirements and routing complexity, you can design optimal event-driven architectures.