Building Asynchronous Messaging with Amazon SQS - Designing Standard and FIFO Queues
Learn how to choose between Standard and FIFO queues, use DLQs for failed message handling, and automate processing with Lambda event source mappings.
Overview of SQS
SQS is a fully managed message queuing service. While SNS provides one-to-many fan-out, SQS provides one-to-one point-to-point messaging. It decouples web application request handling from heavy backend processing, improving scalability and fault tolerance. SQS stores messages redundantly across multiple AZs, so messages are not lost due to a single AZ failure. The message retention period defaults to 4 days and can be extended up to 14 days. The Visibility Timeout defaults to 30 seconds, controlling how long a message being processed by one consumer remains invisible to other consumers.
Queue Design and DLQ
Standard queues provide throughput of tens of thousands of messages per second, with no guarantee of message ordering (best-effort ordering). They guarantee at-least-once delivery, so consumer-side idempotency is essential. FIFO queues guarantee ordering per message group ID and achieve exactly-once processing through deduplication IDs. Throughput is 300 messages per second (3,000 with batching), extendable to thousands per second with high-throughput mode. DLQs automatically move messages that have failed processing maxReceiveCount times, preventing them from blocking the main queue. After investigating DLQ messages and fixing the root cause, you can redrive (resend) them to the main queue. The DLQ retention period can be set independently from the source queue, so setting a generous period (e.g., 14 days) for investigation is recommended.
Lambda Integration and Large Messages
SQS and Lambda event source mappings automatically process queue messages with Lambda functions. By configuring batch size (up to 10,000) and batch window (up to 300 seconds), you can process multiple messages at once and reduce the number of Lambda invocations. Enabling partial batch responses (ReportBatchItemFailures) allows reprocessing only the failed messages within a batch, avoiding duplicate processing of successful messages. With FIFO queues, Lambda's concurrency is limited to the number of message groups. The Extended Client Library enables sending and receiving messages larger than 256 KB via S3, supporting asynchronous processing of large payloads. Message encryption is configured with SSE-SQS (free) or SSE-KMS. For more on message queuing, you can also explore related books on Amazon.
Design Best Practices and Pitfalls
Set Visibility Timeout longer than the consumer's maximum processing time. If it's too short, messages get redelivered causing duplicate processing. Idempotency is mandatory for Standard queue consumers. Use DynamoDB conditional writes or database unique constraints to prevent duplicate processing. The FIFO deduplication window is fixed at 5 minutes; sending a message with the same deduplication ID within 5 minutes causes the subsequent message to be discarded. Always enable long polling (set WaitTimeSeconds between 1 and 20 seconds). Short polling generates frequent empty responses, causing unnecessary request charges. Leverage message attributes (MessageAttributes) for routing and filtering to minimize message body parsing and improve consumer performance.
Pricing and Limit Considerations
SQS Standard queues cost approximately $0.40 per million requests, and FIFO queues cost approximately $0.50 per million requests. The first 1 million requests per month are free. Each request handles up to 256 KB, with every 64 KB chunk counted as one request. Long polling (WaitTimeSeconds=20) reduces empty responses and optimizes request counts. DLQ messages are billed at the same rate as regular queue messages, so process or delete DLQ messages periodically. The in-flight message limit is 120,000 for Standard and 20,000 for FIFO. Exceeding this limit returns an OverLimit error, so designing consumer scaling to maintain processing throughput is critical. When using SSE-KMS encryption, additional KMS API call costs apply. For high-volume queues, consider SSE-SQS (no additional cost).
Comparison with Other Messaging Services
SQS is a pull-based point-to-point queue. SNS is push-based pub/sub, delivering a single message simultaneously to multiple subscribers. The SNS + SQS fan-out pattern subscribes multiple SQS queues to an SNS topic, enabling independent parallel processing in each queue. Amazon MQ is a managed version of Apache ActiveMQ and RabbitMQ, chosen when compatibility with existing protocols like JMS or AMQP is required. EventBridge is designed for event-driven architectures, routing events from over 100 AWS services and SaaS applications using rule-based filtering. SQS is optimized for high-volume message buffering and the worker pattern, excelling in cost efficiency and operational simplicity.
Summary
SQS is a service for building loosely coupled architectures through asynchronous messaging. Standard queues provide throughput of tens of thousands of messages per second, while FIFO queues offer ordering guarantees and deduplication. Lambda event source mappings automate message processing, and DLQs handle failed messages. Long polling optimizes request counts, and the Extended Client Library supports messages larger than 256 KB. Ensuring idempotency and properly configuring Visibility Timeout are critical design decisions that determine production stability.