Message Queues

Overview

A message queue lets a producer hand off work to a consumer asynchronously through a durable buffer, so the two never have to be available at the same time. This decouples services, smooths traffic spikes, and improves resilience because messages persist until they are processed. The trade-off is added complexity: ordering, duplicates, and failure handling all become the application's responsibility.

Syntax / Usage

Producers publish messages; a broker stores them; consumers pull or receive and then acknowledge. Until a message is acked, the broker can redeliver it, which is why consumers must handle at-least-once delivery.

[ Producer ] --> [ Queue / Topic ] --> [ Consumer ]
                        |  (persist, retry)
                        v
                  [ Dead-Letter Queue ]  (after N failed attempts)

consume(msg):
    try:
        process(msg)          # must be idempotent
        ack(msg)
    except:
        nack(msg)             # redeliver or route to DLQ

Queues (SQS, RabbitMQ) fan work to competing consumers; logs/streams (Kafka) keep an ordered, replayable partition per key so multiple consumer groups can read independently.

Examples

An e-commerce checkout publishes an order.placed event; separate consumers handle email, inventory, and analytics. The checkout returns immediately without waiting for those side effects.

A video platform pushes upload jobs to a queue; a pool of workers scales up during peaks and drains the backlog afterward, so the API stays responsive.

A payments service uses Kafka partitioned by account ID to guarantee per-account ordering, while still processing different accounts in parallel across the consumer group.

Common Mistakes

• Assuming exactly-once delivery; most brokers are at-least-once, so consumers must be idempotent
• Expecting global ordering when only per-partition (or no) ordering is guaranteed
• Omitting a dead-letter queue, so poison messages retry forever and block progress
• Making messages too large or embedding data that should be fetched by reference
• Ignoring consumer lag monitoring, so backlogs grow silently until they overflow

See Also

system-design-scalability system-design-microservices system-design-apis