Message Queues & Event Streaming
Message queues and event streams let you build systems that are asynchronous, decoupled, and resilient under load. They're the backbone of background jobs, workflows, and event-driven architectures — work gets handed off and processed reliably instead of blocking the request.
The catch is that messaging is distributed-systems territory: messages can be delivered twice, arrive out of order, or get stuck. Design for that from the start and it's a superpower; ignore it and it's a source of subtle data bugs.
TL;DR
- Use a queue for work distribution (each job processed once).
- Use a stream/log for events many consumers read independently.
- Assume at-least-once delivery: make consumers idempotent.
- Plan retries, backoff, and dead-letter queues from day one.
Quick Example
Producer enqueues work; a worker consumes and acknowledges — written so a redelivery is harmless:
Queue vs Stream
See Event-Driven Architecture.
Core Concepts
Delivery semantics
- At-most-once — may lose messages, never duplicates.
- At-least-once — never lose, but duplicates happen (the common default).
- Exactly-once — hard; only under specific constraints.
Build consumers to be safe under at-least-once.
Idempotency
Processing the same message twice should have the same effect as once. Use a dedupe table of processed IDs, natural keys (orderId), or conditional writes (INSERT ... ON CONFLICT DO NOTHING).
Ordering
Ordering is per-queue or per-partition, not global. If order matters, choose partition keys carefully (e.g. per account) or serialize work per entity.
Retries & dead-letter queues
Retry with exponential backoff; send poison messages to a dead-letter queue after a cap, and alert on DLQ depth so it stays actionable.
Patterns
Outbox (reliable publish)
To update the database and publish a message atomically: write the domain change and an outbox row in one transaction, then a publisher reads the outbox and emits to the broker. This avoids "DB updated but message lost" and vice versa.
Saga (distributed workflows)
Break a multi-service workflow into steps with compensating actions, coordinated by events or an orchestrator. See Microservices.
Background jobs & fan-out
Enqueue slow work (emails, reports, image resizing) for workers to process; one event can fan out to many consumers — streams are naturally good at this. See Background Jobs.
Operational Concerns
- Backpressure: if producers outpace consumers, queues grow and latency climbs — scale consumers, throttle producers, prioritize, or shed load. See Rate Limiting.
- Observability: track queue depth/lag, consumer error rate, message age, and DLQ count. See Logging.
Common Mistakes
Assuming exactly-once delivery
Unbounded retries
FAQ
Queue or stream — which do I need?
Use a queue when each message represents work to be done once (send this email, resize this image). Use a stream when an event should be readable by many independent consumers and possibly replayed (analytics, integrations, audit logs).
Kafka or RabbitMQ?
Kafka is a partitioned, retained log built for high-throughput streaming and replay. RabbitMQ is a flexible message broker great for task queues and routing. Pick based on whether you need durable replayable streams (Kafka) or work distribution and routing (RabbitMQ).
How do I avoid processing a message twice?
Make consumers idempotent: track processed message IDs, use natural business keys, or write conditionally. Since most brokers are at-least-once, dedupe rather than assuming exactly-once.
What is a dead-letter queue?
A separate queue for messages that repeatedly fail processing. Routing poison messages to a DLQ (after capped retries) keeps them from blocking the main queue, and lets you inspect and replay them.
Related Topics
- Event-Driven Architecture — Designing around events
- Background Jobs — Async work off the request path
- Microservices — Sagas and the outbox in context
- Rate Limiting — Shaping producer load
- Logging — Monitoring queue health