Dead Letter Queues and Replay

Every messaging system eventually receives a message it cannot process: a malformed payload from a buggy producer, an event that refers to a record that does not exist yet, a consumer bug triggered by an unusual value, or a downstream service that is down. Without a plan, that message does one of two bad things. It is retried forever, burning capacity and — in ordered systems — blocking every message behind it. Or it is dropped, and an order, a payment, or an email silently disappears.

A dead-letter queue (DLQ) is the middle path: after a bounded number of attempts, the failing message moves to a separate queue where it no longer blocks anything but is preserved, with context, for someone to investigate. Replay moves it back once the cause is fixed. Both are simple to set up and easy to get wrong, because a DLQ nobody watches is just a slower way to lose data.

TL;DR

Quick Example

An Amazon SQS queue whose messages move to a dead-letter queue after five failed receives, defined with Terraform:

In a standard SQS queue, a message keeps its original enqueue time when it is dead-lettered, so the DLQ's retention must be longer than the source queue's — otherwise messages can expire soon after arriving.

Core Concepts

Transient Versus Permanent Failures

A consumer that treats every exception the same either retries hopeless messages many times or gives up on messages that would have succeeded a minute later.

Retry Policies

Retries should be bounded (a maximum number of attempts or a maximum age), spaced out (exponential backoff with jitter, so a recovering dependency is not hit by synchronized retries), and visible (attempt counts in logs and metrics). Where the broker supports it, use delayed redelivery rather than sleeping inside the consumer, which holds a worker hostage.

Poison Messages

A poison message fails every time it is processed. In a queue with competing consumers, it wastes a worker on every attempt. In an ordered stream such as a Kafka partition, a consumer that keeps retrying it stops processing everything behind it, and consumer lag climbs until someone notices.

What to Keep With a Dead Letter

The original payload is not enough to debug a failure days later. Preserve:

Dead-Lettering in Common Brokers

Amazon SQS

A redrive policy moves messages to the DLQ when their receive count exceeds maxReceiveCount. For permanent failures the consumer can dead-letter a message immediately, with context, instead of waiting for five receives:

Unexpected exceptions propagate without deleting the message, so the redrive policy eventually catches anything the code did not anticipate.

RabbitMQ

Queues can name a dead-letter exchange. Messages are dead-lettered when a consumer rejects them without requeueing, when their TTL expires, when the queue overflows, or when a quorum queue's delivery limit is exceeded. RabbitMQ adds an x-death header recording the reason, source queue, and count. Delayed retries are commonly built from a "wait" queue with a TTL whose dead-letter exchange routes messages back to the work queue. See RabbitMQ.

Kafka

Kafka brokers do not dead-letter for consumers: a consumer simply decides whether to commit an offset. Blocking a partition on one bad record is rarely acceptable, so the common pattern is:

Retry topics hold records for increasing delays before they are consumed again; after the last attempt, the record goes to a dead-letter topic and the original offset is committed. Spring for Apache Kafka provides this as non-blocking retries, and Kafka Connect has built-in dead-letter topic support for connectors. If ordering per key matters, sending one record aside while later records for the same key continue can apply events out of order — park the key's subsequent records too, or make consumers tolerate reordering. See Kafka Consumer Groups.

Operating Dead-Letter Queues

Alert and Own

Alert when dead letters appear in critical queues, when the oldest message passes an age threshold, and when the rate of new dead letters jumps. Each DLQ should belong to the team that owns its consumer, with a runbook covering how to inspect, replay, and discard messages. One DLQ per consumer, rather than a shared dumping ground, keeps ownership clear.

Triage by Cause

Grouping dead letters by error type on a dashboard turns thousands of messages into a handful of decisions.

Retention and Data Protection

Dead letters contain real customer data. Apply the same access controls, encryption, and retention rules as the main data store, and make retention long enough to cover a weekend, a holiday, and the time to ship a fix.

Replaying Safely

  1. Fix the root cause and deploy it; replaying into the same bug only moves messages in a circle.
  2. Inspect a sample of each error group and decide: replay, transform and replay, or discard.
  3. Confirm idempotency: consumers must deduplicate by message or business key, because some replayed messages may already have been partly processed. See Idempotency.
  4. Check for staleness: replaying a three-day-old "update address" message could overwrite a newer address. Use version numbers or timestamps so stale updates are ignored.
  5. Replay gradually, starting with a few messages, at a rate downstream systems can absorb, and watch for messages failing again.
  6. Record what was replayed and discarded, by whom, and why.

On SQS, a message move task returns messages from a DLQ to their source queue at a controlled rate:

Best Practices

Test the Failure Path

Publish a malformed message and a message that triggers a transient failure in integration tests, and assert that the first is dead-lettered with context and the second is retried and succeeds.

Make Messages Self-Describing

Include a message ID, a schema version, and the identifiers a consumer needs. A dead letter that can be understood on its own is far faster to triage.

Keep Business Outcomes Out of the DLQ

A declined payment is a normal result, not a failure. Model it explicitly so the DLQ contains only messages that genuinely need a human.

Automate Only Known-Safe Replays

Automatic redrive is reasonable for well-understood transient causes with a cap on attempts. Everything else needs triage first.

Common Mistakes

Unbounded Retries

A DLQ Nobody Watches

Replaying Before Fixing

Losing the Context

FAQ

What is a dead-letter queue?

A separate queue or topic that receives messages a consumer could not process after a bounded number of attempts. It keeps failing messages from blocking or slowing the main flow while preserving them for investigation and replay.

How many retries should happen before dead-lettering?

Enough to ride out ordinary transient failures — often three to five attempts with exponential backoff spanning seconds to minutes. Permanent failures should skip retries entirely. Tune using how long your dependencies' typical blips last.

Does Kafka have dead-letter queues?

Not at the broker level for consumers. Consumers implement them by publishing failed records to retry and dead-letter topics, often through framework support such as Spring for Apache Kafka. Kafka Connect provides a built-in dead-letter topic option for connectors.

Should every queue have a DLQ?

Every queue whose messages matter should have a defined failure path — usually a DLQ with alerting. For purely disposable messages, such as cache invalidation hints that are superseded by the next message, dropping after retries may be acceptable, but make that a deliberate decision.

How long should dead letters be kept?

Longer than the time to detect a problem, fix it, and deploy — at minimum several days, often the broker's maximum. Balance that against data protection rules, since dead letters contain the same personal data as the main flow.

Related Topics

References