Dead-letter queue (DLQ)
A side queue where messages land after N failed attempts, so one poison message stops blocking the pipeline and can be inspected and replayed later.
A dead-letter queue (DLQ) is a parking spot for messages your workers keep failing on. After a set number of attempts, say five, the queue moves the message aside instead of retrying it forever. The rest of the pipeline keeps flowing, and an engineer can look at the parked messages, fix the bug, and replay them. Without a DLQ, one corrupt PDF in an Ingestion pipeline can be retried thousands of times and starve everything behind it.
The post office's undeliverable-mail desk: letters with bad addresses go there instead of riding the delivery van in circles every day.
Treating the DLQ as a bin. Without an alert on depth and an owner, messages age out at retention and the data is silently lost; the DLQ just made the failure quieter.
In SQS a redrive policy moves a message to the DLQ once its receive count exceeds maxReceiveCount; set it to 3–5 for deterministic failures and higher only if failures are transient and you lack backoff. Set DLQ retention longer than the source queue's (SQS caps at 14 days) because a message's age is counted from its original enqueue. Kafka has no built-in DLQ: consumers (or Kafka Connect's errors.deadletterqueue.topic.name) publish failures to a separate topic with headers recording the error, attempt count and source offset. Temporal sidesteps queues for workflows: activities retry with a policy and a failed workflow stays inspectable, but non-retryable error types still need a parking path. The DLQ is only half the pattern: you need (1) classification of errors into retryable vs poison, (2) an alert on DLQ depth, and (3) a redrive tool (SQS StartMessageMoveTask) that replays after the fix, which only works if consumers are idempotent.
SQS redrive policy with maxReceiveCount=4, DLQ retention 14 days (source 4 days), CloudWatch alarm on ApproximateNumberOfMessagesVisible >0 for customer-facing queues; for Kafka, a <topic>.dlq topic with error/attempt/offset headers; error classifier that sends schema/validation errors straight to the DLQ.
A legal-contracts ingestion queue processes 40k PDFs/day. A batch of scanned contracts from one counterparty has malformed XRef tables and crashes the parser. With maxReceiveCount=4, each poison message burns 4 attempts (~2 minutes with backoff) and moves to contracts-dlq. The DLQ depth alert fires at 50 messages; the on-call sees all share error=PdfXrefError and source=counterparty-17, ships a fallback to OCR, and redrives 312 messages. Healthy throughput never dropped.
I'd attach a DLQ to every async consumer with maxReceiveCount around 4, classify errors so validation failures go straight to the DLQ without retries, and treat DLQ depth >0 as a page-worthy signal for anything customer-facing. The follow-up: 'how do you replay safely?' Idempotent consumers plus a rate-limited redrive, after you've fixed the cause, never before.
Numbers worth knowing
- maxReceiveCount
- 3–5with exponential backoff between attempts; higher only for known-transient dependencies
- SQS retention ceiling
- 14 daysDLQ retention must exceed the source's because age counts from first enqueue
- Wasted work per poison message
- maxReceiveCount × job coste.g. 4 × a $0.04 OCR+embed job = $0.16 each
How to decide
| When | Do this |
|---|---|
| Error is deterministic (bad schema, corrupt file, 4xx) | Send to DLQ immediately; retrying cannot help. |
| Error is transient (429, 5xx, timeout) | Retry with backoff up to maxReceiveCount, then DLQ. |
| Workflow has multi-step state | Use Temporal retry policies; route non-retryable failures to a parking table with full context. |
| DLQ holds >1% of a day's volume | Stop redriving; it is a systemic bug, open an incident. |
Cost maths
Without a DLQ a poison message retries until retention: at a 30 s visibility timeout that is ~2,880 attempts/day. If each attempt runs OCR plus embedding at $0.04, one bad file costs ~$115/day and occupies a worker slot permanently. With maxReceiveCount=4 it costs $0.16 once.
Operating it
- More than one DLQ per team with no shared redrive tooling
- DLQ redrives happen weekly: invest in error classification and auto-remediation
- Messages expire from the DLQ unreviewed
- DLQ depth: alert >0 for customer-facing pipelines, >100 for batch
- Age of oldest DLQ message: alert >50% of DLQ retention
- DLQ inflow rate as % of source throughput: alert >0.5%
When it goes wrong
An e-commerce catalog search showed stale prices for ~3% of SKUs for two weeks.
A merchant complaint; investigation found 41k messages in the price-update DLQ, oldest 13 days.
A supplier added a currency field the consumer rejected. The DLQ caught it correctly but had no alarm, and redrive would have hit retention the next day.
Depth alarm >0 paging the owning team, schema evolution with tolerant parsing, and a weekly DLQ review in the ops checklist.
Where it shows up
Sources
- Using dead-letter queues in Amazon SQS — AWS, 2025docs
- Apache Kafka documentation — Apache Software Foundation, 2025docs
Check yourself
Three questions. Every option is explained once you answer.