Idempotency
Running an operation twice leaves the same state as running it once. It is the property that makes retries and at-least-once queues safe.
An operation is idempotent if doing it once or five times leaves the system in the same state. Networks drop responses, so clients and queues retry, and a retry means the same request can arrive twice. If your write is idempotent, the second copy is harmless: no duplicate chunks in the index, no second charge on a card, no second email. You get there by giving each request a stable key and refusing to apply the same key twice. Pairs naturally with Exponential backoff and a Dead-letter queue (DLQ).
Pressing the lift call button ten times still brings one lift. Pressing a vending-machine button ten times can drop ten snacks. Retries are only safe on the first kind of button.
Generating the idempotency key inside the retry loop. A fresh UUID per attempt makes every retry look like a new request, so you get exactly the duplicates the key was meant to prevent. The key must be minted once, before the first attempt, by the caller.
Idempotency is designed in with a deterministic key, never bolted on. For data writes, derive the record id from the input (sha256(doc_id | chunk_no | embed_model_version)) and upsert, so a replay overwrites instead of appending. For side-effecting APIs, accept an Idempotency-Key header (IETF httpapi draft; Stripe's design), store key → (request_hash, response) in the same transaction as the side effect, return the stored response on replay, and reject a reused key with a different body with 422. Keep keys at least as long as your longest retry horizon: Stripe prunes after 24 hours, but an SQS message can be redelivered for up to 14 days, so a queue consumer needs a key window that covers retention. Kafka, SQS standard queues and Temporal activities are all at-least-once: idempotent consumers are how you get effectively-once outcomes without distributed transactions. The traps are hidden non-idempotent steps inside an otherwise safe pipeline: auto-increment ids, count = count + 1, webhooks and emails fired mid-job, and LLM calls whose output differs per run, so a replay writes a different summary under the same key.
Caller-minted Idempotency-Key (UUIDv7) on every mutating API; Postgres table idempotency_keys(account_id, key, request_hash, response, created_at) with a unique index on (account_id, key), written in the same transaction as the side effect; 24 h retention for synchronous APIs, retention ≥ queue retention for consumers. Data writes use content-derived ids plus upsert.
An insurance-claims intake worker parses a 30-page PDF into 58 chunks, embeds them and crashes before acking. SQS redelivers after the 5-minute visibility timeout. Chunk ids are sha256(claim_id|chunk_no|embed=v4) and the write is a Qdrant upsert, so the replay overwrites the same 58 points: index count stays 58, not 116. The 'claim received' email is sent only after a row in notifications(claim_id, kind) with a unique constraint is inserted, so the redelivery hits a unique-violation and skips the second email.
If the system has a queue or a retry anywhere, I assume at-least-once delivery and make every consumer idempotent with a key derived from the input, not a random UUID minted at write time. For external side effects I store the idempotency key and response in the same transaction as the effect. Expect the follow-up: 'what if the LLM call in the middle is non-deterministic?' Answer: persist the model output keyed by input hash before any downstream write, so replays reuse it instead of regenerating.
Numbers worth knowing
- Stripe idempotency-key retention
- 24 hourskeys can be pruned after that; a retry later is treated as new
- SQS max message retention
- 14 daysa consumer's dedupe window must cover this or redeliveries slip through
- Key-store overhead
- 1 indexed insert + 1 read per request~1–3 ms on Postgres when the key table fits in memory
How to decide
| When | Do this |
|---|---|
| Write is a pure data upsert (vectors, rows) | Derive the id from content + version; no key table needed. |
| Request triggers an external effect (charge, email, ticket) | Store key + response transactionally with the effect; replay the stored response. |
| Pipeline contains a non-deterministic LLM step | Cache the model output by input hash before downstream writes so replays are byte-identical. |
| Same key arrives with a different body | Reject with 422; never silently apply the new body. |
Cost maths
Cost = key-table storage + one extra lookup per request. At 2M mutating requests/day × 1 KB stored response × 1-day retention ≈ 2 GB of rows, under $1/month of Postgres storage. Compare with one duplicate side effect: a single double-charged $120 claim payout or a duplicate 58-chunk ingestion that doubles retrieval hits for that claim. The table pays for itself on the first prevented duplicate.
Operating it
- Key-table writes exceed ~5k/s on one Postgres primary: move keys to DynamoDB or Redis with a conditional put
- Duplicate rate in reconciliation >0.01% of writes
- Consumers start doing cross-service effects (payments, outbound email) rather than local upserts
- Idempotent-replay rate (stored response served): alert if >2% of requests, signals client retry storms or timeouts set too low
- Key-conflict 422s (same key, different body): alert on any sustained rate >0, it means a client bug
- Duplicate rows found by reconciliation: alert >0.01% of writes
When it goes wrong
Some clinical intake summaries appeared twice in the clinician's queue, with slightly different wording.
Nightly Reconciliation job found 0.4% of intake ids with two summary rows; wording differed, so a content-hash dedupe never caught them.
Summary rows were keyed by a random UUID generated after the LLM call. Kafka redelivered after a consumer rebalance, the model produced a second summary, and the insert had nothing to collide with.
Key the summary by intake_id + prompt_version, upsert, and persist the model output before publishing downstream; added a unique constraint so a regression fails loudly.
Where it shows up
Sources
- The Idempotency-Key HTTP Header Field (draft) — IETF HTTPAPI WG, 2025spec
- Idempotent requests — Stripe, 2025docs
- Implementing Stripe-like Idempotency Keys in Postgres — Brandur Leach, 2017blog
Check yourself
Three questions. Every option is explained once you answer.