Durable agents: checkpoints, idempotent tools and approvals that wait for days
An agent run is a long-lived workflow that will be interrupted. How to checkpoint each step, make every side effect safe to replay, resume at step 7 after a crash, and park a run for a human without holding a worker, with a runnable stdlib example.
A demo agent loop runs in one process for thirty seconds. A production agent processing an insurance claim runs for nine steps, calls a payment system, and waits three days for an adjuster to approve the payout. In that time the process will be redeployed, a node will be reclaimed, and something will run out of memory. The question is not whether the run is interrupted but what happens next: start over and pay twice, or resume exactly where it stopped.
That property is durable execution. It rests on three rules.
- Checkpoint every step. After each step completes, persist its output before starting the next one. On restart, load the last checkpoint and continue from the following step.
- Make every side effect idempotent. Execution is at-least-once: a step that crashed after acting but before checkpointing will run again. An idempotency key turns at-least-once execution into exactly-once effects.
- Waiting is a state, not a thread. A run waiting for a human is a row in a database with status
waiting, not a sleeping process. Approval resumes it, on any worker, days later.
What goes in a checkpoint
Everything the next step needs and nothing it can recompute cheaply: the step counter, the accumulated state, the message history, and the recorded output of every model call and tool call. The last point is the one teams miss. An LLM call is not deterministic; if resume re-calls the model for step 3, it may choose a different tool at step 4 and your checkpoint for step 4 now describes a run that no longer exists. On resume, completed steps are read from the checkpoint, never re-executed.
Keep checkpoints small. Message histories grow with every tool result; store large artefacts (retrieved documents, files, screenshots) in object storage and checkpoint the reference. A 2 MB checkpoint written after each of 40 steps is 80 MB per run, and that shows up in latency and storage bills long before it shows up in a design review.
Idempotency: the key is the step, not the attempt
The idempotency key must be identical every time the same logical step runs, so derive it from the run and the step: claim-0042:step6. A fresh UUID per attempt defeats the purpose, because the retry carries a new key and the payment goes through twice. The key must also be honoured by the receiver of the side effect: Stripe's API, for example, accepts an Idempotency-Key header and returns the original result when a key repeats. Your own checkpoint cannot protect you here, because the dangerous crash is the one between the payment succeeding and the checkpoint being written.
For tools whose backend has no idempotency support, wrap them: check-then-act (read whether the ticket, email or record already exists, keyed by the run and step) and a Reconciliation job job that compares your step log with the external system. When designing the tool interface itself, take the key as a required argument; that is good tool design for agents.
Resuming at step 7: a runnable example
The program below processes two insurance claims. Each parks at step 5 for approval. Claim 0042 crashes during step 7; claim 0043 crashes in the worst place, inside step 6 after the payment succeeds but before its checkpoint is written. Save as durable.py and run python3 durable.py. Standard library only (sqlite3, json).
"""Checkpoint/resume with idempotency keys, stdlib only.
A 9-step claims-processing run. Each step's result is checkpointed in SQLite before
the next step starts. Side-effecting tools take an idempotency key derived from
(run_id, step), so a step replayed after a crash cannot pay twice.
"""
import json
import os
import sqlite3
import tempfile
DB = os.path.join(tempfile.mkdtemp(), "runs.db")
def connect():
conn = sqlite3.connect(DB)
conn.execute("CREATE TABLE IF NOT EXISTS checkpoints (run_id TEXT, step INTEGER, output TEXT,"
" PRIMARY KEY (run_id, step))")
conn.execute("CREATE TABLE IF NOT EXISTS runs (run_id TEXT PRIMARY KEY, status TEXT)")
# Stands in for the payment provider's own dedupe table: it lives on THEIR side.
conn.execute("CREATE TABLE IF NOT EXISTS payments (idem_key TEXT PRIMARY KEY, amount INTEGER)")
return conn
# ---- Tools ----
def pay_claimant(conn, idem_key, amount):
"""Idempotent: a repeated key returns the original result instead of paying again."""
row = conn.execute("SELECT amount FROM payments WHERE idem_key = ?", (idem_key,)).fetchone()
if row:
return {"paid": row[0], "replayed": True}
conn.execute("INSERT INTO payments VALUES (?, ?)", (idem_key, amount))
conn.commit()
return {"paid": amount, "replayed": False}
class Crash(Exception):
pass
class WaitingForApproval(Exception):
pass
CRASH_AT = {("claim-0042", 7), ("claim-0043", 6)} # (run, step) to kill once
APPROVALS = {} # run_id -> decision, written by a human days later
def maybe_crash(run_id, n):
if (run_id, n) in CRASH_AT:
CRASH_AT.discard((run_id, n))
raise Crash(f"worker killed during step {n} (OOM, deploy, spot reclaim)")
def step_fn(n, conn, run_id, state):
if n == 4:
state["amount"] = 1840
if n == 5 and run_id not in APPROVALS:
raise WaitingForApproval("adjuster must approve payouts over 1000")
if n == 5:
state["approved_by"] = APPROVALS[run_id]
if n == 6:
state["payment"] = pay_claimant(conn, f"{run_id}:step6", state["amount"])
maybe_crash(run_id, n) # worst case for step 6: money moved, checkpoint not yet written
return {"step": n, **state}
def run(run_id, total_steps=9):
conn = connect()
conn.execute("INSERT OR IGNORE INTO runs VALUES (?, 'running')", (run_id,))
done = conn.execute("SELECT step, output FROM checkpoints WHERE run_id = ? ORDER BY step",
(run_id,)).fetchall()
state = json.loads(done[-1][1]) if done else {}
state.pop("step", None)
start = len(done) + 1
print(f"[{run_id}] resuming at step {start}" if done else f"[{run_id}] starting")
for n in range(start, total_steps + 1):
try:
out = step_fn(n, conn, run_id, state)
except WaitingForApproval as e:
conn.execute("UPDATE runs SET status = 'waiting' WHERE run_id = ?", (run_id,))
conn.commit()
print(f" step {n}: parked ({e}); no worker is held while waiting")
return "waiting"
conn.execute("INSERT INTO checkpoints VALUES (?, ?, ?)", (run_id, n, json.dumps(out)))
conn.commit() # the checkpoint is durable before the next step begins
print(f" step {n}: done" + (f" -> {out['payment']}" if n == 6 else ""))
conn.execute("UPDATE runs SET status = 'completed' WHERE run_id = ?", (run_id,))
conn.commit()
return "completed"
def run_until_done(run_id):
"""What a supervisor does: restart crashed runs; leave parked runs alone."""
while True:
try:
return run(run_id)
except Crash as e:
print(f" CRASH: {e}; restarting")
if __name__ == "__main__":
for run_id in ["claim-0042", "claim-0043"]:
print(run_until_done(run_id)) # both park at step 5
APPROVALS[run_id] = "adjuster-17" # ...a human approves days later
print(run_until_done(run_id))
conn = connect()
for run_id, count, total in conn.execute(
"SELECT substr(idem_key, 1, 10), COUNT(*), SUM(amount) FROM payments GROUP BY 1"):
print(f"{run_id}: {count} payment, total {total}")Output:
[claim-0042] starting
step 1: done
step 2: done
step 3: done
step 4: done
step 5: parked (adjuster must approve payouts over 1000); no worker is held while waiting
waiting
[claim-0042] resuming at step 5
step 5: done
step 6: done -> {'paid': 1840, 'replayed': False}
CRASH: worker killed during step 7 (OOM, deploy, spot reclaim); restarting
[claim-0042] resuming at step 7
step 7: done
step 8: done
step 9: done
completed
[claim-0043] starting
step 1: done
step 2: done
step 3: done
step 4: done
step 5: parked (adjuster must approve payouts over 1000); no worker is held while waiting
waiting
[claim-0043] resuming at step 5
step 5: done
CRASH: worker killed during step 6 (OOM, deploy, spot reclaim); restarting
[claim-0043] resuming at step 6
step 6: done -> {'paid': 1840, 'replayed': True}
step 7: done
step 8: done
step 9: done
completed
claim-0042: 1 payment, total 1840
claim-0043: 1 payment, total 1840- Claim 0042 resumes at step 7, not step 1, because steps 1–6 have checkpoints. Step 6 is never repeated.
- Claim 0043 has no checkpoint for step 6, so step 6 runs again, and the payment tool returns
replayed: True. One payment each, as the final lines show. - Approval parks the run and returns. Nothing runs, or holds memory, while the adjuster decides.
What durable execution engines do for you
The example is about 100 lines because it ignores concurrency, timers, retries and versioning. Engines handle those. Describe them accurately, because their models differ.
| Temporal | LangGraph checkpointer | AWS Step Functions | |
|---|---|---|---|
| Unit of durability | Workflow event history | Graph state per thread_id | State machine execution |
| Resume model | Deterministic replay of code | Restart from last checkpoint | Continue at next state |
| Side effects | Activities, retried | Nodes; re-run on resume | Task states, retried |
| Human wait | Signal + durable timer | interrupt() + Command(resume) | .waitForTaskToken callback |
| Main pitfall | Non-determinism on code change | Node re-runs from its start | Payload size limits |
- Temporal. Workflow code is replayed from a persisted event history; side effects live in activities, whose results are recorded so replay reads them instead of re-executing. Workflow code must therefore be deterministic: model calls belong in activities, not in workflow code. Activities are retried with configurable backoff, so they still need idempotency keys. Humans interact through signals, and a workflow can wait on a durable timer for days. Changing workflow code while runs are in flight requires versioning.
- LangGraph. A checkpointer (InMemorySaver for tests; SqliteSaver or PostgresSaver for real use) persists graph state per
thread_idas the graph advances.interrupt()pauses a node for human input and the run resumes when you invoke the graph again with the same thread andCommand(resume=...). The documented caveat: on resume, the interrupted node restarts from its beginning, so any side effect beforeinterrupt()runs twice unless it is idempotent or moved into its own node. - AWS Step Functions. Standard workflows persist each state transition; the
.waitForTaskTokenpattern pauses a task until an external system returns the token, up to the service's one-year execution limit. It fits approval steps well; it is clumsy for open-ended agent loops whose next step the model decides. - Others in the same family include Restate, DBOS and Inngest. Pick on operational fit (managed vs self-run, language SDKs, how runs are inspected), not feature lists.
Approvals that wait for days
- Give every wait a TTL and an escalation. An approval nobody answers should expire to a defined outcome (reject, escalate, remind), never wait forever. Track runs in
waitingolder than the TTL. - Re-validate on resume. Three days later the claim may have been withdrawn or the price changed. The first thing the resumed step does is re-read the facts it is about to act on.
- Record who approved what. The approval is part of the run's state and audit trail: approver, time, and the exact payload they saw.
- Cap the run. Stop conditions still apply: maximum steps, maximum spend, maximum wall-clock age. A run that exhausts retries goes to a dead-letter queue for a human, not into an infinite retry loop with backoff.
How you know it is broken in production
- Duplicate side effects: count idempotency-key replays per tool; a non-zero rate is expected after crashes, but a key replayed with a different payload is a bug, so alert on it.
- Resume rate: resumes per completed run. Above ~0.1 means workers die too often; look at memory and deploy cadence.
- Stuck runs: runs in
runningwith no checkpoint for more than 10× the p95 step time, and runs inwaitingpast their TTL. - Checkpoint size p95 and growth per step; a sudden jump usually means someone checkpointed a whole document instead of a reference. This is the Observability & traces layer for agents.
Key takeaways
- Durable execution = checkpoint every step, make every side effect idempotent, and represent waiting as stored state rather than a sleeping process.
- Checkpoint the recorded outputs of model and tool calls; on resume, read completed steps, never re-call the model.
- Derive Idempotency keys from run and step, and have the side effect's receiver honour them. The dangerous crash is after the effect, before the checkpoint.
- Temporal replays deterministic workflow code over recorded activity results; LangGraph restores state per thread and re-runs an interrupted node from its start. Both are at-least-once for side effects.
- Approvals need a TTL, re-validation on resume, and an audit record. Monitor replays, resume rate, stuck and expired runs.
Sources
- Temporal Workflows — Temporal Technologies, 2026docs
- LangGraph: Persistence — LangChain, 2026docs
- LangGraph: Interrupts — LangChain, 2026docs
- Idempotent requests — Stripe, 2026docs
- Discover service integration patterns in Step Functions — AWS, 2026docs
- Building effective agents — Anthropic, 2024blog