Durable execution & checkpointing
Running agent workflows so progress is checkpointed and they resume after crashes, deploys or long waits without redoing completed side effects.
Agents can run for minutes or hours, and may pause for days waiting for a human to approve something. If the server restarts halfway, an ordinary program forgets everything and starts over — possibly sending the same email twice.
Durable execution saves progress after each step, so the run picks up exactly where it left off after a crash or a long wait.
A video game with autosave: when the power cuts, you reload at the last checkpoint, not the start, and you don't collect the same reward twice.
Restarting failed runs from scratch, or replaying them by re-calling the model. Repeated side effects send duplicate emails and double-charge; regenerated model outputs take a different path than the one already half-executed.
Approaches: workflow engines that record an event history and deterministically replay code (Temporal, Restate, AWS Step Functions-style state machines), step-level checkpointing to a database after each The agent loop (ReAct) turn (LangGraph checkpointers), or queue-based state machines. The core rule: each side-effecting step's result is persisted, so on resume completed steps are read from the log, not re-executed.
Agent-specific concerns: model calls are non-deterministic, so their outputs must be recorded as activity results, never regenerated on replay; tools with side effects need Idempotency keys because a crash can land between "action done" and "result saved"; Human in the loop waits become durable timers or signals; retries use Exponential backoff and land in a Dead-letter queue (DLQ) when exhausted; and code changes must be versioned so a deploy doesn't break in-flight runs. Watch history size: Temporal warns at 10,240 events / 10 MB and hard-limits a workflow at 51,200 events / 50 MB, so long agent loops use continue-as-new and keep large payloads out of history.
Short runs (< 5 min, no human wait): checkpoint state to Postgres after each turn. Long or human-gated runs: a workflow engine (Temporal or similar) with model calls and tools as activities, idempotency keys on every side effect, retry policy with backoff and a max of 5 attempts, payloads > 100 KB stored externally by reference, continue-as-new every ~1,000 turns, versioned workflow code.
Clinical-trial onboarding agent: 7 steps over 2–5 days — verify site, create accounts, request IRB document, wait for coordinator approval, provision EDC access, email welcome pack, schedule kickoff. The worker crashes after step 5 during a deploy. On restart the engine replays history, skips 1–5, and retries step 6 with idempotency key site-4412:welcome, so the coordinator gets exactly one email. The 3-day approval wait cost nothing while paused.
Answer "what if the process dies mid-run?" with three things: recorded model outputs, idempotent side effects, versioned workflow code. Recommend a workflow engine once runs span human waits or exceed a few minutes; a checkpoint table is fine for short runs.
Expect: "what about deploys with 10k runs in flight?" — versioning/patching APIs and draining old workers, not a big-bang restart.
Numbers worth knowing
- Temporal history warning
- 10,240 events or 10 MBper workflow execution, Temporal docs
- Temporal history hard limit
- 51,200 events or 50 MBsame source; use continue-as-new before it
- Activity retries
- ≤5 with exponential backoffthen dead-letter for a human
How to decide
| When | Do this |
|---|---|
| Runs finish in seconds and have no side effects | No durability layer; just retry the whole run. |
| Runs have side effects but finish in minutes | Checkpoint per turn + idempotency keys. |
| Runs wait on humans or external events for hours to days | Workflow engine with durable timers and signals. |
Cost maths
Without durability, a crash costs a full re-run: P(crash) × run cost + duplicate-side-effect cost. 20k runs/month, 2% interrupted by deploys or crashes, $0.60 of tokens each → $240/month of re-run tokens; the bigger cost is duplicates: if 30% of interrupted runs resend an invoice email, that's 120 duplicate customer emails a month. Engine overhead ≈ a few cents of storage and compute per 1,000 runs.
Operating it
- Runs routinely wait on humans for > 1 hour
- Deploys interrupt > 1% of in-flight runs
- Workflow history approaches 10k events (switch to continue-as-new)
- Runs resumed vs restarted after failure (alert on any restart-from-scratch)
- Duplicate side-effect detections by idempotency key (alert on any)
- Workflow history size p99 (alert at 8k events)
- Dead-letter queue depth (alert > 0 for > 1 hour)
When it goes wrong
After a Friday deploy, 340 customers received two copies of their contract-renewal email.
Customer replies asking which version was correct; the email provider's logs showed paired sends 6 minutes apart.
The renewal agent checkpointed after each model call but sent email before writing the checkpoint; the deploy killed workers between send and save, and on restart the step re-ran without an idempotency key.
Email sends carry an idempotency key renewal:{contract_id}:{version} honoured by the provider; checkpoint and send are reconciled on resume; deploys drain workers first.
Related terms
Where it shows up
Sources
- Temporal Workflow Execution limits — Temporal, 2025docs
- Persistence — LangChain (LangGraph docs), 2025docs
Check yourself
Three questions. Every option is explained once you answer.