What is durable execution, and why do long-running agents need it?
- 2 min — definition: event history, replay
- 2 min — why agents: deploy frequency vs run duration, approvals
- 3 min — onboarding example with timers and signals
- 3 min — determinism, idempotency gap, history size, versioning
- 2 min — platform ownership and managed vs self-host
Durable execution persists every step of a workflow — model calls, tool results, timers, decisions — to an event history, so a crash, deploy or timeout resumes from the last completed step instead of restarting. On recovery, completed steps are replayed from the history rather than re-executed, which is what makes resumption safe for side-effecting tools. Engines: Temporal, Restate, DBOS, AWS Step Functions; LangGraph checkpointers give a lighter version for graph state.
Why agents need it: runs are long and waits are longer. A team deploying 20 times a day (every ~72 minutes) running 2-hour agent tasks will see most runs span a deploy. Human approvals stretch runs to days. Without durability, every deploy either kills in-flight runs or forces you to drain for hours.
Employee onboarding agent: provision accounts in 5 systems (IdP, HRIS, payroll, laptop order, Slack), wait up to 3 days for a manager to approve elevated access, then finish. Each tool call is an activity with its own retry policy and idempotency key; the approval is a durable signal with a 72-hour timer that escalates to the manager's manager. A deploy at hour 30 doesn't matter — the worker restarts, replays the history (no tool re-executed), and keeps waiting. Over 1,800 runs a quarter, resume success is 99.9% and orphaned runs found by the nightly reconciliation job dropped from 40 to 2.
The cost is discipline. Workflow code must be deterministic on replay: model calls, clock reads, random ids and tool calls are wrapped as recorded activities. Each activity still needs idempotency for the gap between “side effect happened” and “completion recorded”. Keep state small: Temporal caps event history (around 50k events / 50 MB per workflow, with warnings far earlier), so store transcripts and tool payloads in blob storage and keep references in workflow state. Plan versioning: a run paused for 3 days resumes on new code — use the engine's versioning/patching API and keep prompts pinned per run so a mid-flight prompt change doesn't alter replay.
Org scope. Durable execution is shared infrastructure owned by the platform team (cluster, upgrades, versioning guidance, dashboards for stuck and failed runs); product teams write workflows. Build vs buy: managed Temporal Cloud or Step Functions before self-hosting — the operational burden is a persistence cluster plus history shards. Roll out by moving the longest-running, side-effect-heaviest agent first. Failure mode: storing 200k-token transcripts in workflow state until history limits or replay latency take the workers down.
Assumptions: 10k runs/day, 30 steps per run, ~3 history events per step (scheduled, started, completed) → ~900k events/day, ~90 per run (well under limits). With payloads in blob storage and ~1 KB per event, history ≈ 0.9 GB/day; 30-day retention ≈ 27 GB. Deploys every 72 min vs 2-hour runs → P(run spans ≥1 deploy) ≈ 1 − e^(−120/72) ≈ 81%.
What the answer sounds like at each level
Durable execution saves progress after each step so the agent can resume after a crash instead of starting over.
Explains replay semantics, determinism constraints, activities with idempotency, durable timers for approvals, and keeping payloads out of history.
Runs durable execution as shared infrastructure with versioning policy and stuck-run dashboards, sequences adoption by risk, and makes the managed-vs-self-host call.
Follow-ups the interviewer will push
1A prompt change ships while 300 runs are paused. What happens?
Pin prompt and model version per run at start, so in-flight runs replay and continue on their original versions; new runs pick up the change. Use engine versioning for any code-path change, and retire old versions when their runs drain.
2Now 10× runs. What scales and what breaks?
Workers scale horizontally; the persistence layer and history size are the limits. Event count per run × runs per day sets storage and shard load, so keep steps coarse and payloads external, and watch schedule-to-start latency.
3Why not just checkpoint the transcript to Postgres after each step?
You can for simple cases, but you'll rebuild timers, signals, retries, versioning and exactly-once-ish step recording. That's the engine; buy it unless you have a reason not to.
Red flags interviewers listen for
- Thinks durability removes the need for idempotent tools.
- Calls the model or reads the clock directly in deterministic workflow code.
- Stores full transcripts in workflow history.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Semantics | Correct replay model and what must be wrapped as activities. |
| Edge cases | Idempotency gap, versioning of paused runs, history limits. |
| Operations | Reconciliation, stuck-run metrics, resume success rate. |
| Org scope | Shared platform, adoption order, build-vs-buy reasoning. |
Sources
- Temporal documentation — Temporal Technologies, 2025docs
- Designing robust and predictable APIs with idempotency — Stripe (Brandur Leach), 2017blog