How do you make agent tool calls reliable — retries, timeouts and idempotency?
- 2 min — why: compounding failure maths
- 3 min — timeouts, backoff with jitter, retryable vs not
- 3 min — idempotency key design and the dropped-response example
- 2 min — retry amplification, budgets, circuit breakers
- 2 min — ownership: runtime policy vs endpoint contracts
Agents make 10–50 tool calls per task, so transient failure is the normal case. With a 2% transient failure rate per call and 20 calls, only 0.98²⁰ = 67% of tasks see no failure. Three attempts per call drop per-call failure to 0.02³ = 8×10⁻⁶ and lift that to 99.98% — retries are worth it, if they are safe.
- Timeouts: set from the dependency's latency distribution (e.g. p99 1.2 s → timeout 3 s), always shorter than the caller's own deadline.
- Retries: only on transient errors (timeouts, 429, 503), with exponential backoff and full jitter — sleep = random(0, min(cap, base × 2ⁿ)), base 200 ms, cap 5 s — honouring
Retry-After. - Idempotency: every mutating tool accepts an idempotency key; the server stores key → result and returns the stored result on replay (Stripe's design keeps keys for 24 hours).
- Non-transient errors go back to the model as actionable text, not retried.
E-commerce fulfilment agent calls purchase_shipping_label(order_id, carrier, service, idempotency_key). The key is derived deterministically: hash(run_id, step_id, tool_call_id). The carrier API commits the purchase, then the connection drops before the response arrives; the tool layer retries after 340 ms with the same key and receives the original label — one charge, not two. A 400 address_unverifiable returns to the model as “address failed carrier validation: missing apartment number”, and the agent asks the customer instead of retrying.
There are two retry layers — the tool runtime and the model choosing to call again — and only the runtime should handle transient errors. Stacked retries amplify outages: 3 layers × 3 attempts is 27× load on a dependency that is already failing. Use a retry budget (retries ≤ 10% of requests per dependency, as in Google SRE guidance) and circuit breakers that return “carrier API unavailable, try later” to the model. The subtle hole: the model “helpfully” re-calls a mutating tool with slightly different arguments, which a per-call key cannot dedupe — so also dedupe on business identity (one active label per order + shipment), enforced by the system of record. Permanently failed operations go to a dead-letter queue with the run id.
Org scope. The platform team owns retry policy, budgets, circuit breakers and key derivation in the shared tool runtime; system-owning teams must expose idempotent mutating endpoints as a condition of tool registration — non-idempotent tools require a human gate. Measure tool error rate, retry rate and duplicate side effects per 10k runs (target 0). Failure mode: a retry storm during a carrier outage that also exhausts the agent's step budget, turning a 5-minute blip into a backlog of failed runs.
Assumptions: 20 tool calls per task, 2% independent transient failure per call. No retries: 0.98²⁰ = 66.8% clean tasks. Up to 3 attempts: per-call failure 0.02³ = 8×10⁻⁶ → 0.999992²⁰ ≈ 99.98%. Extra load from retries in steady state ≈ 2% + 0.04% ≈ 2.04% — well inside a 10% retry budget. During an outage the budget, not the retry count, caps load.
What the answer sounds like at each level
Retry failed tool calls with exponential backoff and make write operations idempotent so retries don't duplicate them.
Quantifies task-level reliability with and without retries, uses full jitter and timeouts from p99, derives keys deterministically, and dedupes on business identity against model-level re-calls.
Centralises retry budgets and circuit breakers, makes idempotent endpoints a tool-registration requirement across teams, and tracks duplicate side effects as a zero-tolerance metric.
Follow-ups the interviewer will push
1Where does the idempotency key come from — the model?
No. The runtime derives it from run id, step id and tool-call id so a replay of the same step reuses it and a new step gets a new one. A model-generated key can be re-generated differently and defeat dedupe.
2Now the carrier API is down for 20 minutes. What does the user see?
The circuit opens after the error budget trips; the tool returns a clear unavailability message; the agent tells the user the label will be created when the carrier recovers, and the run parks on a durable timer instead of spinning.
3The downstream API doesn't support idempotency keys. Options?
Wrap it: a small service that records key → outcome before forwarding and checks for an existing business object (label for this order) before creating. If even that can't be guaranteed, gate the tool behind approval.
Red flags interviewers listen for
- Retries every error, including validation failures.
- Retries at the model layer and the tool layer and the HTTP client simultaneously.
- Relies on the idempotency key alone and ignores business-level duplicates.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Reliability maths | Computes task success with and without retries and the amplification factor. |
| Idempotency | Deterministic keys plus business-identity dedupe. |
| Overload safety | Retry budgets, circuit breakers, single retry layer. |
| Org scope | Idempotent endpoints as a registration contract; platform-owned policy. |
Sources
- Designing robust and predictable APIs with idempotency — Stripe (Brandur Leach), 2017blog
- Exponential Backoff and Jitter — AWS Architecture Blog (Marc Brooker), 2015blog
- Addressing Cascading Failures (Site Reliability Engineering, ch. 22) — Google, 2016book