Debugging drill: one tenant's agent runs are looping — hitting the 50-step cap and costing 30× normal. Walk through it.
- 1 min — contain: circuit-break that tenant's agent spend
- 2 min — symptoms and ranked hypotheses
- 3 min — group-by (tool, args hash, result hash) query and what each pattern means
- 2 min — fix the tool, add no-progress detection, budgets, trajectory tests
Symptoms: one tenant only; runs end at the step cap, not a natural stop; token spend 30× baseline; other tenants on the same model and prompt version are fine. Hypotheses ranked: (1) a tool returns an error or empty result the model can't act on (tenant-specific permissions, a 403 or schema the model doesn't understand) and it retries forever; (2) pagination returns the same cursor, so 'get next page' never ends; (3) tenant data contains instruction-like text — indirect injection — that redirects the loop; (4) tenant config disables a tool the prompt still describes; (5) huge tool outputs truncated mid-result, so the model re-requests.
The one query that splits them: for that tenant's looping runs, group steps by (tool name, argument hash, result hash). Identical call + identical result repeated → (1), (2) or (5) — then read the result: error text → (1), same cursor → (2), truncation marker → (5). Varied calls drifting off-task after one specific document is read → (3). Calls to a tool that returns 'not enabled' → (4). Fix: correct the tool (actionable error messages, cursor validation), and add loop guards: repeated identical call/result ≥3 times ends the run with a clear failure.
Prevention: per-run token budgets and per-tenant spend circuit breakers in the gateway; stop conditions that detect no-progress, not just step count; tool errors written for the model ('cursor expired — restart from page 1'); and per-tenant trajectory regression tests built from this incident. The agent platform team owns the guards and budgets so every product team inherits them — a loop in one product is a bill for all of them.
Cost of a loop: context grows each step; assume 8k-token base plus 1.5k per step. Over 50 steps, average context ≈ 8k + 25 × 1.5k ≈ 45k → 50 × 45k ≈ 2.3M input tokens per run vs ~60k for a normal 5-step run (~38×). At an illustrative $3 / M input (no cache) ≈ $6.90 per looping run; with prompt caching reads at 0.1× most of the context, ≈ $1. At 2,000 runs/day for that tenant: ~$14k/day uncached — a budget breaker within hours, which is why the per-tenant circuit breaker should trip at ~3× the tenant's trailing daily spend.
What the answer sounds like at each level
Lowers the step cap and reads a few traces.
Groups steps by call/result hash to classify the loop, fixes the tool, adds repeat-call detection.
Adds per-tenant budgets and circuit breakers at the platform layer, builds trajectory tests from the incident, and quantifies the cost exposure.
Follow-ups the interviewer will push
1Now it's a regulated tenant and the loop sent 40 duplicate emails.
Side-effecting tools need idempotency keys scoped to the run and step, and approval above a threshold. Add a per-run cap on side-effect calls separate from the step cap, and report the incident to the tenant.
2Now loops appear across all tenants after a model upgrade.
That's a model-behaviour change, not data: replay a frozen set of trajectories on old vs new model and compare tool-call distributions. Roll back the model pin while you adjust tool descriptions.
Red flags interviewers listen for
- Raises or lowers the step cap and calls it fixed.
- No per-tenant cost visibility, so the loop is found on the invoice.
- Blames the model without looking at what the tools returned.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Containment | Stops the spend first, scoped to the tenant. |
| Diagnosis | Uses call/result hashing to classify the loop rather than reading traces at random. |
| Prevention | Platform-level budgets, no-progress stop conditions, model-readable tool errors. |
Sources
- Building effective agents — Anthropic, 2024blog
- ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., 2022paper