How do you evaluate an agent? Outcome vs trajectory evaluation.
- 2 min — outcome vs trajectory, with a dangerous-route example
- 3 min — stochasticity: pass^k, confidence intervals
- 3 min — catalog example: sandbox, assertions, judge, gate
- 2 min — judge calibration and invariant-based trajectory checks
- 2 min — eval platform ownership and gate rollout
Outcome evaluation checks the end state in the environment: is the row correct, do the tests pass, is the order in the right status. Trajectory evaluation checks the path: right tools, valid arguments, no unsafe or wasted steps, step count within budget. Outcome tells you whether it works; trajectory tells you why not, and catches runs that reached the right answer by a dangerous route (deleted and recreated a record instead of updating it).
Agents are stochastic, so report reliability, not a single pass. τ-bench (Yao et al., 2024) introduced pass^k — the probability that all k trials of a task succeed — and showed strong function-calling agents below 50% pass^1 and under 25% pass^8 on its retail domain. Independence makes this brutal: 80% per trial gives 0.8⁴ = 41% pass^4. Size matters too: at n=100 tasks and an 80% pass rate, the 95% confidence interval is ±7.8 points, so a 3-point “improvement” is noise.
E-commerce catalog operations agent (“fix the size attributes for all women's running shoes from brand X”). The golden set is 240 tasks against a seeded catalog snapshot in a sandbox database. Outcome: SQL assertions on the final catalog state (correct attributes, no other SKUs touched). Trajectory: hard assertions (never calls delete_sku, ≤15 steps, every write preceded by a read of the same SKU) plus an LLM judge with a 4-item rubric for softer behaviour — asked for clarification when “size” was ambiguous between EU and US. Each task runs 4 trials; the release gate is pass^4 ≥ 70% and zero safety-assertion failures.
Prefer verifiable outcome checks over judges wherever the environment allows; calibrate any judge on 100–200 human-labelled trajectories and track agreement (target Cohen's κ ≥ 0.6) per judge version. Don't over-specify trajectories — assert invariants (forbidden tools, read-before-write, budget) rather than an exact sequence, or every valid alternative path fails. Evaluate against seeded sandboxes, never live services, so runs are reproducible; record tool responses for replay where a sandbox is impossible.
Org scope. An eval-platform team owns the harness, sandboxes, judge calibration and the CI gate; each product team owns its task set and adds every production incident as a regression task within a week. Build vs buy: buy or adopt the harness (Inspect, Braintrust, LangSmith-style tooling), build the environments — seeded sandboxes of your own systems are the irreplaceable part. Roll out the gate across teams in report-only mode for a month, then blocking. Failure mode: evaluating on live APIs, where yesterday's pass and today's fail differ because the data changed, not the agent.
Assumptions: 240 tasks × 4 trials = 960 runs; ~18 steps and ~250k input tokens per run, mostly cached → ≈ $0.40 per run at illustrative Oct 2026 prices ($3 / M input, $15 / M output, cache reads 0.1×). Full run ≈ $380 and ~25 min at 40 parallel sandboxes; nightly for a month ≈ $11.5k. 95% CI half-width at n=240, p=0.8: 1.96 × √(0.16/240) ≈ ±5 points.
What the answer sounds like at each level
Check the final result against the expected answer, and look at the tool calls to see where it went wrong.
Separates outcome and trajectory checks, uses seeded sandboxes, runs k trials and reports pass^k with confidence intervals, and calibrates judges against humans.
Owns an eval platform split (harness vs task sets), stages the CI gate across teams, and turns incidents into regression tasks on a deadline.
Follow-ups the interviewer will push
1Pass^1 went from 78% to 81% on 100 tasks. Ship?
Not on that alone: ±7.8 points at n=100 swamps a 3-point gain. Use paired comparison on the same tasks (McNemar-style on discordant tasks), add trials, and check pass^k and safety assertions didn't regress.
2Now 10× more tasks in the set. How do you keep eval cost sane?
Tier it: a 60-task smoke set on every PR, the full set with k=4 nightly and on release. At ~$0.40 per trial, 2,400 tasks × 4 = ~$3,800 per full run — nightly is fine, per-PR is not.
3How do you evaluate an action the sandbox can't simulate, like sending an email?
Replace it with a recording stub that captures the call, then assert on the captured payload (recipient on allow-list, no internal data) — outcome becomes “the right message would have been sent”.
Red flags interviewers listen for
- Single run per task, reported as a pass rate with no variance or interval.
- Exact-sequence trajectory matching that fails valid alternative paths.
- Uses an uncalibrated LLM judge as the only signal.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Measurement | pass^k, confidence intervals and paired comparison. |
| Environment | Seeded sandboxes and recorded stubs; no live services. |
| Judge hygiene | Calibrated against human labels with tracked agreement. |
| Org scope | Harness vs task-set ownership; staged gate; incident-to-regression SLA. |
Sources
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao et al., 2024paper
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., 2023paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper