What makes a good tool for an agent?
- 2 min — tools as an interface (ACI); the absolute-path example
- 4 min — design rules: few, task-named, enums, limits, actionable errors
- 3 min — before/after example with step and token numbers
- 2 min — measuring tools from trajectories
- 2 min — registry ownership and sprawl control
Tool design is interface design for a reader that sees only the name, description and schema. Anthropic's Building effective agents (2024) calls this the agent-computer interface (ACI) and recommends investing in it as much as in prompts — their SWE-bench agent spent more effort on tools than on the overall prompt, and switching file tools to require absolute paths removed a whole class of errors (“poka-yoke” the arguments).
Concrete rules: few and distinct (selection accuracy degrades as near-duplicate tools accumulate; keep a default set ≤15–20); named for the task (search_contracts, not post_v2_query); descriptions that say when not to use the tool; enums and required fields instead of free strings; compact, ranked results with a limit (default 10) and stable ids; and actionable errors that tell the model what to try next.
Legal contract review agent. v1 exposed list_documents (returned 4,000 rows ≈ 90k tokens) plus a thin get_clause(doc_id, offset, length). Traces showed 30% of runs overflowing context and the model paging through clauses by offset. v2 exposed search_contracts(query, counterparty?, limit=10) returning title, id, effective date and the best-matching clause snippet (~1.2k tokens), and get_clause(doc_id, clause_type) with clause_type as an enum of 14 values (indemnity, limitation_of_liability, termination…). Median steps fell from 11 to 5 and context overflows went to 0 on the 120-task eval.
Evaluate tools like a public API: run the task set, read trajectories, and count wrong-tool picks, invalid arguments and redundant calls per 100 tasks — those are tool-description bugs as often as model bugs. Give each tool a response-size budget (e.g. ≤2k tokens, truncate with a “refine your query” hint). For safety, mutating tools take an idempotency key, run with least-privilege credentials, and are separate from read tools so a approval gate can target them.
Org scope. Tools are owned by the team that owns the underlying system, published to a shared registry (often as MCP servers) with a description review, a schema version and a per-tool eval. The platform team owns the registry, selection metrics and the size budget lint. Build vs buy: wrap vendor APIs yourself rather than exposing a vendor's 200-endpoint surface; generated wrappers of every endpoint are the most common cause of tool sprawl. Failure mode: tool count creeps to 60, selection accuracy drops, and someone lengthens the system prompt instead of consolidating or loading tools on demand.
Endpoint wrappers
- list_documents → 90k tokens
- get_clause(offset, length)
- 11 median steps
- 30% context overflow
Task-level tools
- search_contracts(limit=10)
- get_clause(clause_type enum)
- 5 median steps
- 0 overflows (n=120)
What the answer sounds like at each level
Clear names and descriptions, typed parameters, and small results so the model doesn't get confused.
Designs task-level tools with enums, limits and actionable errors; measures wrong-tool and bad-argument rates from trajectories; separates read and write tools.
Runs tools as a governed registry with owning teams, schema versions, response-size budgets and per-tool evals, and controls sprawl with on-demand loading.
Follow-ups the interviewer will push
1You have 70 tools across five teams. Selection accuracy dropped from 94% to 81%. What now?
Measure confusion pairs from trajectories, merge near-duplicates, and switch to a two-stage approach: a small always-on set plus on-demand loading (a tool-search tool or per-route tool sets). Re-run the eval per change; don't fix it with prompt text.
2Should a tool return JSON or prose?
Compact structured text with stable ids and only the fields the next step needs. Raw API JSON wastes tokens on fields the model ignores; prose loses the ids it needs for the next call.
3Now a regulated tenant: contracts under legal hold. What changes in the tools?
Authorization moves into the tool: results are filtered by the user's matter-level ACL at query time, mutating tools are disabled under hold, and every call is audit-logged with the user principal — not the agent's service account.
Red flags interviewers listen for
- Proposes exposing every endpoint of an existing API as a tool.
- Fixes wrong tool selection only by adding more instructions to the system prompt.
- No response-size limit on list or search tools.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Interface design | Task-level tools, enums, limits, explicit when-not-to-use, actionable errors. |
| Measurement | Wrong-tool, bad-argument and redundant-call rates per 100 tasks from trajectories. |
| Safety | Idempotent, least-privilege mutating tools separated from reads. |
| Org scope | Tool ownership by system owners, shared registry, sprawl control. |
Sources
- Building effective agents — Anthropic (Schluntz & Zhang), 2024blog
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., 2023paper