Walk me through how tool calling works end to end.
- 3 min — the 5-step protocol with stop reason and call ids
- 2 min — worked example with token growth and latency per turn
- 3 min — execution boundary: validation, authz, untrusted results
- 2 min — caching and parallel calls
- 2 min — shared runtime ownership
1) The request carries the conversation plus tool definitions — name, description and a JSON Schema for inputs (Anthropic: tools[] with input_schema; OpenAI: tools[] with function.parameters). Definitions are tokens: 20 tools at ~150 tokens each is ~3k tokens on every turn. 2) The model emits a tool-call block (tool_use with an id, name and arguments) and the API returns with stop_reason: "tool_use". 3) Your code validates the arguments against the schema, authorises the call as the end user, and executes it. 4) You append a tool_result tied to the call id and call the model again with the whole transcript. 5) It calls another tool or answers. Multiple independent calls can arrive in one turn (parallel tool use); run them concurrently and return all results in one message.
The model never touches your systems. It proposes; your runtime disposes.
Clinical intake scheduling: “book my follow-up with the cardiologist who saw me in March”. Turn 1: model calls search_encounters(patient_id, specialty="cardiology", since="2026-03-01") → runtime returns {provider_id: "P-4471", date: "2026-03-12"}. Turn 2: get_availability(provider_id="P-4471", window_days=14) → 6 slots. Turn 3: the model asks the patient to choose rather than booking, because book_slot is marked as requiring confirmation. Three round trips, each re-sending a transcript that grows from ~4k to ~6k tokens; at p95 1.8 s per model turn and 400 ms per tool, the patient waits ~7 s.
The execution boundary is the security boundary. Validate arguments with the schema (reject, don't coerce), authorise with the user's identity from the session — never an id the model wrote into the arguments — and treat every tool result as untrusted input that can carry prompt injection. Return errors as short actionable text (“no encounter in cardiology since 2026-03-01; try since=2025-09-01”), not stack traces. Each round trip re-sends the prefix, so put system prompt and tool definitions first and cache them: reads cost 0.1× base input.
Org scope. Tool execution belongs in one shared runtime (an LLM gateway or agent service) owned by the platform team: schema validation, authz, timeouts, audit log and per-tool metrics live there once. Product teams register tools; they do not hand-roll loops. Build vs buy: provider SDK tool runners are fine for prototypes; production needs your own executor because authz and audit are yours. Failure modes: hallucinated tool names (reject with the list of valid names), malformed JSON (≈0 with strict/constrained schemas, otherwise budget a repair retry), and argument values that are schema-valid but semantically wrong.
Per turn: system 1.5k + tools 3k + transcript 0.5k→2.5k tokens. Three turns ≈ 15k input, 0.6k output. assume $3 / M input and $15 / M output tokens (illustrative list prices, Oct 2026); cache writes 1.25× input (5-minute TTL), cache reads 0.1×. Uncached: 15k × $3/M + 0.6k × $15/M ≈ $0.054. With the 4.5k prefix cached (written once, read twice): ≈ $0.033 — about 40% cheaper, and TTFT drops because the prefix isn't re-processed.
What the answer sounds like at each level
Model gets tool schemas, returns a tool call, my code runs it and sends the result back, repeat until it answers.
Covers stop reasons, call ids, parallel tool calls, schema validation, user-identity authz and prompt caching of the tool prefix, with token counts per turn.
Centralises execution, authz, audit and per-tool SLOs in a shared runtime, and treats tool definitions as a versioned API with owners and eval coverage.
Follow-ups the interviewer will push
1The model passes patient_id in the arguments. Do you use it?
No. Identity and tenant come from the authenticated session; a model-supplied id is at best a hint that must match the session's ACL, otherwise the call fails closed. Otherwise an injected document can redirect the call to another patient.
2Now 10× traffic and tool definitions are 6k tokens. What do you do?
Cache the prefix (system + tools) — at 0.1× read price the 6k tokens cost like 600. Then cut definitions: drop rarely used tools from the default set and load them on demand, and trim descriptions that eval shows don't change selection accuracy.
3How do you handle a tool that takes 45 seconds?
Don't block the turn. Return a job handle immediately ({status: "running", job_id}), give the model a check_job tool or resume the loop on a callback, and set the tool timeout well below the request timeout so the run degrades gracefully.
Red flags interviewers listen for
- Implies the model executes the function or calls the API itself.
- Trusts identifiers in the model's arguments for authorization.
- Returns raw stack traces to the model as tool results.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Protocol accuracy | Correct message sequence, call ids, stop reason, parallel calls. |
| Security | User-identity authz in the executor; tool results treated as untrusted. |
| Cost and latency | Quantifies per-turn transcript growth and uses prefix caching. |
| Org scope | One shared executor with audit and per-tool metrics; teams register tools. |
Sources
- Tool use with Claude — Anthropic, 2025docs
- Prompt caching — Anthropic, 2025docs
- ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., 2022paper