Agentic RAG
RAG where the LLM decides what to retrieve, from which source, and when to search again, in a bounded loop instead of one fixed retrieval step.
In basic Retrieval-Augmented Generation (RAG), the system searches once and answers. In agentic RAG, the model acts like a researcher: it searches, reads the results, notices what is missing, searches again with a better query, or uses a different tool (a database, an API) before answering.
It handles complex questions that need several pieces of information, at the cost of more time and more model calls.
A paralegal who reads the first search results, notices a gap, and runs another search, instead of answering from whatever came back first.
Making every query agentic. Most questions are answered by one retrieval, and the loop adds latency, cost and new failure modes for no gain.
Retrieval becomes tools (search_contracts(query, filters), run_sql, get_document) inside an The agent loop (ReAct) in the ReAct pattern: plan, call a tool, observe, decide whether the evidence suffices, repeat. Variants: routers that choose an index, Query decomposition into sub-questions, multi-hop retrieval where entities from one result seed the next query, and self-reflective loops (Self-RAG) that grade retrieved passages before using them.
Costs compound. Each step re-sends a growing context, so tokens grow faster than step count; a bad early search derails later steps; and evaluation must cover trajectories (Trajectory evaluation), not just final answers. Production systems bound the loop with Stop conditions & step budgets (max 6–8 tool calls, a token budget, a wall-clock timeout, abstain after N empty searches), keep tools permission-scoped (ACL (access control list), Least privilege), treat tool output as untrusted against Prompt injection (indirect), and log every step for Observability & traces. Anthropic's 'Building effective agents' guidance applies: use the simplest workflow that works, and add autonomy only where it measurably helps.
Router (small classifier or cheap LLM call) sending ~80% of questions to single-shot RAG; agent built on plain tool calling or LangGraph; max 8 tool calls, 60k-token budget, 45 s timeout; abstain after 2 empty searches; permission-scoped tools; every step traced (OpenTelemetry spans); trajectory evals on 100 multi-hop questions.
Legal due diligence: 'Which of our 1,200 vendor contracts auto-renew in Q1 2027 and lack a termination-for-convenience clause?' Step 1: run_sql on contract metadata → 86 contracts renewing in Q1. Step 2: search_contracts for termination-for-convenience language filtered to those 86 → hits in 61. Step 3: get_document on the 25 without hits to confirm absence → 19 confirmed. Answer lists 19 contracts with clause citations after 3 tool rounds, 14 s and ~6× the tokens of a single-shot call.
Commit: single-shot retrieval is the default and the agentic path is the exception, chosen by a router or by low retrieval confidence. Put a number on it: ~6× cost per agentic query, so if 20% of traffic escalates, blended cost roughly doubles. Follow-ups: 'how do you stop runaway loops?' (step, token and time budgets plus abstention), and 'how do you evaluate it?' (trajectory evals: right tools, right order, no redundant calls).
Numbers worth knowing
- Cost multiplier vs single-shot
- 3–10×3–5 tool rounds with context re-sent each round
- Latency
- 5–30 s3–5 rounds on a hosted frontier model
- Step cap
- 6–8 tool callsbeyond this, success rarely improves and cost grows
How to decide
| When | Do this |
|---|---|
| Question is answerable from one passage | Single-shot RAG; don't start a loop. |
| Question needs joins across sources or multi-hop entities | Agentic path with bounded steps. |
| Questions repeat the same multi-step shape | Hard-code it as a workflow (fixed chain), not an open-ended agent. |
Cost maths
Single-shot: 5k in + 400 out at $3 / $15 per M (illustrative) = $0.015 + $0.006 = $0.021. Agentic, 4 rounds with growing context: inputs 5k + 8k + 11k + 14k = 38k → $0.114, outputs 1.5k → $0.0225, total ≈ $0.14 (6.6×). With 20% of traffic escalated: 0.8 × $0.021 + 0.2 × $0.14 = $0.045 blended, 2.1× single-shot. Prompt caching on the shared prefix cuts the agentic input cost by about half.
Operating it
- Single-shot answers abstain or fail on more than 15% of questions that need multiple sources
- Agentic share of traffic above 30% (router too eager)
- Steps per query p95 above 6
- Tool calls per query p95: alert above 6
- Cost per query p99: alert above 10× single-shot
- Abstention rate on the agentic path: alert outside 3–15%
- Trajectory eval score on multi-hop set: alert at −5 points
When it goes wrong
Some research queries ran for 2 minutes and cost ~$3 each; p99 latency on the assistant blew through its SLO.
Cost-per-query p99 alarm and traces showing 40 near-identical search calls in a row.
When no document existed, the agent kept paraphrasing the query; there was no step cap and no rule to stop after empty results.
Max 8 tool calls, a token budget, abstain after 2 empty searches with a 'not found in corpus' answer, and a trajectory eval case for unanswerable questions.
Related terms
Sources
- ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., 2022paper
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Asai et al., 2023paper
- Building effective agents — Anthropic, 2024blog
Check yourself
Three questions. Every option is explained once you answer.