Oracle experiment
Replace one pipeline stage with perfect input (e.g. gold passages) and measure the ceiling. The gap shows which stage to fix first.
When a Retrieval-Augmented Generation (RAG) system gets answers wrong, you need to know whether retrieval missed the evidence or the model fumbled it. An oracle experiment answers that by cheating on purpose: you hand the model the known-correct passages from your Golden set and see how well it does. If accuracy jumps, retrieval is the problem. If it barely moves, the problem is in generation.
A mechanic who swaps in a known-good part to see whether the car runs; if it does, the old part was the fault.
Upgrading the model because answers are wrong. If the oracle score is far above production, a better model on the same retrieval buys little.
Run the same golden set under controlled substitutions: (a) closed-book, no context, measuring Parametric knowledge; (b) production retrieval; (c) oracle context: gold passages only; (d) oracle + distractors: gold passages mixed with production noise at production length; optionally (e) oracle with the reranked order, or an oracle rewritten query. Gaps localise error: (c) − (b) is the retrieval gap; 100% − (c) is the generation gap (prompt, model, reasoning); (c) − (d) is the distractor cost, which shows whether Context assembly or Lost in the middle effects matter. Liu et al. (2023) used closed-book and oracle baselines in exactly this way. The method generalises to agents: give the planner the correct tool, or the correct tool result, and see where success jumps. It requires labelled gold evidence, which is the main cost.
Four-arm run on the golden set every quarter and before any major investment: closed-book, production, oracle, oracle + distractors; same model, prompt and judge across arms; report gaps with paired bootstrap CIs.
Legal research assistant, 250 golden questions with gold passages. Closed-book 41%, production retrieval 68%, oracle 89%, oracle + production distractors 83%. Retrieval gap = 21 points, generation gap = 11, distractor cost = 6. The team had planned a model upgrade (worth at most ~11 points); they spent the quarter on statute-citation-aware retrieval instead, closing the retrieval gap to 8 points.
Say: before choosing what to fix in a RAG system, I run closed-book, production, oracle and oracle-plus-distractors on the golden set; the gaps tell me whether to invest in retrieval, assembly or the model, and give leadership a ceiling for each investment. Expect: 'what if you don't have gold passages?' Label 100 questions' evidence first; it is a few days of work and it decides a quarter of roadmap.
| Gap | Computed as | Value | Owner |
|---|---|---|---|
| Retrieval | oracle − production | 21 | Search team |
| Distractor | oracle − oracle+noise | 6 | RAG app team |
| Generation | 100 − oracle | 11 | Prompt / model |
Numbers worth knowing
- Gold-evidence labelling
- 4–8 min per questiondomain-literate labeller marking supporting passages
- Minimum useful set
- ~100–150 questionsgaps > 8 points detectable with paired tests
- Run cost
- 4× a normal eval runfour arms, same items
How to decide
| When | Do this |
|---|---|
| Retrieval gap > generation gap | Invest in retrieval: hybrid, rewriting, chunking, reranker |
| Oracle score itself is low (< 80%) | Model, prompt or task framing problem; consider decomposition or a stronger model |
| Distractor cost > 5 points | Tighten context assembly: fewer chunks, better ordering |
Cost maths
Labelling 150 questions × 6 min = 15 hours ≈ $900. Four arms × 150 items × ~$0.02 = $12 per run. Against one quarter of a three-person team (~$150k loaded) pointed at the wrong component, the experiment is the cheapest decision tool available.
Operating it
- Roadmap debate between 'better model' and 'better retrieval'
- Production accuracy plateaued for two releases
- New model generation available: re-run the oracle arm to see the new ceiling
- Quarterly four-arm results; track each gap as a trend
- Retrieval gap; alert if it grows > 5 points between runs (corpus or index drift)
When it goes wrong
A team spent six weeks and a 3× cost increase migrating to a larger model; accuracy rose 2 points
Post-launch review asked why the gain was small; a retroactive oracle run explained it
The oracle arm scored 91% with the old model; production was 66%. Retrieval was missing evidence in most failures, which no model can fix
Reverted to the cheaper model, invested in retrieval, and made a four-arm oracle run a required input to model-upgrade proposals
Related terms
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
Check yourself
Three questions. Every option is explained once you answer.