Query decomposition
Splitting a multi-part or multi-hop question into sub-questions, retrieving for each, then combining the evidence.
Some questions really contain several questions: 'Which of our three biggest suppliers had a recall last year, and what did their contracts say about liability?' One search can't find all of that. Query decomposition breaks it into smaller questions, searches for each one (sometimes using the answer of one to ask the next), and then gives the model all the evidence together.
Planning a trip by asking separately about flights, then hotels near the airport you landed at, then restaurants near that hotel.
Decomposing everything. On simple questions it triples latency and cost, and the planner sometimes invents sub-questions the user didn't ask. Gate it on intent and measure answer quality per intent, not overall.
Two shapes. Parallel decomposition: independent sub-questions ('revenue of A in 2025', 'revenue of B in 2025') retrieved concurrently and merged — latency ≈ one planner call + slowest retrieval. Sequential (multi-hop): each step depends on the previous answer ('who acquired X?' → 'who is the CEO of that acquirer?'), as in Self-Ask (Press et al. 2022) or least-to-most prompting (Zhou et al. 2022) — latency grows per hop, 1–3 s each with an LLM in the loop. Planner output should be structured ([{id, question, depends_on}]) so you can execute a DAG and cap it (≤ 4 sub-questions, ≤ 3 hops). Give each sub-question its own retrieval budget (k=5–8) rather than one shared top-k, otherwise the most 'popular' sub-question takes all the slots. Only decompose when an intent classifier flags the question as multi-part: on single-intent questions decomposition adds 1–2 s and can drift the topic. Full Agentic RAG generalises this into a loop where the model decides when to search again.
Gate on intent (comparison, multi-hop). Small-model planner with JSON schema output, max 4 sub-questions and 3 sequential hops, per-sub-question k=5, parallel execution of independent nodes, overall 6 s timeout with partial-answer fallback.
Financial filings assistant: 'How did gross margin change for the two largest cloud providers between FY2024 and FY2025?' Planner (Haiku-class, 400 ms) emits 4 parallel sub-questions (2 companies × 2 years). Each retrieves top 5 from 10-K chunks (in parallel, 60 ms). Generator gets 20 chunks with sub-question labels. Without decomposition, a single query returned 8 chunks all from one company's FY2025 filing.
Recommend it behind a gate: "I classify first; only comparison and multi-hop intents get decomposed, with a cap of 4 sub-queries and per-sub-query budgets." Expect: how do you stop runaway hops? Hard caps, a dependency DAG instead of free-form loops, and a fallback to answer with what's found plus an explicit gap statement.
Numbers worth knowing
- Added latency, parallel
- 300–800 msone small-model planner call + concurrent retrieval
- Added latency, sequential
- 1–3 s per hopLLM answer needed between hops
- Cap
- ≤ 4 sub-questions, ≤ 3 hopsbeyond this, quality rarely improves and cost grows linearly
How to decide
| When | Do this |
|---|---|
| Comparison across entities or periods | Parallel decomposition with per-entity budgets. |
| Answer to part 1 is needed to ask part 2 | Sequential hops, capped at 3. |
| Single-intent question | Don't decompose; at most a rewrite. |
Cost maths
Planner call: ~600 input + 150 output tokens at assumed $0.25/$1.25 per M (small model, Oct 2026) ≈ $0.00034. Retrieval × 4 is infrastructure. The real cost is generation input: 20 chunks × 500 tokens = 10k tokens at $3/M = $0.03 vs $0.012 for a 8-chunk single query. On 15% of 1M queries/month that's ~$2,700/month extra.
Operating it
- > 25% of traffic routed to decomposition — the classifier is too eager; retune
- Sequential hop count p95 = cap — questions need Agentic RAG or better indexing (e.g. precomputed entity tables)
- Synthesis input > 12k tokens p95 — cut per-sub-question k or summarise sub-answers first
- Share of queries decomposed — alert outside an expected band (e.g. 5–20%)
- Sub-questions per query and hop count p95 — alert at cap
- Answer quality for decomposed vs non-decomposed intents — alert if decomposed underperforms baseline
When it goes wrong
A clinical-guidelines assistant became slow (p95 9 s) and occasionally answered questions nobody asked.
Latency SLO breach plus a rise in 'irrelevant answer' feedback; traces showed 3–6 sub-questions on simple queries.
Decomposition ran on every query; the planner split 'dose of amoxicillin for a 20 kg child' into weight, age and allergy sub-questions and retrieved allergy guidance.
Added an intent gate (only 11% of queries now decompose), capped sub-questions at 3 and required each to be grounded in a span of the user's text.
Related terms
Sources
- Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask) — Press et al., 2022paper
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models — Zhou et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.