Query rewriting
Turning the user's raw message into one or more better search queries: resolving pronouns, expanding jargon, stripping chit-chat.
Users type messy questions, often in the middle of a conversation: 'what about for contractors?' means nothing to a search engine on its own. Query rewriting uses rules or an Large Language Model (LLM) to turn that into a complete, searchable query — 'parental leave policy for contractors' — before retrieval runs. It can also expand abbreviations or produce a few variants of the query.
A good reference librarian who listens to your rambling request and writes down the precise thing to look up.
Letting the rewriter 'improve' precise queries. An LLM will happily rewrite ORA-01555 into 'Oracle database snapshot error', losing the exact token BM25 needed. Skip rewriting for identifier-like queries or always fuse with the original.
Three common rewrites. Contextualisation (standalone question from chat history) is the highest-value one: in multi-turn chat, 20–40% of follow-ups contain pronouns or ellipsis that break retrieval. Expansion (synonyms, acronyms, domain terms) helps BM25 most. Multi-query generates 3–5 paraphrases, retrieves for each and fuses with Reciprocal Rank Fusion (RRF) (often called RAG-Fusion) — +2–6 recall points at 3–5× retrieval cost. Use a small fast model (Haiku-class, GPT-4o-mini-class, or a fine-tuned 1–3B model) with Structured output / decoding to emit JSON {queries: [...], filters: {...}}, and pass the original query as one of the legs so a bad rewrite can't remove a good match. Ma et al. (2023) trained a small rewriter with RL from downstream answer quality. Latency: 150–500 ms for a small hosted model, < 50 ms for a self-hosted 1B — usually the largest added cost in the retrieval path.
Standalone-question rewrite for every turn after the first, using a small model with JSON output and a 2-second timeout falling back to the raw query. Original query always fused as one leg. Multi-query (3 variants) only for intents where it measurably helps.
Internal IT helpdesk chat. Turn 1: 'How do I get VPN access on my new laptop?' Turn 2: 'and on my phone?' Raw retrieval on turn 2 returns phone-reimbursement policy. Rewriter output: {"queries": ["VPN access setup on mobile phone", "GlobalProtect iOS Android enrollment"]} → top hit is the mobile VPN enrollment page. Cost: one 300-token call to a small model, ~200 ms.
Recommend it conditionally: "In chat, a standalone-question rewrite is mandatory; multi-query fan-out only when I measure a recall gain worth 3× the retrieval cost." Always keep the original query as a leg. Expect the follow-up: how do you evaluate a rewriter? Compare retrieval recall@20 on the golden set with raw vs rewritten queries, segmented by first-turn vs follow-up.
Numbers worth knowing
- Follow-ups needing contextualisation
- 20–40% of turnsmulti-turn assistant traffic (typical sampled logs)
- Added latency
- 150–500 ms hosted, < 50 ms self-hosted 1B~300 input / 60 output tokens
- Multi-query recall gain
- +2–6 recall@20 points3–5 variants fused with RRF; costs 3–5× retrieval
How to decide
| When | Do this |
|---|---|
| Multi-turn chat | Always produce a standalone query. |
| Query looks like an identifier, code or exact title | Skip rewriting; search verbatim. |
| Recall on paraphrase-heavy queries is low and latency budget allows | Multi-query with RRF, original included. |
Cost maths
Per query: ~300 input + 60 output tokens. Assume a small model at $0.25/M input and $1.25/M output (illustrative Oct 2026): 300 × 0.25e-6 + 60 × 1.25e-6 = $0.00015 → $150 per million queries. With Prompt caching of the fixed instructions (~250 tokens, reads at 0.1×) input cost drops ~70%.
Operating it
- Rewriter p95 > 400 ms — self-host a fine-tuned 1–3B rewriter
- Rewrite changes identifier tokens in > 1% of queries — add a verbatim-token guard
- Follow-up recall still < first-turn recall by > 10 points — add history summarisation
- Recall@20 raw vs rewritten on the golden set, by turn position — alert if rewritten < raw
- Rewriter timeout / fallback rate — alert above 2%
- Identifier tokens dropped by rewrite — alert above 0.5% of queries
When it goes wrong
Searches for specific legal citations ('§ 1983 claims') started returning general civil-rights articles.
Exact-citation slice of the golden set dropped from 0.9 to 0.55 recall@10 after the rewriter launch.
The rewriter normalised '§ 1983' into 'federal civil rights lawsuits', and the original query was not kept as a leg.
Added the original query as a fused leg, a regex guard that preserves citations and codes verbatim, and the citation slice as a release gate.
Related terms
Where it shows up
Sources
- Query Rewriting for Retrieval-Augmented Large Language Models — Ma et al., 2023paper
- Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods — Cormack et al., 2009paper
Check yourself
Three questions. Every option is explained once you answer.