HyDE (hypothetical document embeddings)
Hypothetical Document Embeddings: an LLM writes a fake answer, you embed it and search with it — document-to-document matching instead of question-to-document.
Questions and answers often look quite different, so their embeddings may not be close. HyDE asks an Large Language Model (LLM) to write a short pretend answer to the question first. That pretend answer might contain wrong facts, but it looks like the real documents, so searching with its embedding finds them better. The fake text is only used for searching and is never shown to the user.
Describing the photo you're looking for ('a red barn in snow at sunset') to find similar photos, rather than asking 'show me winter farm pictures'.
Showing or caching the hypothetical document as if it were evidence. It's an unverified generation; if it leaks into the context or a Semantic cache, you've laundered a hallucination into a 'retrieved' source.
Gao et al. (2022) generated hypothetical documents with InstructGPT, embedded them with an unsupervised encoder (Contriever) and averaged the vectors, matching fine-tuned retrievers zero-shot. It works because it converts an asymmetric query–document match into a symmetric document–document match (Symmetric vs asymmetric embeddings). Costs: one generation call (300–1,500 ms, ~150 output tokens) on the critical path, and the generated text carries the model's priors — if the model's guess is wrong in a specific way (wrong product name, wrong statute), it pulls retrieval toward documents about that wrong thing. Modern asymmetric embedders trained with query instructions (e5, bge, Qwen3-Embedding) narrow the gap HyDE closes, so gains on in-domain data with a good model are often 0–3 points. It helps most for short, vague queries in specialised domains and for zero-shot setups with no labelled data. Always keep the original query embedding as a leg (average or fuse with Reciprocal Rank Fusion (RRF)).
Off by default. Enable per intent when offline eval shows ≥ 3 recall@20 points: small model, 1 draft of ≤ 120 tokens, temperature 0.3, 1.5 s timeout falling back to the raw query, RRF-fused with the original-query leg.
Clinical-guidelines search, query 'kid wheezy after cold, what inhaler'. Raw query vs guideline chunks: top-1 cosine 0.52, top hit is a cold-remedy leaflet. HyDE draft (Haiku-class, 600 ms): 'For a child with post-viral wheeze, a short-acting beta-agonist such as salbutamol via spacer is first-line…' Its embedding hits the paediatric asthma guideline section at cosine 0.79. The draft's dose numbers are wrong, which is harmless because the draft is never shown.
Pitch it narrowly: "HyDE is a zero-shot patch for short, vague queries in jargon-heavy domains. With a modern instruction-tuned embedder and some labelled data I'd fine-tune or rewrite instead — it costs an LLM call on the critical path." Expect: what's the failure mode? Confident wrong drafts steer retrieval toward the wrong entity; mitigate by fusing with the original query.
Numbers worth knowing
- Added latency
- 300–1,500 msone ~120–150-token generation on a small hosted model
- Typical gain
- 0–3 recall@20 points in-domain; larger zero-shotmodern instruction-tuned embedder vs Contriever-era baselines
How to decide
| When | Do this |
|---|---|
| No labelled data, short vague queries, specialised domain | Try HyDE fused with the raw query. |
| Labelled pairs available (≥ 5k) | Fine-tune the embedder instead (Fine-tuning an embedding model). |
| Queries name specific entities or codes | Skip HyDE; it may substitute the wrong entity. |
Cost maths
Per query: ~80 input + 120 output tokens on a small model at assumed $0.25/$1.25 per M (Oct 2026) ≈ $0.00017, plus one extra embedding. At 1M queries/month ≈ $170 — the cost that matters is +0.3–1.5 s latency, which makes it a poor fit for search-as-you-type.
Operating it
- Latency budget < 1 s end to end — remove HyDE or move to async prefetch
- Gain disappears after an embedder upgrade — turn it off
- Recall@20 with vs without HyDE on the golden set — alert if gain < 1 point (turn it off)
- HyDE timeout/fallback rate — alert above 3%
- Entity mismatch: draft names an entity absent from the query — alert above 5% of drafts
When it goes wrong
Searches for a niche insurance rider returned documents about a similarly named, more common rider.
Agents' 'wrong document' flags clustered on queries with product names; traces showed HyDE drafts naming the common rider.
The LLM didn't know the niche product and confidently described the popular one; the draft's embedding dragged retrieval there. The original query was not fused.
Fused the raw-query leg with RRF and disabled HyDE when the query contains a catalog product name (dictionary match).
Sources
- Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE) — Gao et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.