What would you do if the correct document exists but consistently ranks too low?
- 1 min — ranking or recall? Check candidate position.
- 2 min — fixes in cost order.
- 2 min — look for the class-level pattern.
- 1 min — guard against boost sprawl.
First confirm it's a ranking problem, not retrieval: if it appears at rank 40 of 50 it's a ranking problem; if it's absent from 200 candidates it's a recall problem. For ranking, the fixes in order of cost: add a reranker; improve the chunk so the answer-bearing text isn't buried; add contextual headers; apply a metadata boost; fine-tune the embedding model on query-document pairs from your own click data.
A common cause is a long chunk where the relevant sentence is one line in two thousand tokens — the embedding is dominated by the rest. Smaller chunks with parent expansion fixes this directly, and it's cheaper than fine-tuning.
Check whether it's systematic. If one document ranks low, fix the document. If a class of documents ranks low — all FAQs, all tables, everything from one source — you have a structural problem in chunking or parsing, and fixing the class is the real work. Staff engineers look for the pattern before fixing the instance.
Org scope: per-document boosts and pinned results become an unowned pile of special cases within a quarter. Allow them only with an owner and an expiry date, and track the count as a health metric.
What the answer sounds like at each level
Add a reranker or boost the document so it ranks higher.
Confirms ranking vs recall from candidate position, then applies fixes in cost order: reranker, chunk restructuring, contextual headers, boosts, fine-tuning.
Buckets failures by document class to find structural causes, fixes the class, and governs boosts so they don't accumulate into unowned special cases.
Follow-ups the interviewer will push
1When is fine-tuning the embedding model worth it?
When you have ≥5–10k real query-document pairs (clicks, accepted citations) and the vocabulary is domain-specific — clinical notes, legal clauses, internal product names. Expect a few to ~10 points of Recall@10 on in-domain queries; the cost is a re-embed of the whole corpus and an owned training pipeline.
2A metadata boost for 'official docs' fixes this case. Risk?
It reorders every query, including ones where community content is the better answer. Ship it behind an A/B or golden-set comparison across all query types, not just the failing one, and give it an owner and expiry.
Red flags interviewers listen for
- Jumps to fine-tuning before checking chunking or adding a reranker.
- Fixes one document by hand without asking whether the class is affected.
- Doesn't check candidate position to distinguish ranking from recall.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Diagnosis | Separates ranking from recall with a concrete check. |
| Prioritisation | Orders fixes by cost and expected gain. |
| Systemic thinking | Groups failures by document class and fixes the structure. |
The FAQ pages nobody could find
Illustrative exampleSupport flagged that our assistant kept missing answers that sat plainly in our FAQ pages, while finding long-form guides reliably. Individually each miss looked like a one-off ranking issue.
I owned ranking quality that quarter and wanted the class-level cause rather than a per-document patch.
I grouped a quarter's worth of low-rank complaints by source document type, and the pattern was immediate — FAQ-format content was over-represented by a wide margin. The cause was our chunker splitting on headings, which separated each question from its answer: the question-chunk matched user queries but contained no answer, and the answer-chunk contained the answer but matched nothing. I changed the chunker to treat a Q&A pair as an atomic unit, backfilled that content type, and added a chunk-level sanity check that flags chunks ending in a question mark with no following body.
Recall on FAQ-sourced questions roughly doubled and the complaint category disappeared from the support queue. The broader change was that we started bucketing quality complaints by document type as standard triage, which surfaced two more structural issues over the following months.
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper
- Passage Re-ranking with BERT — Nogueira & Cho, 2019paper