How would you detect that your embedding model is the bottleneck?
- 1 min — oracle experiment
- 2 min — BM25 vs dense, Recall@50 vs @5
- 1 min — neighbour inspection and score spread
- 1 min — cheap fix vs real fix
Isolate the stage. Run an oracle experiment: feed the known-correct chunk directly to the LLM. If answers are good, generation is fine and the problem is upstream. Then compare retrievers on the same golden set — if BM25 finds documents your dense retriever misses, and a reranker promotes documents that were ranked low but present, your embeddings are ordering poorly.
The clearest signal: Recall@50 is high but Recall@5 is low. The right documents are in the candidate pool but the embedding can't rank them. That's an embedding-quality problem, and a reranker is the cheap fix while a better embedding model is the real one.
Add qualitative evidence: inspect nearest neighbours for domain terms. If your model places "warrant" (finance) next to "warrant" (legal), it doesn't understand your domain. Also check score distributions — if everything scores 0.78–0.82, the space isn't discriminating and your thresholds are meaningless. Org scope: the oracle and retriever-comparison runs belong in the shared eval harness as one-click diagnostics, so any team can localise a regression without the retrieval team in the loop.
| Signal | Means | Fix |
|---|---|---|
| Oracle answers good | Generation OK | Look upstream |
| High R@50, low R@5 | Bad ranking | Reranker |
| BM25 finds, dense misses | Lexical gap | Hybrid + RRF |
| Scores all 0.78–0.82 | No separation | New/tuned model |
What the answer sounds like at each level
Try a different embedding model and see if results improve.
Isolates stages with an oracle experiment and compares BM25 vs dense recall on a golden set.
Reads the Recall@50 vs Recall@5 gap as a ranking problem, inspects neighbour quality and score distributions, and picks reranker as the cheap fix and a better or fine-tuned embedding as the real one.
Follow-ups the interviewer will push
1Recall@50 is 0.92 and Recall@5 is 0.55. What do you ship this week?
A cross-encoder reranker over the top 50 — typically the biggest nDCG@10 jump per day of work. Start fine-tuning the embedding model on mined hard negatives in parallel.
2BM25 finds 20% of answers dense retrieval misses. What does that tell you?
Exact-term matching matters: part numbers, error codes, statute names. Ship hybrid retrieval with RRF; it is cheaper than chasing those with a new embedding model.
Red flags interviewers listen for
- Swaps models without isolating the failing stage.
- Uses only end-to-end answer quality as the signal.
- Treats absolute cosine thresholds as meaningful without checking score distributions.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Isolation | Localises the failing stage before acting. |
| Metrics | Reads recall at different k as a ranking signal. |
| Prioritisation | Distinguishes the quick reranker fix from the durable embedding fix. |
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper
- The Probabilistic Relevance Framework: BM25 and Beyond — Robertson & Zaragoza, 2009paper