How would you troubleshoot irrelevant documents being retrieved?
- 1 min — split the symptom: ranking problem vs coverage problem.
- 3 min — walk the pipeline: parse → chunk → embed → filter → rank → k.
- 1 min — the debug endpoint and what it returns.
- 1 min — lock fixes into the golden set.
Work backwards through the pipeline, and don't guess. In order: Is the correct document in the index at all, parsed correctly? Is it chunked such that a chunk contains the answer? Does the query embedding land near it (check with the raw text as the query)? Are filters excluding it? Is it in the candidate pool but ranked low? Is top-k too large, letting weak results in?
Build a debug endpoint that returns, for one query: the rewritten query, applied filters, raw candidates from each retriever with scores, post-fusion ranks, post-rerank ranks, and the final context. Ninety percent of these investigations end in the first five minutes once you can see that.
Distinguish "irrelevant results appear" from "relevant results missing" — they're different bugs. Irrelevant results that rank above relevant ones is a ranking problem. Irrelevant results filling the list because nothing relevant exists is a coverage problem, and the fix is content or abstention, not retrieval tuning.
Org scope: the debug endpoint belongs to the retrieval platform, not each product team, and it must run with the caller's tenant and ACL scope — a debug tool with god-mode access is itself a leak path. Coverage gaps route to the content owners, not to engineering.
What the answer sounds like at each level
Checks the document is indexed, inspects the chunks, and looks at similarity scores for the query.
Walks the pipeline stage by stage with a per-query debug trace and separates ranking bugs from coverage gaps.
Turns one-off debugging into a platform capability (tenant-scoped debug endpoint, labelled failure buckets) and routes coverage gaps to content owners with a weekly report.
Follow-ups the interviewer will push
1The correct chunk is in the index but never in the top 200. Next step?
Embed the chunk's own text as the query: if it doesn't retrieve itself at rank 1, the index or embedding is broken (wrong model version, normalisation mismatch, ANN recall). If it does, the user's phrasing is far from the text — try hybrid, query rewriting or HyDE.
2You've fixed 20 queries by hand. How do you make sure they stay fixed?
Add them to the golden set with the expected document ids and run it as an eval gate in CI, so any chunking, model or index change that regresses them blocks the merge.
Red flags interviewers listen for
- Starts by changing the embedding model before looking at a single trace.
- Treats 'irrelevant results' and 'missing results' as the same bug.
- Proposes a debug tool that bypasses tenant isolation.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Method | Systematic stage-by-stage elimination rather than guessing. |
| Tooling | Describes a per-query trace with per-stage candidates and scores. |
| Root cause | Distinguishes ranking, recall and coverage failures and fixes each differently. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper