How would you handle questions that cannot be answered from the knowledge base?
- 1 min — four layers of defence
- 2 min — threshold sweep on answerable vs unanswerable
- 1 min — the business trade-off and who decides
- 1 min — abstention log as content roadmap
Layered defence. (1) Relevance gate: if the top reranker score is below a calibrated threshold, don't call the LLM at all. (2) Prompt-level refusal instruction with a fixed "I don't have information on that" form. (3) Post-generation groundedness check — verify each claim is supported by context. (4) Route: hand off to a human, to search, or to a different tool.
A support bot that says "I don't have that documented — connecting you to an agent" is worth far more than one that invents an answer. Set the threshold by sweeping it against a labelled set of answerable and unanswerable questions, and pick the point on the precision/coverage curve the business wants.
Name the business trade-off explicitly: abstention rate versus hallucination rate. These move in opposite directions and the right point is a product decision, not an engineering one. Also log every abstention — that log is your content roadmap. Org scope: the threshold is owned jointly — product picks the operating point, the ML team owns calibration and re-tunes it on every model or embedding change, support owns the handoff queue.
What the answer sounds like at each level
Tell the model to say 'I don't know' when the answer isn't in the documents.
Builds layered defence: relevance gate, refusal prompt, groundedness check, routing, with a threshold tuned on labelled data.
Frames abstention vs hallucination as a product decision with an explicit operating point, and turns the abstention log into a content roadmap.
Follow-ups the interviewer will push
1Now you swap the reranker. What happens to the threshold?
It is invalid. Reranker scores are not calibrated across models, so a 0.3 on one is not a 0.3 on another. Re-sweep on the labelled answerable/unanswerable set before rollout, or the refusal rate can double or halve overnight.
2Now it's a regulated tenant where a wrong answer costs more than a refusal.
Move the operating point toward precision, e.g. accept 25% abstention to get hallucination under 1%, require a citation on every claim, and route abstentions to a human queue with an SLA.
Red flags interviewers listen for
- Relies only on a prompt instruction to refuse.
- Picks a similarity threshold by eye with no labelled set.
- Treats abstention rate as purely an engineering metric.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Defence in depth | Gates before, during and after generation. |
| Calibration | Sets the threshold from a labelled sweep and re-tunes on model change. |
| Product sense | Hands the operating point to the business with the curve. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper