What would you do if RAG quality suddenly drops after a production deployment?
- 1 min — mitigate: rollback or flag off.
- 2 min — bisect: retrieval vs generation with same-query traces.
- 2 min — non-deploy causes.
- 2 min — prevention: gates, canaries, pinning, index versions, postmortem.
Incident response order: mitigate, then diagnose. Roll back or flag off first if the deploy is the obvious suspect — don't debug in production while customers are affected. Then bisect by layer: did retrieval change (compare candidate sets for the same queries before and after), or did generation change (same context, different answers)?
Check the things that change without a deploy, because "sudden" doesn't always mean "ours": a silently updated hosted model version, an expired API key degrading to a fallback path, an upstream source system changing its export format, a bulk ingest that flooded the index with low-quality content, or a config change nobody counted as a deploy.
Prevention is the real answer and it's what distinguishes the level: eval gates in CI that block a merge on golden-set regression, canary deploys with automated quality comparison, pinned model versions, and index versioning so you can roll back data as well as code. Then say the unglamorous part — write the postmortem, and make the action item a detection improvement, not just a fix. "We found it because a customer told us" is the finding, not the cause.
Org scope: treat prompts, model versions, index builds and bulk ingests as deploys — same change log, same approval, same rollback path — so the on-call engineer can see every change in one place.
What the answer sounds like at each level
Roll back the deploy and look at logs to find what changed.
Mitigates first, bisects retrieval vs generation with before/after traces, and checks non-deploy changes like model versions and bulk ingests.
Prevents recurrence with eval gates, canaries, pinned models and versioned indexes, treats data and prompt changes as deploys, and writes a postmortem whose action item improves detection.
Follow-ups the interviewer will push
1Rollback doesn't fix it. Now what?
The cause isn't in the code you rolled back. Diff everything else in the window: provider model version (check response headers / model id in traces), index version and recent ingests, config and prompt changes, upstream source formats. Replay a golden-set query with Friday's exact context to isolate generation vs retrieval.
2The provider silently updated the model behind your alias. How do you stop that happening again?
Pin dated model snapshots rather than floating aliases, record the model id on every trace, and run the golden set daily so an upstream change is detected within 24 hours even when you didn't change anything. Plan deprecations as scheduled migrations.
Red flags interviewers listen for
- Debugs in production before mitigating.
- Assumes the cause must be the most recent code deploy.
- Postmortem action items are fixes only, with no detection improvement.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Incident order | Mitigates before diagnosing. |
| Diagnosis | Bisects by layer using reconstructable traces and checks non-code changes. |
| Prevention | Eval gates, canaries, version pinning and detection-focused postmortems. |
A quality regression nobody deployed
Illustrative exampleOn a Monday our assistant's answer quality dropped noticeably — vague answers, more abstentions — with no deploy in the preceding four days. I was on call.
Restore quality, find the cause, and close the detection gap that let it run over a weekend unnoticed.
I started from traces rather than hypotheses. Comparing Friday and Monday traces for the same queries, retrieval scores were visibly lower across the board, which pointed at the index rather than the prompt or the model. The ingestion log showed a bulk re-import from one source system over the weekend: they had changed their export to include navigation chrome and footers on every page, so thousands of chunks were now mostly boilerplate, and that boilerplate was crowding candidate pools. I rolled the index back to the Friday snapshot to mitigate within the hour, then added a boilerplate-stripping step and a per-source chunk-quality check — mean content-to-template ratio — that fails ingestion on a sharp drop. For detection, I added a daily automated eval run against the golden set with an alert on regression, which is what should have caught it on Saturday.
Quality was back within about 90 minutes and the root cause was fixed the same day. The daily eval alert has since caught three regressions before any customer noticed, including one from an upstream model change we hadn't been told about. The postmortem also changed how we treat data changes — we started gating bulk ingests the same way we gate code deploys.
Sources
- Site Reliability Engineering, ch. 15: Postmortem Culture — Google, 2016book
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper