How would you monitor a RAG system after deployment?
- 2 min — three tiers: system, quality, drift.
- 3 min — traces and what they must contain.
- 3 min — LLM judge: calibration, sample size, cost.
- 2 min — alerts, owners and the weekly low-confidence review.
Three tiers.
- System: per-stage latency (p50/p95/p99), error rates, ingestion lag, index size, token spend per query, cache hit rates.
- Quality: continuous eval on a golden set per release; online proxies — abstention rate, citation click-through, thumbs up/down, conversation abandonment, escalation to human, retry rate.
- Drift: query distribution shift, score distribution shift, share of queries with no high-confidence result, new-term rate.
The single most useful signal is the low-confidence query log: queries where retrieval scored poorly or the system abstained. It's simultaneously a quality alarm, a content gap report, and a product roadmap. Review it weekly with the content team.
Make traces fully reconstructable — store query, rewrite, candidates with scores, final context, prompt, response, model version, index version. Without that you cannot debug a complaint from last Tuesday. Then use an LLM-as-judge on a sample of live traffic for faithfulness and relevance, but calibrate the judge against human labels first and re-calibrate when you change judge models, or you're monitoring a number that drifts on its own.
Org scope: buy the tracing backend (OpenTelemetry into Langfuse, Arize Phoenix, Datadog or similar) and build the quality layer — golden sets, judge prompts and calibration are domain-specific. The platform owns dashboards and alert routing; each product team owns its golden set and the weekly low-confidence review.
Assumptions: 1M queries/day; trace ≈ 30 KB (4k-token context ≈ 16 KB + prompt, response, scores); judge input 4,000 tokens, output 200, at illustrative $3/M input and $15/M output (Oct 2026).
- Trace storage: 30 GB/day; 90-day retention ≈ 2.7 TB → ~$62/month in object storage at $0.023/GB-month (hot query index in an observability vendor costs more).
- Judge sample size to detect a 2-point faithfulness drop from 90% (α = 0.05, power 0.8, two daily windows): n ≈ (1.96 + 0.84)² × 2 × 0.9 × 0.1 / 0.02² ≈ 3,500/day (0.35% of traffic).
- Judge cost: 3,500 × (4,000 × $3/M + 200 × $15/M) = 3,500 × $0.015 ≈ $53/day ≈ $1.6k/month. Judging 1% instead would be ~$4.5k/month for no extra detection power at this effect size.
- Alerts: golden-set faithfulness −2 points vs last release; abstention rate +50% over 24h; retrieval p95 > SLO for 10 min; ingestion lag > 1h.
What the answer sounds like at each level
Track latency, errors and user feedback, and run the eval set before releases.
Monitors system, quality and drift tiers, keeps reconstructable traces, and runs a calibrated LLM judge on sampled live traffic.
Sizes the judge sample statistically, sets alert thresholds tied to SLOs, decides buy (tracing) vs build (quality layer), and assigns the low-confidence review to named owners.
Follow-ups the interviewer will push
1Traffic grows 10×. Does the judge bill grow 10×?
No — the sample size you need to detect a given regression depends on the effect size, not on traffic. Keep a fixed ~3,500 judged samples/day (see estimate) and let the sampling rate fall from 0.35% to 0.035%.
2Traces contain regulated patient data. What changes?
Redact PII before traces leave the request path, store raw traces in-region with access logging and a short retention (e.g. 30 days), and run the judge on redacted text or inside the compliant boundary. The judge provider becomes a data processor under contract.
Red flags interviewers listen for
- Monitors only latency and errors, not answer quality.
- Trusts an uncalibrated LLM judge.
- Traces too thin to reconstruct a past answer.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Coverage | System, quality and drift signals with concrete metrics. |
| Debuggability | Fully reconstructable traces with model and index versions. |
| Rigour | Calibrated judge with a statistically sized sample and thresholds. |
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Site Reliability Engineering, ch. 6: Monitoring Distributed Systems — Google, 2016book