One line per term, the default to reach for, and the number to remember. Print it or save it as a PDF.
A fixed, versioned set of realistic inputs with known-good answers or relevant documents, used to score every change the same way.
Default: 200–500 items sampled from production logs, stratified by intent, 10–15% unanswerable; two labellers on 20%+ of items with κ reported; JSONL in git with a schema and a version tag; 70/30 dev/holdout split; quarterly refresh from error-analysis clusters.
95% CI half-width: ±5.5 / ±3.5 / ±2.5 points (n = 200 / 500 / 1,000 at p ≈ 0.8)
The code that runs a system against a golden set, scores outputs with metrics and judges, and reports diffs, ideally on every pull request.
Default: promptfoo or a thin in-house runner around the production service; pinned model snapshot ids for system and judge; response cache keyed by (stage, input hash, config hash); PR tier of 50–100 stratified items in < 5 min; nightly full run; results stored per run for trend charts.
PR-tier runtime: < 5 min (50–100 items, cached retrieval, parallelism 16–32)
Of all the relevant documents for a query, the fraction that appear in the top k results. The ceiling on what a RAG system can answer.
Default: Report recall@100 (first stage) and recall@k_final (what the prompt sees) on the golden set, per slice.
First-stage target: recall@100 ≥ 0.90 (before investing in reranking or prompts)
Of the results you returned, the fraction that are relevant. Precision@k measures how much of the top k is useful versus noise.
Default: For retrieval: report precision@k at the final k alongside recall@k and nDCG@10.
Final-context precision target: ≥ 0.6 (5–8 chunks, single-hop Q&A)
Normalised Discounted Cumulative Gain: graded-relevance ranking score that rewards the best results near the top. 1.0 = ideal order.
Default: nDCG@10 with graded labels (0–3), linear gain unless you have a reason for exponential, computed with ranx or pytrec_eval; per-slice reporting; paired bootstrap (10k resamples) for system comparisons.
BEIR average nDCG@10: ~0.42 BM25, ~0.50–0.55 strong dense + rerank (zero-shot across BEIR datasets; approximate, varies by subset)
Mean Reciprocal Rank: average of 1/rank of the first relevant result per query. Answers 'how far down is the first good hit?'
Default: MRR@10 for single-target retrieval (FAQ match, entity lookup, code definition), reported with hit@1, hit@10 and miss rate, per slice, with a paired bootstrap CI when comparing systems.
MS MARCO dev MRR@10: ~0.18 BM25, ~0.39–0.40 BERT-class cross-encoder (passage ranking; approximate published figures)
Share of an answer's claims that the provided context supports. Measures grounding, not truth: a faithful answer from a wrong doc is still wrong.
Default: Ragas faithfulness or an in-house claim-split + judge with a pinned model; claims checked against cited spans when citations exist; 150 human-labelled claims for calibration; report the mean and the share of answers below 0.8; slice by claim type (numbers, dates, entities, negations).
Production gate: ≥ 0.9 mean, < 5% of answers below 0.8 (customer-facing RAG)
RAG metric: are the relevant chunks ranked at the top of the context? Rank-weighted precision, so noise above the evidence costs more.
Default: Ragas context_precision (reference-based when the golden set has reference answers), computed on the assembled context, not the first-stage list; reported with context recall and faithfulness as a triad, per slice.
Healthy range: ≥ 0.75 (5–8 chunk contexts after a cross-encoder reranker)
Latency at a percentile: p50 is the typical request, p95/p99 the slow tail. SLOs use the tail because averages hide the users who wait.
Default: OpenTelemetry spans per stage with exponential histograms; dashboards show p50, p95, p99 for end-to-end, TTFT and each stage; SLO on p95 for interactive endpoints; per-intent max_tokens; timeouts at 2–3× the stage p99 with fallbacks.
Fan-out tail probability: 1 − 0.99^n (n parallel calls, each with 1% chance of p99 latency; 63% at n = 100)
Queries per second: request rate a system handles. For LLMs, convert QPS to concurrency and tokens/s; that is what sizes GPUs and bills.
Default: Capacity sheet per endpoint: daily volume, peak-hour share, LLM calls per request, input and output tokens per call, end-to-end latency → peak QPS, concurrency, TPM, $/day.
Peak-to-average ratio: 2–4× (consumer and business-hours traffic; batch jobs excluded)
Service Level Objective: a target for a measured indicator over a window, e.g. 99.5% of answers under 5 s over 30 days. Its gap is the error budget.
Default: Per user-facing endpoint: availability SLO 99.5%, latency SLO on p95 total and TTFT, one sampled quality SLI; 28- or 30-day rolling window; multi-window burn-rate alerts (14.4×/1 h + 5 min page, 6×/6 h + 30 min page, 1×/3 d ticket); written error-budget policy agreed with product.
99.5% monthly budget: 3.6 hours (30-day window, full-outage equivalent)
Replace one pipeline stage with perfect input (e.g. gold passages) and measure the ceiling. The gap shows which stage to fix first.
Default: Four-arm run on the golden set every quarter and before any major investment: closed-book, production, oracle, oracle + distractors; same model, prompt and judge across arms; report gaps with paired bootstrap CIs.
Gold-evidence labelling: 4–8 min per question (domain-literate labeller marking supporting passages)
Reading real failures, labelling why each happened, and counting the categories, so you fix the biggest cause instead of guessing.
Default: Weekly 60–90 minute session: sample 50–100 failures (thumbs-down, judge failures, golden misses) from a trace viewer (Langfuse, LangSmith, Braintrust, Phoenix); one-line note per trace on the first error; cluster into ≤ 12 categories; spreadsheet with counts, owner and a linked eval check per category.
Sample size: 50–100 traces (enough to see categories ≥ 5% reliably)
Production behaviour shifting from what you evaluated: user inputs, corpus, provider model, or what 'correct' means changes under you.
Default: Pin dated model snapshots; daily 100-item regression run; weekly full golden set with per-slice trends; dashboards for intent mix (PSI), mean top reranker score, abstention rate, answer length and thumbs-down; index freshness and canary-query probes every 5–15 min; golden items carry an as_of date and are reviewed quarterly.
PSI alarm: > 0.2 (intent or length distribution vs 30-day baseline; 0.1–0.2 = watch)
Decide which number a change should move, build the cases that measure it, then change the prompt, model or retriever and read the delta with error bars.
Default: A golden-set of 100–300 cases in version control (JSONL), sliced by intent; graders in order of cost: schema/regex/code checks, then a binary LLM judge validated to kappa ≥0.6 on 100 human-labelled cases.
95% CI half-width, 80% pass rate: ±12.4 pts at n=40; ±5.5 at n=200; ±2.5 at n=1,000 (normal approximation, independent cases)
Offline eval scores a fixed test set before release; online eval measures real users after it. Offline is the gate, online is the verdict.
Default: Offline: 200–500 case golden set, paired runs, per-slice CIs, in CI.
Explicit feedback rate: 0.5–3% of turns (thumbs buttons in chat UIs; skewed toward negative experiences)
A written scoring guide: one observable criterion per line, a small scale (usually pass/fail), and an anchored example for each score.
Default: 3–6 binary criteria per task, each with one real passing and one real failing anchor; stored in git next to the judge prompt with a version id; judge returns JSON {criterion: {critique, pass}}; re-validated on 100 human-labelled cases after every edit.
Criteria per rubric: 3–6 (beyond ~6, split into separate judges or the judge starts conflating them)
Grading by asking which of two answers is better, not scoring each alone; more reliable for subjective quality, but needs order swaps to cancel position bias.
Default: Judge from a third model family, both orders, ties allowed, explicit 'length is not quality' instruction, 300–500 prompts, report win rate excluding ties with a Wilson CI, plus a verbosity check by length bucket.
Strong judge agreement with human preference: >80% (GPT-4 on MT-Bench pairs, Zheng et al. 2023; similar to human–human agreement)
People labelling model outputs against a rubric; the ground truth that LLM judges are calibrated against, and the most expensive data in the eval stack.
Default: Engineers label the first 100 traces themselves (fastest way to learn the failure modes).
Labelling throughput: 20–60 items/hour (binary criteria; ~25/h for expert review of long clinical or legal outputs, 60/h for short answers)
How often independent graders agree beyond chance, usually Cohen's kappa; below ~0.6 your labels, and any judge validated against them, are unreliable.
Default: Cohen's kappa for two graders (sklearn.metrics.cohen_kappa_score), Krippendorff's alpha for 3+ or missing labels, reported with a bootstrap 95% CI on ≥100 items, alongside judge recall on the fail class.
Minimum kappa to trust labels: ≥0.6 (substantial from 0.61) (Landis & Koch bands; ≥0.7 if the judge gates releases)
A public, fixed test set and scoring rule (MMLU, GPQA, SWE-bench) for comparing models; a coarse filter for model choice, never a substitute for your own eval.
Default: Shortlist with task-matched public results (independent re-runs where available), then decide on your own 200–500 case eval run through the same harness for every candidate.
GPQA Diamond size: 198 questions → ±6.4 pts at 70% (95% normal-approximation CI)
When test items (or close paraphrases) leaked into a model's training data, so its score measures memory instead of skill.
Default: A private held-out slice (20–30% of the golden set) that never goes into prompts, fine-tuning or synthetic-data seeds; a canary GUID in every eval file; MinHash dedupe (datasketch, Jaccard ≥0.8) of any training or few-shot data against eval data; time-split cases created after the candidate model's cutoff.
Score drop on a fresh parallel maths set: up to 8 points (GSM8K → GSM1k, some model families, Zhang et al. 2024)
A CI check that runs the eval suite on every prompt, model or config change and blocks the merge or rollout if quality drops beyond the measured noise.
Default: GitHub Actions or GitLab CI job running promptfoo, Inspect or a pytest harness; 100-case smoke slice per PR, full 300–500 cases nightly and before release; 3 samples per case for both baseline and candidate; paired bootstrap or McNemar on the delta; block if CI lower bound < −2 points or any safety/PII deterministic check regresses; cache baseline results keyed by (prompt version, model snapshot, retriever config, judge version).
Paired vs unpaired 95% CI on a delta: ±3.6 vs ±6.4 pts (300 cases, 80% pass rate, 10% discordant pairs)
Randomly split live users between the current and new version, then compare a pre-chosen outcome metric with a significance test to decide what ships.
Default: User-level randomisation through a feature-flag service (GrowthBook, Statsig, LaunchDarkly + warehouse), 50/50 split, fixed 14-day horizon, one decision metric plus latency, cost and complaint guardrails, SRM check daily, CUPED where pre-period data exists.
Users per arm for a 1-pt change on a 10% rate: ≈ 14,400 (80% power, α=0.05, two-sided)
Whether an eval or A/B delta is bigger than the noise; report confidence intervals, use paired tests on shared items, and size n before you run.
Default: Report every eval score as rate ± 95% CI (Wilson for rates near 0 or 1).
95% CI half-width at 80% pass: ±12.4 (n=40), ±7.8 (n=100), ±5.5 (n=200), ±2.5 (n=1,000) (normal approximation, independent cases)
Model-generated inputs, labels or conversations that fill eval or training gaps; fast to make, trustworthy only after filtering, dedupe and human spot checks.
Default: Generator: a frontier model different from the system under test; explicit axis grid; seeds from real documents or de-identified traces; MinHash (datasketch) plus embedding-similarity dedupe; validator pass for answerability; 10% human review; tag every case source: synthetic and report its slice separately.
Keep rate after filtering: 30–60% (dedupe + validator + human sample; above 90% usually means filters are too loose)
The loop that turns production traces and feedback into labelled failures, new eval cases and fixes, so each week of traffic makes the system measurably better.
Default: Tracing with OpenTelemetry GenAI conventions into Langfuse, Arize Phoenix, Braintrust or LangSmith; PII redaction at capture; a saved query for thumbs-down, escalations and low judge scores; a 1–2% random sample; a 60–90 minute weekly review with a named owner; a failure taxonomy in a shared sheet; new cases committed to the golden set within the same week.
Traces read per weekly session: 50–100 (two engineers plus one domain owner, 60–90 minutes)
Deliberately attacking your own AI system (jailbreaks, prompt injection, data extraction, unsafe tool use) to find failures before attackers or users do.
Default: Written threat model mapped to the OWASP LLM Top 10; 1–2 day manual red-team before launch with domain experts; garak or PyRIT plus promptfoo red-team runs in CI nightly; ASR per category judged by a validated classifier (e.g., Llama Guard 4 12B or a tuned judge); successful attacks added to a zero-tolerance safety slice in the eval gate.
Public human red-team dataset: ~39k attacks (Ganguli et al. 2022 (Anthropic))
An input crafted to make a model ignore its safety training or system instructions: role-play framing, encoded text, many-shot examples, adversarial suffixes.
Default: Provider moderation or Llama Guard 4 (12B) on inputs and outputs for open-ended chat; Prompt Guard 86M as a cheap pre-filter; no credentials or confidential rules in prompts; regex and entropy scanners for secrets on outputs; jailbreak suite (garak probes + public datasets + your own finds) in the nightly red-team run.
Many-shot jailbreak scaling: effectiveness rises with shot count, tested up to 256 shots (Anil et al. / Anthropic 2024)
Detecting and masking personal data (names, emails, IDs, health details) before it reaches a model, a log or a vendor; recall matters more than precision.
Default: Microsoft Presidio (regex + checksum recognisers, spaCy en_core_web_lg or a transformer NER) at the gateway and at trace capture; per-region recognisers for national IDs and phone formats; deterministic per-conversation placeholders with a short-lived encrypted vault for re-insertion; irreversible masking in logs; 30–90 day log retention; per-entity recall measured on 500 labelled samples from your traffic.
Recall target, structured IDs: ≥0.99 (email, phone, card (Luhn), IBAN via regex + checksum)
A proxy between your apps and model providers that centralises auth, routing, fallback, rate limits, budgets, caching, redaction and logging.
Default: LiteLLM proxy (or Portkey / Envoy AI Gateway) as a stateless deployment, ≥2 replicas per region, Redis for rate-limit and budget counters, Postgres for keys and spend; model aliases per use case; virtual keys per team with monthly budgets and per-minute token limits; per-route timeouts; streaming passthrough; OpenTelemetry export with team, alias, model, prompt version and cost on every span.
Added latency budget: ≤10 ms p99 in-region (excluding redaction; measure with streaming on)
Capping requests and tokens per key, tenant or user per time window, usually with token buckets, so one caller cannot exhaust shared provider quota or budget.
Default: Token buckets in Redis (atomic Lua) at the gateway, keyed per tenant and per user, on both requests and estimated tokens; tenant ceilings summing to ≤80% of provider limits; separate queue for batch and eval traffic via the provider Batch API; honour retry-after, else exponential backoff with full jitter, max 3 retries; per-agent-task token and step budgets.
Anthropic Start tier, Sonnet-class (Oct 2026): 1,000 RPM, 2M ITPM, 400k OTPM (per model class; higher tiers raise all three)
Routing a request to a backup model or provider when the primary errors, times out or is overloaded; only safe if the backup has passed the same evals.
Default: Fallback chains defined per alias in the gateway (LiteLLM fallbacks, Portkey configs); same model on a second provider first; per-provider circuit breakers; route-specific TTFT timeouts (e.g., 8 s chat, 30 s long documents); retry budget ≤10%; every fallback model × prompt pair in the eval gate; weekly forced-failover drill on 1% of traffic; served-model id logged on every response.
Circuit breaker open threshold: ~20% errors over 30 s, or 5 consecutive failures (per provider × model; half-open probe every 30 s)
Treating prompts as immutable, versioned bundles pinned to a model snapshot and logged on every trace, so any output traces back to exactly what produced it.
Default: Prompt bundle (template, few-shots, tool schemas, output schema, pinned model snapshot, retrieval params) in git or a registry (Langfuse prompt management, Braintrust); content hash as version id; labels production / canary resolved at runtime with a short TTL cache; every change through the eval gate; version hash on every trace span.
Rollback time with label-based release: < 1 minute (label move + runtime cache TTL of 30–60 s)
Total spend to answer one request: input, cache writes and reads, output and reasoning tokens, retrieval, reranking, tool calls and retries.
Default: Gateway logs per call: input, cache-write, cache-read, output and reasoning tokens, model, and computed cost; dashboards of p50/p95 cost per query and per successful task by route; prompt caching on stable prefixes; thinking budgets set per route from evals; max_tokens set; reranker to top 5; Batch API for offline jobs.
Cache multipliers (Anthropic): write 1.25× (5 min) or 2× (1 h); read 0.1× base input (Oct 2026 pricing docs)
A provider retiring a model snapshot on a set date; every dependent prompt, eval and fallback must be migrated and re-validated before requests start failing.
Default: Pinned snapshots in prompt bundles; a deprecation calendar fed by provider deprecation pages and email notices; a gateway query listing usage by snapshot; a standing migration checklist: gate → prompt fix → judge re-validation → shadow → canary → cut-over, finishing ≥3 weeks before retirement; quarterly capacity reserved for migrations; open-weights fallback for routes that cannot tolerate forced changes.
Anthropic notice before retirement: ≥60 days (publicly released models, per Anthropic's deprecation policy (Oct 2026))
Whether stated confidence matches actual accuracy: of answers given 80% confidence, about 80% should be right. Needed for abstention and routing thresholds.
Default: Confidence signal = agreement across 3–5 samples and/or a verifier score (not verbalised confidence); isotonic regression (sklearn IsotonicRegression) fitted on 500–1,000 labelled production examples; reliability diagram and ECE with 10 bins in the eval report; thresholds chosen from the coverage–accuracy curve and the cost ratio of errors to reviews; recalibration in the eval gate on every model or prompt change.
Labelled examples for a stable 10-bin ECE: ~1,000 (500 minimum) (fewer leaves bins with too few points)
Running an operation twice leaves the same state as running it once. It is the property that makes retries and at-least-once queues safe.
Default: Caller-minted Idempotency-Key (UUIDv7) on every mutating API; Postgres table idempotency_keys(account_id, key, request_hash, response, created_at) with a unique index on (account_id, key), written in the same transaction as the side effect; 24 h retention for synchronous APIs, retention ≥ queue retention for consumers.
Stripe idempotency-key retention: 24 hours (keys can be pruned after that; a retry later is treated as new)
Pushing back on producers when a consumer is saturated: bounded queues, 429s and load shedding instead of unbounded buffering until latency or memory collapses.
Default: Gateway token bucket per provider at 90% of the published TPM/RPM limit; bounded queue per dependency (size = Little's-law concurrency × 1.5); max queue wait 10 s for interactive, minutes for batch; 429 + Retry-After on overflow; priority classes interactive > batch > eval; deadlines propagated via request headers.
Latency multiplier at 90% utilisation: 10× service time (M/M/1 approximation; real tails are worse with bursty LLM traffic)
A side queue where messages land after N failed attempts, so one poison message stops blocking the pipeline and can be inspected and replayed later.
Default: SQS redrive policy with maxReceiveCount=4, DLQ retention 14 days (source 4 days), CloudWatch alarm on ApproximateNumberOfMessagesVisible >0 for customer-facing queues; for Kafka, a <topic>.dlq topic with error/attempt/offset headers; error classifier that sends schema/validation errors straight to the DLQ.
maxReceiveCount: 3–5 (with exponential backoff between attempts; higher only for known-transient dependencies)
Wait longer after each failed retry (1 s, 2 s, 4 s…) plus random jitter, so clients don't hammer a struggling service in sync.
Default: Full jitter, base 1 s, cap 30 s, max 4 retries, bounded by the request deadline; honour Retry-After; retry at the SDK layer only (disable UI and gateway retries); circuit breaker opens after 50% errors over 20 requests; retry budget 10% of requests.
Calls from layered retries: 3 × 3 × 3 = 27 (three layers each retrying 3 times for one user action)
Run the new version (green) beside the old (blue), verify it on real infra, then switch all traffic at once, keeping blue warm for instant rollback.
Default: Switch via load-balancer target groups (ALB weighted), Kubernetes Service selector, Argo Rollouts blueGreen strategy, or an Elasticsearch/OpenSearch/Qdrant alias; dual-write during overlap; automated smoke + golden-set check before flip; blue kept warm 24 h for stateless, 7 days for indexes.
Infra during overlap: 2× serving cost (both colours at full capacity; vector indexes also 2× storage)
Writing the same change to two stores, e.g. Postgres and a vector index. Done naively it drifts; the outbox pattern plus CDC makes it reliable.
Default: Postgres transactional outbox; Debezium Postgres connector (pgoutput) → Kafka topic keyed by entity id; idempotent upsert consumer with version check (WHERE incoming_version > stored_version); nightly reconciliation by content hash; outbox rows deleted after 7 days.
Typical CDC end-to-end lag: 1–5 s (Debezium + Kafka at moderate load, excluding embedding time)
Copy real production requests to a new version whose responses users never see, then compare outputs, latency and cost before risking live traffic.
Default: Gateway-level async mirror at 5% (Envoy/Istio mirrorPercentage or app fork), separate API key and concurrency pool, side-effecting tools stubbed with recorded responses, outputs logged with the primary's to a comparison table, nightly pairwise judge run with position randomisation, 7–14 day window.
Sample rate: 1–10% (enough for ≥1,000 paired comparisons per key intent)
Release to a small slice of real traffic first (1%, then 5%, 25%…), compare against the old version, and roll back automatically if metrics degrade.
Default: Flag-driven split by hashed user/tenant id; concurrent control slice of equal size; steps 1/5/25/50/100; gates on 5xx, p95 latency, refusal rate, citation validity and thumbs-down; automated rollback (Argo Rollouts/Flagger analysis or flag guardrails); bake ≥1 h at 1–5%, ≥24 h before 100% for prompt or model changes.
Samples at 1% of 50k req/day: 500/day (detects a 5xx spike; cannot detect a 0.2% quality regression)
A runtime switch that turns code paths, models or prompts on or off per user, tenant or percentage without a redeploy.
Default: LaunchDarkly, Unleash, Flagsmith or Statsig behind the OpenFeature SDK; local evaluation with streamed rules; safe default coded for every flag; separate flags for model id, prompt version and tool enablement; bucketing by tenant id for B2B; 90-day expiry lint on release flags; flag changes logged to the audit trail.
Flag evaluation latency: <1 ms (local) (SDK evaluates cached rules in-process; remote evaluation adds a network hop)
A stable name (e.g. products-read) that points to a physical index, so you can rebuild a new index and repoint readers atomically with no client change.
Default: Read alias + write alias per logical index; physical names encode embedding model, dimension, chunker and version; swaps via Elasticsearch/OpenSearch _aliases or Qdrant change_aliases; config-row emulation for pgvector; old index retained 7 days then deleted by a scheduled job.
Alias swap time: <1 s (single atomic cluster-state update in Elasticsearch/OpenSearch)
Background rewriting of an index's segments to drop deleted and overwritten entries, reclaiming space and restoring search speed and recall.
Default: Leave engine optimizers on (Qdrant deleted_threshold 0.2, Milvus auto-compaction, Elasticsearch tiered merge); per-table autovacuum tuning for pgvector (vacuum_scale_factor 0.02–0.05); heavy force-merge or REINDEX only off-peak and only on cold indexes; rebuild + alias swap when weekly churn >30%.
Qdrant vacuum trigger: 20% deleted (default optimizer `deleted_threshold` per segment)
Being able to answer new questions about production from its telemetry. For LLM systems: traces per request with prompts, tool calls, tokens, cost and quality.
Default: OpenTelemetry SDK with GenAI semantic conventions (pinned version); OTLP to Langfuse, Phoenix, Honeycomb or Datadog; spans for retrieval, rerank, generation, each tool; attributes for model, prompt version, tokens, cost, tenant id; content capture redacted, 100% of errors and negative feedback + 5% sample; content retention 30 days, metrics 13 months.
Content capture rate: 100% errors/negative feedback, 5–10% otherwise (keeps storage and PII exposure bounded)
Time to live: how long a cached or stored item is valid before it expires. Too long serves stale data; too short throws away savings.
Default: Redis with SET key value EX <ttl> and ±10% jitter; cache keys include corpus/prompt version; CDC-driven invalidation by tag; single-flight on misses; DynamoDB TTL with an expires_at filter in reads; retention TTLs enforced in the store (S3 lifecycle rules, index ILM).
Anthropic prompt cache TTL: 5 min default, 1 h option (1-hour writes cost 2× base input vs 1.25× for 5-minute; reads 0.1×)
A scheduled job that compares the source of truth with derived stores (index, cache, ACLs) and repairs drift that events and retries missed.
Default: Nightly job (Temporal workflow or Kubernetes CronJob) comparing registry vs index on (doc_id, content_hash, acl_hash, model_version); 1,024 hash buckets; repairs enqueued to the normal ingestion queue at a capped rate; delete circuit breaker at 1% of corpus; ACL-only sync every 15 minutes for permissioned sources; report emitted as metrics.
Delete circuit breaker: abort if >1% would be deleted (guards against empty or truncated source reads)
When a dependency fails, serve a reduced but useful response (smaller model, cached answer, sources only) instead of an error page.
Default: Ladder defined per feature in config; circuit breaker per provider (open at 50% errors over 20 calls or p95 >2× SLO); fallback provider through an LLM gateway (LiteLLM, Portkey, or in-house); 1–5% of traffic routed to the fallback continuously to keep it warm and evaluated; rung recorded on traces; user-visible degraded banner.
Keep-warm traffic to fallback: 1–5% (proves the fallback works and keeps its eval current)
A blameless written review after an incident: timeline, impact, root causes, what went well, and owned action items with due dates.
Default: Shared template (summary, impact, timeline, contributing factors, went well, actions); triggers written into the on-call policy; draft within 5 business days; review meeting; actions as tickets with P0/P1/P2 and owners; monthly audit of open actions; quality incidents (wrong answers) count as incidents.
Draft deadline: ≤5 business days (memory and logs fade fast)