Evals & LLMOps

A fixed, versioned set of realistic inputs with known-good answers or relevant documents, used to score every change the same way.

The code that runs a system against a golden set, scores outputs with metrics and judges, and reports diffs, ideally on every pull request.

Of all the relevant documents for a query, the fraction that appear in the top k results. The ceiling on what a RAG system can answer.

Of the results you returned, the fraction that are relevant. Precision@k measures how much of the top k is useful versus noise.

Normalised Discounted Cumulative Gain: graded-relevance ranking score that rewards the best results near the top. 1.0 = ideal order.

Mean Reciprocal Rank: average of 1/rank of the first relevant result per query. Answers 'how far down is the first good hit?'

Share of an answer's claims that the provided context supports. Measures grounding, not truth: a faithful answer from a wrong doc is still wrong.

RAG metric: are the relevant chunks ranked at the top of the context? Rank-weighted precision, so noise above the evidence costs more.

Latency at a percentile: p50 is the typical request, p95/p99 the slow tail. SLOs use the tail because averages hide the users who wait.

Queries per second: request rate a system handles. For LLMs, convert QPS to concurrency and tokens/s; that is what sizes GPUs and bills.

Service Level Objective: a target for a measured indicator over a window, e.g. 99.5% of answers under 5 s over 30 days. Its gap is the error budget.

Replace one pipeline stage with perfect input (e.g. gold passages) and measure the ceiling. The gap shows which stage to fix first.

Reading real failures, labelling why each happened, and counting the categories, so you fix the biggest cause instead of guessing.

Production behaviour shifting from what you evaluated: user inputs, corpus, provider model, or what 'correct' means changes under you.

Decide which number a change should move, build the cases that measure it, then change the prompt, model or retriever and read the delta with error bars.

Offline eval scores a fixed test set before release; online eval measures real users after it. Offline is the gate, online is the verdict.

A written scoring guide: one observable criterion per line, a small scale (usually pass/fail), and an anchored example for each score.

Grading by asking which of two answers is better, not scoring each alone; more reliable for subjective quality, but needs order swaps to cancel position bias.

People labelling model outputs against a rubric; the ground truth that LLM judges are calibrated against, and the most expensive data in the eval stack.

How often independent graders agree beyond chance, usually Cohen's kappa; below ~0.6 your labels, and any judge validated against them, are unreliable.

A public, fixed test set and scoring rule (MMLU, GPQA, SWE-bench) for comparing models; a coarse filter for model choice, never a substitute for your own eval.

When test items (or close paraphrases) leaked into a model's training data, so its score measures memory instead of skill.

A CI check that runs the eval suite on every prompt, model or config change and blocks the merge or rollout if quality drops beyond the measured noise.

Randomly split live users between the current and new version, then compare a pre-chosen outcome metric with a significance test to decide what ships.

Whether an eval or A/B delta is bigger than the noise; report confidence intervals, use paired tests on shared items, and size n before you run.

Model-generated inputs, labels or conversations that fill eval or training gaps; fast to make, trustworthy only after filtering, dedupe and human spot checks.

The loop that turns production traces and feedback into labelled failures, new eval cases and fixes, so each week of traffic makes the system measurably better.

Deliberately attacking your own AI system (jailbreaks, prompt injection, data extraction, unsafe tool use) to find failures before attackers or users do.

An input crafted to make a model ignore its safety training or system instructions: role-play framing, encoded text, many-shot examples, adversarial suffixes.

Detecting and masking personal data (names, emails, IDs, health details) before it reaches a model, a log or a vendor; recall matters more than precision.

A proxy between your apps and model providers that centralises auth, routing, fallback, rate limits, budgets, caching, redaction and logging.

Capping requests and tokens per key, tenant or user per time window, usually with token buckets, so one caller cannot exhaust shared provider quota or budget.

Routing a request to a backup model or provider when the primary errors, times out or is overloaded; only safe if the backup has passed the same evals.

Treating prompts as immutable, versioned bundles pinned to a model snapshot and logged on every trace, so any output traces back to exactly what produced it.

Total spend to answer one request: input, cache writes and reads, output and reasoning tokens, retrieval, reranking, tool calls and retries.

A provider retiring a model snapshot on a set date; every dependent prompt, eval and fallback must be migrated and re-validated before requests start failing.

Whether stated confidence matches actual accuracy: of answers given 80% confidence, about 80% should be right. Needed for abstention and routing thresholds.

Running an operation twice leaves the same state as running it once. It is the property that makes retries and at-least-once queues safe.

Pushing back on producers when a consumer is saturated: bounded queues, 429s and load shedding instead of unbounded buffering until latency or memory collapses.

A side queue where messages land after N failed attempts, so one poison message stops blocking the pipeline and can be inspected and replayed later.

Wait longer after each failed retry (1 s, 2 s, 4 s…) plus random jitter, so clients don't hammer a struggling service in sync.

Run the new version (green) beside the old (blue), verify it on real infra, then switch all traffic at once, keeping blue warm for instant rollback.

Writing the same change to two stores, e.g. Postgres and a vector index. Done naively it drifts; the outbox pattern plus CDC makes it reliable.

Copy real production requests to a new version whose responses users never see, then compare outputs, latency and cost before risking live traffic.

Release to a small slice of real traffic first (1%, then 5%, 25%…), compare against the old version, and roll back automatically if metrics degrade.

A runtime switch that turns code paths, models or prompts on or off per user, tenant or percentage without a redeploy.

A stable name (e.g. products-read) that points to a physical index, so you can rebuild a new index and repoint readers atomically with no client change.

Background rewriting of an index's segments to drop deleted and overwritten entries, reclaiming space and restoring search speed and recall.

Being able to answer new questions about production from its telemetry. For LLM systems: traces per request with prompts, tool calls, tokens, cost and quality.

Time to live: how long a cached or stored item is valid before it expires. Too long serves stale data; too short throws away savings.

A scheduled job that compares the source of truth with derived stores (index, cache, ACLs) and repairs drift that events and retries missed.

When a dependency fails, serve a reduced but useful response (smaller model, cached answer, sources only) instead of an error page.

A blameless written review after an incident: timeline, impact, root causes, what went well, and owned action items with due dates.