Shipping LLM features safely

Here is a worked incident — hypothetical, but assembled from failure modes that recur — on an e-commerce catalog assistant that answers product questions and returns structured size recommendations. Nothing about it is unusual, which is the point.

The release worked as designed: a regression the offline gate missed was caught on 5% of traffic and reversed in two minutes with no deploy. But the gate should have caught it, and it did not for two reasons that recur everywhere. The overall score rose while one slice collapsed, and the golden set had no rows for that slice. This post builds the path that makes this incident boring and then makes it rarer.

Bundle into one versioned config, logged on every request: prompt template and id, exact model version (never a floating alias), sampling settings, retrieval config, tool schemas, and the judge prompt used to evaluate it. Prompt versioning is what made “1,700 canary requests, 32 bad responses” a 15-minute query instead of a forensic project, and it is what makes rollback a one-field change.

Model output is not deterministic, and setting temperature to 0 does not make it so: providers and engines such as vLLM batch requests together, and floating-point reductions that depend on batch composition change outputs between identical calls (He / Thinking Machines, 2025). A gate that compares one run of the baseline with one run of the candidate is comparing two samples of noise.

So the eval gate runs each example 5 times per config, computes per-slice change with a paired bootstrap interval, and blocks only when a slice is worse by more than its tolerance with 95% confidence. Safety is a hard floor: every safety example must pass every repeat. The rules are fixed before anyone sees numbers. The script below is deliberately constructed to fail: it exits with code 1, which is what fails the CI job.

This is v42 with the sizing slice in place. Overall is up 2.4 points (and the interval says even that is not established); orders genuinely improved; sizing is down 27 points with an interval that excludes anything better than −11.7. The gate blocks, correctly. Without per-slice rules the average would have shipped it. Without repeats and intervals, the same rule would fire on noise once a week and people would learn to override it.

Budget it: 42 examples × 5 repeats × 2 configs = 420 calls. At roughly $0.01 a call that is under $5 and, at 20 concurrent requests, about a minute. Run the baseline once per baseline change and cache it, which halves the cost. Keep the gate under 10 minutes or people will route around it.

Shadow traffic sends a copy of real requests to the candidate and records its outputs while users only see the current version. It catches what offline sets miss — the incident above would have shown up as a 1.9% malformed-JSON rate in shadow before any user saw it. It doubles model spend on the mirrored share, and must never trigger side effects: for agents, shadow only read-only tools or point tools at stubs.

A canary exposes a small share of users to the candidate behind a feature flag and compares it with the rest. The question is how long to hold each step, and the answer is a sample-size calculation, not a habit. For comparing two rates with a two-sided α = 0.05 and 80% power:

At 120k requests a day, a 5% canary needs about 27 hours to detect a one-point rise in thumbs-down rate. So the rule is: hold the first canary step for max(time to reach n on your slowest guardrail metric, one full daily cycle), because traffic at 3 a.m. is not traffic at 3 p.m.; later steps carry more traffic and need only reach n. Large regressions show up far sooner — the 4.75× malformed-JSON jump above needed a few hundred requests — so pair the fixed dwell with a fast-trip threshold that rolls back immediately on gross failures.

Guardrails, each with a threshold written down before the ramp starts: error and timeout rate including malformed structured output; p95 and p99 latency (longer outputs show up in the tail first); cost per request; refusal and guardrail-trigger rates; thumbs-down, retries and escalations to a human.

A canary asks “is it safe?”; an A/B test asks “is it better?”. Fix the primary metric and sample size before starting and do not stop early on a good-looking day: peeking at a 5% test daily for two weeks and stopping at the first significant result pushes the false-positive rate well above 5%. Use a sequential method if you must look. Expect offline and online results to disagree sometimes; an offline gain that does not appear online usually means the golden set no longer matches traffic.

Rollback is a flag flip to the previous config version, never a code deploy. Rehearse it quarterly in production with a no-op config change, and time it; the target is under 5 minutes from decision to 0% exposure. Unrehearsed rollbacks fail on the details: a response cache still serving new-version answers, a schema the old prompt cannot read, a downstream consumer that already migrated. Keep prompt and schema changes backward compatible for at least one version so the previous config can always run against current data.

Every incident becomes golden-set rows before the fix merges — the incident above added 12 sizing rows. Write the postmortem around one question: which gate should have caught this, and what does it need so it will next time? Over a year, that feedback loop turns the eval set into a record of everything that has gone wrong, which is exactly what a gate should check.