How do you version and roll out prompt changes safely?
- 2 min — what a version is (the full tuple)
- 3 min — PR → eval gate → flag ramp → rollback
- 3 min — repo vs runtime store trade-off
- 2 min — logging, ownership, failure modes
A prompt is production code that changes behaviour as much as a deploy does. Prompt versioning means the version is the full tuple — template text, model snapshot id, decoding parameters, tool schemas, retrieval config — stored immutably under a content-hash id. Every response is logged with that id, and changes go through review, the eval gate and a staged rollout like any code change.
A change to a contract-clause extraction prompt is a pull request. CI runs the eval gate on the golden set. On pass, the new version id ships behind a feature flag: 1% for an hour, 5% for a day, 25% and 50% for a day each, then 100% — each step gated on judged pass rate, JSON-parse failure rate, p95 latency and cost per call against the control. Rollback is flipping the flag to the previous id: seconds, no redeploy.
A floating model alias ('latest') is an unreviewed prompt change — pin snapshots. The trade-off of a runtime prompt store (non-engineers edit in a UI, instant changes) vs prompts in the repo (code review, tests coupled to code) is real; my default is repo-as-source-of-truth with the flag system as the deploy mechanism, and a UI that opens a PR rather than editing production. Ownership: feature teams own their prompts; the platform owns the registry, the flag ramp and the version id on every log line. Where it breaks: logs without version ids (you cannot attribute a regression), edits in a vendor dashboard that skip the gate, and downstream parsers that silently break when an output format changes at 5%.
At 50k requests/day, a 5% canary = 2.5k requests/day. To detect a 3-point drop in a 90% judged pass rate at α = 0.05, 80% power: n ≈ 15.7 × 0.09 / 0.03² ≈ 1.6k judged requests per arm — about one day at 5% if every canary request is judged (~$40 of judge calls at $0.025 each).
What the answer sounds like at each level
Keep prompts in version control and test changes before deploying them.
Version the whole tuple under a hash id, log it on every response, gate in CI, and ramp behind a flag with metric checks and instant rollback.
I standardise one registry and ramp policy for every team, pin model snapshots org-wide, make UI edits open PRs, and treat a missing version id in logs as a sev-3 bug.
Follow-ups the interviewer will push
1A domain expert wants to edit prompts without a developer. How?
Give them a UI that writes to a branch and opens a PR; the eval gate runs automatically and an engineer approves. They get speed, production keeps review and attribution.
2The 5% canary looks fine on quality but a downstream service is erroring. Why?
Output format drift — a parser expected a field the new prompt renamed or reordered. Add schema validation of outputs to the canary metrics and use structured outputs where the provider supports them.
Red flags interviewers listen for
- Versions only the prompt text, not model snapshot and parameters.
- Edits prompts live in a dashboard with no eval run.
- Rollback requires a code redeploy.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Version definition | Includes model snapshot, params, tool schema and retrieval config, not just text. |
| Rollout mechanics | Staged ramp with metric gates and seconds-level rollback. |
| Org design | One registry and ramp policy across teams; UI edits go through review. |
Sources
- The Site Reliability Workbook, ch. 16: Canarying Releases — Google, 2018book