Prompt versioning
Treating prompts as immutable, versioned bundles pinned to a model snapshot and logged on every trace, so any output traces back to exactly what produced it.
Prompt versioning means every change to a Prompt & system prompt (system prompt, templates, few-shot examples, tool descriptions) creates a new numbered version that cannot be edited afterwards. Each response is logged with the prompt version and model that produced it.
Then you can compare versions in evals, roll back a bad change in seconds, and answer 'why did the bot say that last Tuesday?'
Version control for a recipe in a restaurant chain: every kitchen cooks from recipe v14, and if customers complain you know exactly which recipe they ate and can switch everyone back to v13.
Versioning the system prompt text but not the model snapshot or tool descriptions. When the provider updates a floating alias or someone edits a tool schema, behaviour changes under an unchanged 'version' and the trace history becomes misleading.
A prompt version is more than the system prompt text: it is the full configuration that determines behaviour: template, few-shot examples, tool schemas and descriptions, output schema, model snapshot id (pin claude-sonnet-4-6 or a dated snapshot, never a floating alias), sampling parameters where the model accepts them, and retrieval settings (Top-k (retrieval k vs context k), filters). Hash that bundle; the hash is the version id.
Store prompts either in git next to the code (simplest; deploy = release) or in a prompt registry (Langfuse, Braintrust, PromptLayer, or a table) when non-engineers edit them; in both cases versions are immutable and changes go through review and the Eval gate (CI). Decouple deploy from release with labels (production, canary) pointing at versions, so promotion and rollback are a label move behind a Feature flag, not a code deploy. Log the version hash on every trace for Observability & traces and to slice online metrics. Prompt changes are code changes with a larger blast radius: a one-line edit to a tool description can change which tool an agent calls on 10% of tasks.
Prompt bundle (template, few-shots, tool schemas, output schema, pinned model snapshot, retrieval params) in git or a registry (Langfuse prompt management, Braintrust); content hash as version id; labels production / canary resolved at runtime with a short TTL cache; every change through the eval gate; version hash on every trace span.
Developer-docs assistant. Bundle v42 = {system prompt sha a91f…, 6 few-shot examples, 3 tool schemas, model claude-sonnet-4-6, top_k 8, max_tokens 1200}. A writer edits one tool description in the registry, creating v43; the eval gate runs (CI on delta [−0.9, +2.1], passes), label canary → v43 for 5% of traffic. Next day, online judge score on v43 is 3 points lower on 'migration guide' questions; label moves back to v42 in 30 seconds, no deploy. Trace slicing by version found it; the fix becomes v44 with a new eval case.
Say: "A prompt version is the whole bundle including the model snapshot and tool schemas, hashed, immutable, gated in CI, released by moving a label, and logged on every trace." Expect: "git or a registry?": git when engineers own prompts; a registry with review and the same gate when product or domain experts edit them.
Numbers worth knowing
- Rollback time with label-based release
- < 1 minutelabel move + runtime cache TTL of 30–60 s
- Canary share for prompt changes
- 5% for 24 hbefore promotion; longer for low-traffic routes
- Prompt-resolution overhead
- < 1 ms p99local cache of the label → version map
How to decide
| When | Do this |
|---|---|
| Only engineers edit prompts | Keep them in git; a deploy is a release, with the eval gate in CI. |
| Product managers or domain experts edit prompts | Use a registry with immutable versions, review, the same eval gate, and label-based release. |
| The provider offers floating model aliases | Pin the snapshot in the bundle; treat a snapshot change as a new version with a full gate run. |
Cost maths
Cost is the eval runs a version change triggers: ~$3–5 per PR smoke run and ~$9 for a nightly full run on the Batch API (see Eval gate (CI)). The saved cost is incident time: a 30-second label rollback vs a 20–40 minute revert-build-deploy cycle during which a bad prompt answers every user.
Operating it
- More than 10 prompts in production or non-engineer editors: move from inline strings to a registry with labels.
- Several services share a prompt: publish it as a versioned package or registry entry, not copy-paste.
- Requests without a version hash on the trace: alert on any.
- Online quality metrics sliced by version: alert when a canary version trails production by more than the CI.
- Count of live versions per route: alert above 3 (abandoned canaries).
When it goes wrong
A legal-contracts assistant began citing clause numbers from the wrong contract section on about 8% of answers; nobody had changed 'the prompt'.
Sampled online judging flagged a drop in the citation criterion; traces showed the same prompt version before and after.
The version covered only the system prompt text. A teammate had edited the retrieval tool's description (now 'returns the most relevant sections') in a separate config, and the model started calling it with broader queries.
Redefined the version as a hash over prompt, tool schemas, model snapshot and retrieval params; added a CI check that fails if any bundle member changes without a version bump.
Related terms
Where it shows up
Sources
- Canarying Releases (The Site Reliability Workbook) — Google SRE, 2018book
- Your AI Product Needs Evals — Hamel Husain, 2024blog
Check yourself
Three questions. Every option is explained once you answer.