Blue-green deployment
Run the new version (green) beside the old (blue), verify it on real infra, then switch all traffic at once, keeping blue warm for instant rollback.
In a blue-green deployment you keep two full copies of a system. Blue serves users today. You build green, the new version, next to it, test it with real infrastructure, then flip a switch so all traffic goes to green. If something breaks, you flip back. For AI systems the 'system' is often a whole search index re-embedded with a new model: you build the new index beside the old one and swap with an Index alias.
A theatre with two stages: the next show is fully set up on stage B while stage A is performing, and the audience is turned to face B when it is ready. If B's lights fail, turn them back.
Flipping traffic without keeping blue in sync. If writes after the flip go only to green, rollback to blue serves stale data, so the 'instant rollback' you paid for doesn't exist.
Blue-green buys instant, whole-system rollback at the price of running two copies. The switch is a load-balancer weight change, a Kubernetes Service selector flip, or an index alias swap, all near-atomic. Avoid DNS as the switch: clients and resolvers cache past TTL, so 'instant' becomes 'up to an hour'. The hard part is state. Stateless services flip cleanly; anything with a database or index needs both colours to read compatible data, which means expand/contract schema migrations and, for indexes, a Dual-write or Backfill plus Change data capture (CDC) catch-up so green has every write blue received. Keep blue warm (not scaled to zero) for a rollback window sized to how long problems take to show: hours for latency, days for retrieval quality. Blue-green is all-or-nothing, so pair it with Shadow traffic or a Canary & ramped rollout slice before the flip when the risk is quality rather than crashes.
Switch via load-balancer target groups (ALB weighted), Kubernetes Service selector, Argo Rollouts blueGreen strategy, or an Elasticsearch/OpenSearch/Qdrant alias; dual-write during overlap; automated smoke + golden-set check before flip; blue kept warm 24 h for stateless, 7 days for indexes.
A financial-filings search re-embeds 18M chunks from a 768-d model to a 1024-d model. Green is a new OpenSearch index built by a backfill job over 26 hours, then kept current by the CDC consumer writing to both indexes. Shadow queries show recall@10 on the golden set at 0.91 vs 0.84 for blue. The team swaps the filings-read alias in one _aliases call; p95 rises 12 ms, within budget. Blue stays live with dual-writes for 7 days, then is deleted.
I'd use blue-green when rollback must be instant and the change is all-or-nothing, like a re-embedded index where queries and documents must share a model. I'd say up front it doubles infra for the overlap window and that the real work is keeping state in sync, not the switch. Expect: 'how long do you keep blue?' Long enough to see the slowest-moving signal, usually a week for retrieval quality.
Numbers worth knowing
- Infra during overlap
- 2× serving costboth colours at full capacity; vector indexes also 2× storage
- Switch time
- <1 sLB weight or alias swap; DNS-based switches take up to the TTL plus client caching
- Rollback window for index changes
- ~7 daysretrieval-quality regressions surface in feedback slower than latency does
How to decide
| When | Do this |
|---|---|
| Change is all-or-nothing (new embedding model, schema-breaking) | Blue-green with an alias swap. |
| Change can be split by request | Prefer a Canary & ramped rollout; cheaper and limits blast radius. |
| Risk is answer quality, not crashes | Run Shadow traffic on green before the flip. |
| Database schema changes | Expand/contract migration so both colours run on the same schema. |
Cost maths
Overlap cost = serving cost/day × overlap days. A 3-node OpenSearch cluster at ~$1,400/month per colour for an index of 18M × 1024-d vectors: 7-day overlap ≈ $330 extra. Rebuild cost: 18M chunks × 400 tokens = 7.2B tokens; at $0.02/M for a hosted embedding model (illustrative Oct 2026), ≈ $144. Both are small next to a week of degraded search for every analyst.
Operating it
- Index >100M vectors: 2× capacity for overlap becomes expensive; consider per-shard migration or canary by tenant
- Rebuild takes >48 h: add a CDC catch-up phase and checkpoints
- More than one release a day: move stateless services to canaries and keep blue-green for indexes
- Doc-count delta between blue and green: alert >0.1% during overlap
- Error rate and p95 on green vs blue in the first hour: auto-rollback if errors >2× blue
- Golden-set recall@10 on green: block flip if below blue
When it goes wrong
After rolling back from a bad green index, a legal-research tool was missing 2 days of newly filed court documents.
User reports of 'can't find yesterday's ruling'; registry vs index count diverged by 4,100 docs.
Ingestion was repointed to green at cutover, so blue received no writes after the flip; rollback restored a stale index.
Dual-write to both colours until blue is deleted, plus a Reconciliation job job comparing registry and both indexes before any rollback.
Sources
- BlueGreenDeployment — Martin Fowler, 2010blog
- Aliases — Elastic, 2025docs
Check yourself
Three questions. Every option is explained once you answer.