Blast radius
How much breaks when one thing fails or is compromised: how many users, tenants, records or actions. Design to keep it small and known.
Blast radius is the amount of damage one failure can cause. A bad deploy that reaches 1% of users has a small blast radius; one that reaches everyone at once has a huge one. A leaked key for one tenant is bad; a leaked admin key for all tenants is a catastrophe. You shrink blast radius with staged rollouts (Canary & ramped rollout), isolation between customers (Multi-tenancy), and narrow permissions (Least privilege).
Ships have watertight compartments so a hole floods one section, not the whole hull.
Treating config, prompts and feature flags as not-a-deploy. They often go global instantly and bypass the staged rollout that code changes get.
Measure blast radius on three axes: who (users, tenants, regions), what (records readable, actions executable), and how long (time until detection and containment). Then shrink each. Infrastructure: cells (independent copies of the stack each serving a subset of tenants) and shuffle sharding limit how many tenants one failure touches. Change management: rings and canaries so a bad change hits 1% first; config and prompt changes count as changes. Security: per-tenant, per-task credentials so one compromise exposes one tenant's data, not all. Agents: cap per-run tokens, steps, spend and the number of records any tool call can touch (e.g. a bulk update tool limited to 50 rows without approval). Time: detection speed is part of the radius, which is why canary metrics and egress alerts matter. CrowdStrike's July 2024 incident is the canonical example of an unbounded blast radius from a global, simultaneous content push. Write the expected radius into design docs: 'worst case, one tenant's drafts for 15 minutes.'
Cell-based deployment for multi-tenant serving; staged rollout for code, config, prompts and flags; per-tenant and per-task credentials; agent caps (tokens, steps, spend, rows per tool call); blast-radius statement in every design doc; detection SLOs on canary gates.
A customer-support platform serving 3,000 tenants runs 6 cells of ~500 tenants each, with prompt changes rolled out cell by cell over 2 days. A prompt change that broke ticket-tagging JSON reached cell 1 (500 tenants, 1% traffic slice first), was caught by a schema-validity canary gate within 12 minutes, and affected 140 tickets. Without cells and canaries, the same change would have reached all 3,000 tenants and ~60k tickets/hour.
I'd state blast radius explicitly for each design: who, what and for how long, then shrink each with cells, staged rollouts, per-tenant credentials and per-run caps on agents. Follow-up: 'what's the radius of your agent?' Answer with numbers: records reachable, actions without approval, credential lifetime.
| Control | Shrinks | Typical setting |
|---|---|---|
| Cells | who (tenants) | ≤500 tenants/cell |
| Canary rings | who (traffic) | 1% → 5% → 25% |
| Per-task creds | what (records) | one tenant, 15 min |
| Agent caps | what (actions) | ≤50 rows w/o approval |
| Fast detection | how long | gate alerts <15 min |
Numbers worth knowing
- Tenants per cell
- a few hundredsmall enough that one cell's outage is a contained incident
- Agent bulk-action cap
- ~50 records without approvaltune per tool; higher counts need a human
- Detection target
- <15 minfor config/prompt regressions caught by canary gates
How to decide
| When | Do this |
|---|---|
| Any change that ships globally at once (config, prompts, rules) | Put it through rings like code. |
| Multi-tenant serving with enterprise SLAs | Cells, so one failure stays in one cell. |
| Agent can modify data | Per-call record caps and approvals above them. |
| Credential compromise scenario | Ensure a single key exposes one tenant for minutes, not all tenants indefinitely. |
Cost maths
Cells cost duplication: 6 cells instead of 1 pool adds fixed overhead per cell (load balancers, monitoring, minimum node counts), often 10–25% more infra. The return is bounded incidents: an outage that hits 1/6 of tenants instead of all of them cuts SLA credits and churn exposure roughly 6×.
Operating it
- A single change can reach >10% of users within minutes
- One credential can read data across tenants
- Postmortems list 'global rollout' as a contributing factor
- Share of users/tenants exposed to a change in its first hour: alert if a change reaches >10% in <1 h
- Cross-tenant credential usage: alert on any
- Agent tool calls affecting > cap records: alert and require approval
When it goes wrong
On 19 July 2024 about 8.5 million Windows machines crashed into boot loops worldwide, grounding flights and halting hospital systems.
Immediate, global customer reports.
Per CrowdStrike's RCA, a content configuration update for the Falcon sensor was released to all hosts at once; a mismatch between expected and supplied input fields caused an out-of-bounds read in kernel mode.
Staged deployment of content updates with canary rings, more validation, and customer control over timing; the general lesson is that any global, simultaneous push has an unbounded blast radius.
Where it shows up
Sources
- Workload isolation using shuffle-sharding — AWS Builders' Library, 2019blog
- External Technical Root Cause Analysis — Channel File 291 — CrowdStrike, 2024postmortem
Check yourself
Three questions. Every option is explained once you answer.