Fail closed
When a security check can't run (auth service down, classifier timeout), deny by default. The opposite, fail-open, keeps serving and skips the check.
Fail-closed means: if you can't confirm something is allowed, treat it as not allowed. If the permission service is down, the assistant refuses to search private documents rather than searching everything. Fail-open means the opposite: keep serving and skip the check. Security and privacy controls should fail closed; convenience features (spell-check, a related-articles widget) can fail open.
An electronic door lock that stays locked during a power cut (fail-closed) vs a fire door that swings open (fail-open). Bank vaults are the first kind; emergency exits are the second.
Implicit fail-open through exception handling. try: check() except: log.warning(...) reads like defensive code and silently disables the control whenever the dependency is slow.
Saltzer and Schroeder's 1975 principle of fail-safe defaults: base access decisions on permission, not exclusion, so the default is lack of access. In practice: every security check returns allow/deny/error, and error maps to deny, explicitly in code (catch → deny, not catch → log and continue). Applies to authorization and ACL (access control list) lookups, tenant-context resolution, PII redaction before logging or sending to a model, prompt-injection and content classifiers on high-risk paths, and tool-permission checks in agents. Timeouts are the trap: a guardrail with a 300 ms timeout that falls through on timeout is fail-open under load, exactly when attackers probe. Fail-closed has a cost, availability, so make it cheap: cache positive decisions briefly, run checks in-region with their own SLOs, and design a clear user-facing denial ('can't verify access right now') instead of a blank error. Decide per check and write it down: the decision is a product choice, not something to discover in an incident.
Every security check returns allow/deny/error with error → deny in code; check services have their own SLO and in-region replicas; positive decisions cached ≤60 s; denial UX with retry; fail-open exceptions require a named owner and are listed in the threat model; chaos test that kills each check service and asserts denial.
A clinical-intake assistant redacts PII before sending notes to a hosted model. The redaction service (Presidio-based) has a 400 ms timeout. Old behaviour: on timeout, the raw note was sent 'to avoid blocking clinicians'. New: on timeout the request is queued for redaction and the clinician sees 'summary in progress', ready ~30 s later. During a redaction-service brownout, 2.1% of requests were delayed and zero raw notes left the network.
I'd classify every check: auth, ACL, tenant context, PII redaction and tool permissions fail closed; enrichment and UX features fail open. Then I'd make fail-closed affordable with caching, separate SLOs for the check services and a good denial UX. Follow-up: 'the business says availability matters more' — I'd show the trade per check and get an owner to sign off on any fail-open security control.
Fail-closed (security)
- Auth / ACL lookups
- Tenant context
- PII redaction
- Agent tool permissions
- Error → deny
Fail-open (convenience)
- Related-articles widget
- Spell-check / autocomplete
- Analytics events
- Non-critical enrichment
- Error → skip
Numbers worth knowing
- Positive-decision cache
- ≤60 strades a short revocation lag for availability
- Check service SLO
- ≥ the product's SLOa fail-closed dependency caps your availability at its own
- Availability cost
- = check service downtime on protected pathse.g. 99.95% check SLO ≈ 22 min/month of denials
How to decide
| When | Do this |
|---|---|
| Check protects data access, tenancy, PII or tool execution | Fail closed. |
| Check only improves UX or analytics | Fail open with logging. |
| High-risk content classifier times out | Block or queue for human review; don't pass through. |
| Fail-closed hurts availability too much | Invest in the check's reliability, don't flip it to open. |
Cost maths
If the redaction service has 99.9% availability, fail-closed means ~43 minutes/month of delayed or denied requests on that path. At 10k notes/day, ~300 notes/month wait an extra 30 s. Fail-open means ~300 unredacted clinical notes/month sent to a third party, each a potential reportable breach.
Operating it
- Denials from check-service errors >0.1% of requests: improve the check's SLO
- New security control added: classify open/closed before launch
- Audit finds a catch-and-continue around a security check
- Security-check errors mapped to deny: alert >0.1% of requests
- Security-check errors mapped to allow: alert on any (should be impossible)
- Chaos test results per release: block release on failure
When it goes wrong
During a load spike, an internal assistant returned documents from restricted finance spaces to non-finance staff.
A security review of retrieval logs a week later; 41 queries returned restricted chunks.
The group-lookup client caught timeouts and returned an empty exclusion list; the filter was written as 'exclude restricted groups the user isn't in', so an empty list excluded nothing.
Filters rewritten as allow-lists (fail-safe defaults), error → deny in the lookup client, and a chaos test that times out the IdP and asserts zero restricted results.
Related terms
Where it shows up
Sources
- The Protection of Information in Computer Systems — Saltzer & Schroeder, 1975paper
- Authorization Cheat Sheet — OWASP, 2025docs
Check yourself
Three questions. Every option is explained once you answer.