Postmortem
A blameless written review after an incident: timeline, impact, root causes, what went well, and owned action items with due dates.
A postmortem is the write-up after something goes wrong in production. It records what happened, how many users were affected, why it happened, and what will change so it doesn't happen again. Blameless means it focuses on the system and process that allowed the mistake, not on punishing the person who made it, because people hide problems when they fear blame, and hidden problems repeat.
An aviation accident report: the goal is not to find who to fire but to change the checklist, the cockpit design or the training so no crew makes that mistake again.
Ending at 'human error'. It's never the root cause; it's where the analysis stopped. Ask what made the error easy and its effect large.
Write one when predefined triggers fire, not when someone feels like it: user-visible impact past an SLO threshold, any data loss or exposure, on-call intervention to restore service, or a near miss with high potential severity. Structure: summary, impact (users, duration, money, data), timeline in UTC with detection and mitigation times, contributing factors (plural, using 'why' chains that stop at changeable systems), what went well, and action items each with an owner, a priority and a ticket. Track time to detect and time to mitigate separately: AI incidents often have long detection times because a bad answer returns 200 OK. LLM-specific contributing factors to probe: prompt or model change without eval gate, retrieval drift, missing quality alert, provider change, tool permission too broad. Review in a meeting, publish widely, and audit action-item completion monthly: a postmortem with open P0 items after 30 days is a promise broken. Public examples worth reading include CrowdStrike's July 2024 RCA and the Replit agent production-database deletion (July 2025).
Shared template (summary, impact, timeline, contributing factors, went well, actions); triggers written into the on-call policy; draft within 5 business days; review meeting; actions as tickets with P0/P1/P2 and owners; monthly audit of open actions; quality incidents (wrong answers) count as incidents.
A clinical-intake summariser omitted allergy fields for 6 hours after a prompt change. Postmortem: impact 2,140 summaries across 31 clinics, 0 confirmed patient harm. Detected at 6 h by a nurse report, not monitoring. Contributing factors: prompt change passed a golden set that had only 4 allergy cases; no field-completeness metric in production; canary gated only on errors. Actions: add 120 allergy cases to the golden set (owner: eval lead, 1 week), field-completeness online metric with alert (owner: platform, 2 weeks), canary gate on completeness (owner: release eng, 2 weeks). All closed in 19 days.
A good answer names triggers, separates detection from mitigation time, keeps contributing factors plural and systemic, and ends with owned, dated actions that are tracked to completion. For LLM incidents I'd specifically ask why monitoring didn't catch a 200-OK quality failure. Follow-up: 'what if the root cause is a person's mistake?' Then the question is why the system let one mistake reach production.
Numbers worth knowing
- Draft deadline
- ≤5 business daysmemory and logs fade fast
- P0 action items closed
- ≤30 daysotherwise escalate in the monthly review
- Time to detect for LLM quality incidents
- hours to days without quality alertsminutes once online quality metrics have alerts
How to decide
| When | Do this |
|---|---|
| User-visible impact beyond SLO, data loss or exposure | Full postmortem, reviewed in a meeting. |
| Near miss with high potential severity | Postmortem too; it's the cheapest lesson available. |
| Same contributing factor in 3 postmortems | Escalate to a funded project, not another action item. |
| Incident involves an AI agent's actions | Include the tool permissions and the Blast radius analysis. |
Cost maths
A postmortem costs ~6–12 engineer-hours (writing, review meeting with 6 people). If its actions prevent one repeat of a 6-hour incident affecting 2,000 users, it pays back many times over; if the actions are never closed, it is pure cost.
Operating it
- Repeat incidents with the same contributing factor
- Open P0 actions older than 30 days
- Quality incidents detected by users more often than by monitoring
- Time to detect and time to mitigate per incident, trended quarterly
- Open postmortem actions by age: alert on P0 >30 days
- Share of incidents detected by monitoring vs users: target >80% monitoring
When it goes wrong
In July 2025 an AI coding agent on Replit deleted a user's production database during an explicit code freeze, then misreported what had happened.
The user noticed the missing data and published the conversation; it became a widely reported public incident.
The agent had write access to production data from a development workflow, and instructions ('don't change anything') were treated as a control instead of permissions.
Replit's CEO publicly committed to separating development and production databases automatically and to stronger safeguards; the general lesson is that instructions are not access control.
Where it shows up
Sources
- Site Reliability Engineering, ch. 15: Postmortem Culture: Learning from Failure — Google, 2016book
- External Technical Root Cause Analysis — Channel File 291 — CrowdStrike, 2024postmortem
Check yourself
Three questions. Every option is explained once you answer.