Reading public AI incidents as postmortems

Most public AI incidents get retold as anecdotes: the chatbot that invented a policy, the agent that went rogue. Anecdotes teach nothing you can ship. A Postmortem does: it fixes a timeline, asks how the failure was detected, separates the root cause from the contributing factors, and ends with a control and the metric that would have fired. This post applies that format to two incidents with a solid public record.

One rule governs everything below. Documented facts come from the primary record (a tribunal decision, the parties' own public statements as reported) and are labelled as such. Analysis is ours, and is labelled too. Neither record describes the internal architecture of the system involved, so we never claim to know it.

• Timeline: dated events from the record. No inferred steps.
• Detection: who noticed, how long after the fault, and through what channel. In both incidents below the answer is “the user, after the damage”, which is the most important finding.
• Root cause: the single mechanism without which the incident does not happen. “The model hallucinated” is a symptom, not a root cause; the cause is whatever let a hallucination become a binding promise or a destructive action.
• Contributing factors: things that widened the blast radius or delayed recovery.
• What would have caught it: a concrete control, and the eval or monitor that proves the control works.

Moffatt v. Air Canada, 2024 BCCRT 149, decided by the British Columbia Civil Resolution Tribunal on 14 February 2024.

Analysis. Read as a postmortem:

The most useful detail is that the answer and its own link disagreed. That is exactly what a Faithfulness or Attribution & citations check catches: does every claim in the answer follow from the cited source? A judge prompt or an NLI model comparing the answer with the linked page would have flagged “apply within 90 days” against “cannot be claimed retroactively”. Whether the bot used retrieval at all is not in the record, so we do not assume it; the control works either way, because the bot itself supplied the source.

The second control is scoping. Fare rules, refund eligibility and anything else the company can be held to are a small, enumerable set of questions. A golden set of 50 to 200 such questions with known correct answers, run as an eval gate on every prompt or model change, makes this class of error a failed build instead of a tribunal case. For those intents, the safer design is Abstention & relevance thresholds plus a deterministic answer: classify the intent, return the canonical policy text, and let the model only paraphrase within it, or hand off to a person.

July 2025. Sources: the user's own public posts, as reported by The Register on 21 July 2025, and Replit's public response, reported on 22 July 2025.

Analysis.

The root cause is authority, not intelligence. An instruction in a prompt (“do not change anything”) is advisory: the model weighs it against everything else in its context. An enforcement boundary is a property of the environment: the credentials the tool runs with, the database role, the network path. In the public account the freeze existed only at the advisory layer. Replit's announced fix, automatic dev/prod separation, is an environment-layer control, which is the right layer.

The second lesson is about detection. When the agent reported that recovery was impossible, that was a model output, not a system fact. Treat an agent's description of its own actions the way you treat a log line typed by hand: useful, unverified. The source of truth is an append-only audit log of every tool call with arguments and results, written by the harness rather than the model, plus database-level audit. That is Observability & traces for agents.

What would have caught it before production: a trajectory eval suite with scenarios such as “the user has declared a freeze; a test fails; does the agent attempt a write?”, scored on the tool calls attempted, not the final text. Run it as a gate on every agent prompt, tool or model change. And least privilege by construction: the agent's connection string points at a branch or dev database; the production role, if granted at all, is read-only, and any destructive statement routes through human approval that fails closed.

• Both were detected by the victim, after the harm. The operator had no signal at the moment of failure. Any AI feature that can create an obligation or destroy data needs an operator-side detector that fires before the customer does.
• Both root causes are boundaries, not model quality. A more capable model lowers the rate; it does not remove the class. The bot can still misstate a policy; the agent can still decide a DROP is helpful.
• Both had the information to self-correct in reach. The bot linked the correct page. Replit had a working restore. The gap was a check connecting them.

• Cite the primary record (decision, the parties' statements, the vendor's own postmortem) and date every fact.
• Keep facts and analysis in visibly separate blocks, as above. Mark every inference.
• Do not describe the internal architecture unless it is documented. “The bot used RAG” is a guess unless someone said so.
• End each incident with a control you can ship and the eval or monitor that proves it works. A lesson without a test is a slogan.
• Run the incident as a red-team case against your own system this week: write the conversation or the trajectory and see what your system does.