Error analysis, step by step: from 100 traces to a failure taxonomy
Read real traces, write free-form notes, group them into categories, rank by frequency × impact, and turn the top categories into evals that gate releases.
Most teams start evaluation by choosing metrics: helpfulness, toxicity, a generic “hallucination score”. Then the metrics move and nobody knows what to fix. Error analysis reverses the order. You read what the system actually did, name the ways it failed, count them, and only then write evals, one per failure mode that matters.
The approach below follows the one Hamel Husain and Shreya Shankar teach and write about: open coding, then axial coding, borrowed from qualitative research, with a single domain expert making the calls. Their write-ups are in the sources; this post turns it into a checklist with a working tally script.
Step 1: sample about 100 traces
Pull complete traces: user input, retrieved context, every tool call and result, the final output. A final answer alone hides where things went wrong. Sample for diversity rather than at random: different user segments, long and short sessions, sessions with thumbs-down or escalations, and sessions that hit rare tools. If you have no production traffic yet, generate realistic inputs along a few dimensions (persona × task × difficulty) and run them through the system.
The number 100 is a starting pool, not a quota. Keep reading until new traces stop producing new kinds of failure. Qualitative researchers call this theoretical saturation; in practice it shows up as a flattening curve.
Step 2: open coding
Read each trace and write one short, specific note about the first thing that went wrong, or mark it pass. “Quoted the 2024 deductible; the claim was filed under the 2025 policy” is a useful note. “Bad answer” and “hallucination” are not: they do not tell you what to fix. Do not start from a category list; it makes you blind to everything outside it.
- One owner. One domain expert (the person whose judgement defines “good” for this product) makes the calls. Committees average away the sharp edges. Use agreement checks later, when you hand labelling to others.
- Binary verdicts. Pass or fail, plus the note. 1–5 scales feel richer but drift between sessions and between people.
- First failure only. Downstream errors usually cascade from the first one; fixing upstream fixes them.
- Budget. At 1–2 minutes per trace, 100 traces is two to three focused hours. It is the highest-return time in the project.
Step 3: axial coding
Group the notes into categories that each point at one fix. A model can draft the grouping from your notes, but you edit it: merge categories that share a fix, split ones that need different fixes. Then go back and label every trace with its category. Expect 5–12 categories; more than 15 usually means the names are too specific.
Shankar et al. (2024) found that evaluators refine their criteria while grading (“criteria drift”). Expect your category definitions to change during this step, and re-label the early traces at the end.
Step 4: rank by frequency × impact
Frequency comes from the counts. Impact is a judgement the product owner makes once per category, on a 1–5 scale, before looking at the ranking: what does one occurrence cost a user or the business? A rare category can still top the list: a missed escalation in an insurance assistant matters more than a stiff tone.
"""Turn open-coded trace notes into a ranked failure taxonomy (frequency x impact)."""
from collections import Counter
# One row per reviewed trace: (trace_id, verdict, axial category or None, open-coding note).
# Toy rows for an insurance-claims assistant; in practice you load ~100 from your trace store.
CODED = [
("t01", "fail", "wrong-policy-version", "quoted 2024 deductible, claim filed under 2025 policy"),
("t02", "pass", None, ""),
("t03", "fail", "missed-handoff", "user asked for a human twice, bot kept answering"),
("t04", "fail", "wrong-policy-version", "cited superseded flood exclusion"),
("t05", "fail", "date-math", "said 30-day window open on day 34"),
("t06", "pass", None, ""),
("t07", "fail", "tone", "overly formal reply to bereavement claim"),
("t08", "fail", "date-math", "counted business days as calendar days"),
("t09", "fail", "wrong-policy-version", "pulled rider from a different product"),
("t10", "pass", None, ""),
("t11", "fail", "missed-handoff", "fraud keyword present, no escalation"),
("t12", "pass", None, ""),
]
# Impact 1-5, agreed by the domain owner before ranking: what one occurrence costs.
IMPACT = {"wrong-policy-version": 5, "missed-handoff": 4, "date-math": 3, "tone": 1}
def rank(coded, impact):
fails = [c for _, v, c, _ in coded if v == "fail"]
freq = Counter(fails)
n = len(coded)
rows = [(cat, cnt / n, impact[cat], cnt / n * impact[cat]) for cat, cnt in freq.items()]
return sorted(rows, key=lambda r: r[3], reverse=True), len(fails) / n
if __name__ == "__main__":
rows, fail_rate = rank(CODED, IMPACT)
print(f"traces={len(CODED)} fail_rate={fail_rate:.0%}")
print(f"{'category':<22}{'freq':>6}{'impact':>8}{'priority':>10}")
for cat, f, imp, pri in rows:
print(f"{cat:<22}{f:>6.0%}{imp:>8}{pri:>10.2f}")traces=12 fail_rate=67%
category freq impact priority
wrong-policy-version 25% 5 1.25
missed-handoff 17% 4 0.67
date-math 17% 3 0.50
tone 8% 1 0.08Step 5: turn categories into evals
Each high-priority category becomes either a fix with a regression test or a standing eval. Pick the cheapest evaluator that can detect it:
- Code assertions when the failure is mechanical: date arithmetic, a JSON field missing, a citation id that is not in the context, an escalation keyword with no handoff tool call (a trajectory check). Free, deterministic, run on every commit.
- An LLM judge when it needs reading comprehension: “does the answer use the policy version in force on the claim date?”. One judge per category, binary output, a short rubric written from your notes, and validated against your own labels: aim for ≥ 90% agreement on 50+ held-out traces before trusting it.
- Retrieval metrics when the root cause is upstream: if wrong-policy-version traces all retrieved superseded documents, the eval is a retrieval test with metadata, not a generation judge.
The traces you labelled become the first version of the golden set, with failures over-represented on purpose. Wire the evals into an eval harness and make the top categories an eval gate on prompt, model and retrieval changes.
Key takeaways
- Read traces before choosing metrics; the failure modes are specific to your product and you cannot guess them.
- Open-code freely with one expert and binary verdicts, then group into 5–12 categories that each map to a fix.
- Stop reading at saturation, not at a fixed count; ~100 traces is a pool to start from.
- Rank by frequency × impact so a rare, costly failure outranks a common, harmless one.
- Turn each top category into the cheapest evaluator that detects it: code first, a validated judge second.
Sources
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- A Field Guide to Rapidly Improving AI Products — Hamel Husain, 2025blog
- AI Evals: Everything You Need to Know — Hamel Husain & Shreya Shankar, 2026blog
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper