Error analysis
Reading real failures, labelling why each happened, and counting the categories, so you fix the biggest cause instead of guessing.
Error analysis means looking at actual bad outputs one by one, writing down what went wrong in each, grouping similar problems, and counting them. The biggest group is what you fix first. It sounds slow, but an afternoon of reading 100 failures usually tells you more than a week of tweaking prompts.
A doctor reviewing the charts of patients who came back sick, grouping the causes, and fixing the most common one instead of changing every prescription.
Labelling the most visible symptom instead of the first error. 'Hallucinated a price' is often 'retrieved last year's price list', which needs a retrieval fix, not a prompt fix.
Process: (1) sample 50–100 failures from production traces or golden-set misses, stratified by intent; (2) open coding: one person writes a free-text note per trace on the first thing that went wrong (upstream errors cause downstream ones); (3) axial coding: cluster notes into 6–12 categories with crisp definitions; (4) count, and attach the stage owner (retrieval, assembly, prompt, model, tool, data); (5) fix the largest category, add a targeted check or eval for it, repeat. Shankar et al. (2024) found that evaluation criteria themselves emerge from reading outputs ('criteria drift'), which is why error analysis precedes writing automated judges. LLMs can help cluster notes, but a human must read traces; the value is in seeing the failure. Re-run after each fix: categories shift as big ones close.
Weekly 60–90 minute session: sample 50–100 failures (thumbs-down, judge failures, golden misses) from a trace viewer (Langfuse, LangSmith, Braintrust, Phoenix); one-line note per trace on the first error; cluster into ≤ 12 categories; spreadsheet with counts, owner and a linked eval check per category.
Customer-support assistant, 100 thumbs-down traces from one week. Categories: retrieved wrong product version 27; answer correct but too long, user gave up 18; policy changed, index stale 14; misread user intent (billing vs technical) 12; tool call to order API failed silently 11; true hallucination 8; other 10. The team had been planning hallucination work; the data said version-aware retrieval and index freshness covered 41% of failures.
Say: before writing an eval metric or changing a model, I read 100 failures, code the first error in each, and count; the top two categories become the next sprint and the next eval checks. Expect: 'that doesn't scale': it does not need to; the sample is fixed size, and LLM-assisted clustering speeds up step three, but a human still reads the traces.
Numbers worth knowing
- Sample size
- 50–100 tracesenough to see categories ≥ 5% reliably
- Reading time
- 1–3 min per tracewith a trace viewer showing retrieval, prompt and tool calls
- Saturation
- new categories stop appearing after ~40–60 tracessingle product, one intent family
How to decide
| When | Do this |
|---|---|
| Launching a new product or intent | Error analysis before writing any automated judge |
| A category exceeds 15% of failures | Dedicated fix plus a regression eval for it |
| Categories are spread evenly with no leader | Look upstream: data quality or task definition |
Cost maths
100 traces × 2 min = 3.3 engineer-hours per week ≈ $400 at $120/h loaded. Compared to a sprint of prompt tweaking aimed at the wrong category, it is the highest-return hour in the week.
Operating it
- Thumbs-down rate rises with no obvious cause
- New corpus, tenant or language added
- Automated judge scores and user feedback disagree
- Category share week over week; flag any category growing > 5 points
- Share of categories with a linked automated eval; target 100% for categories > 5%
- Thumbs-down rate by intent; trend
When it goes wrong
Three months of prompt engineering to reduce 'hallucinations' moved user satisfaction by less than a point
A new lead read 100 failed traces in an afternoon
Only 8% were model hallucinations; most were retrieval of the wrong product version and stale content, invisible from aggregate metrics
Version metadata filter and hourly re-indexing; thumbs-down fell from 11% to 6.5% in three weeks. Weekly error analysis became a team ritual
Related terms
Where it shows up
Sources
- Your AI Product Needs Evals — Husain, 2024blog
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.