How do you know if an AI is any good?

Picture a software company showing a law firm its new contract assistant. The salesperson pastes in a 40-page supplier agreement and asks, “Can either side end this contract early?” Three seconds later the assistant answers in tidy paragraphs, naming the right clause and the 90-day notice period. The partners are impressed. Someone asks, “So how often does it get that right?” The room goes quiet.

That quiet is the whole subject of this post. The demo showed that the assistant can answer well. It said nothing about how often it does, on the contracts and questions the firm will actually give it.

Think of hiring a cook. They make one dish for you and it is delicious. You still do not know how they cope with a busy Friday, an unusual allergy, or a dish they have never made. One great plate shows skill; it does not show reliability.

AI demos have the same gap, in three places:

• The questions were chosen. The person running the demo picked ones that work. Real users ask stranger, messier things.
• The answers vary. Ask a language model the same question twice and you can get two different answers. One good answer does not promise the next.
• Fluent is not correct. These systems can produce confident, well-written answers that are wrong, which is called hallucination. In a demo, nobody stops to check the clause number.

The way serious teams check an AI system is an exam. They write down realistic questions — 40 or 50 to start — and for each one record what a correct answer must say or must not say. That list is called a golden set: “golden” because the answers are the trusted reference. Every time the system changes, they run the whole exam and count how many answers pass.

The exam must stay the same between runs. If the score moves after a change, you want to know the system moved, not the questions.

A fair trial for the cook includes the popular dishes, a couple of awkward requests, and at least one order they should refuse. A good golden set mixes:

• Common questions that make up most real use — “What is the notice period?”
• Hard questions the system has got wrong before — clauses that refer to other clauses.
• Trick questions where the right answer is “the contract doesn’t say” or “I can’t help with that.”
• Different people asking differently — a partner’s precise wording, a trainee’s vague one.

An exam of easy questions gives a flattering score that means little. An exam drawn from what people really ask gives a lower, more useful one.

Some answers are easy to mark: if the answer must say “90 days”, a simple check can confirm it. Others are not: is this summary of a contract good? For those, teams write a rubric, like a teacher’s marking guide — does it name the right clause, does it avoid adding terms that are not in the contract, is it under 100 words?

People are the most trustworthy markers and also the slowest. So teams often have a second AI model mark the answers against the rubric, called LLM-as-judge. It is fast but has blind spots — for example, it tends to like longer answers — so careful teams check a sample of its marks against a person’s before trusting it.

Here is the part demos never mention. Say the assistant passes 32 of 40 questions: 80%. That number is less precise than it looks.

Each question is a bit like a coin toss that lands “pass” 80% of the time. With only 40 tosses, luck alone moves the count by several either way. Statisticians put a range around the score, called a 95% confidence interval: the range that would contain the true pass rate in 95 of 100 repeats of the experiment. For 80% on 40 questions it is about ±12 points. In plain words: “80% on 40 questions” means somewhere between about 68% and 92%.

Two practical consequences. First, if a new version scores 85% where the old one scored 80%, both on the same 40 questions, you cannot yet say it is better; the ranges overlap almost entirely. Telling those apart reliably takes hundreds of questions, or re-running the same questions several times. This is what statistical significance is about. Second, anyone quoting a single score without saying how many questions it came from is giving you half a fact. Evan Miller’s 2024 paper Adding Error Bars to Evals makes the same argument for published AI results.

The score tells you whether something changed. Reading the wrong answers tells you what to fix — a habit called error analysis. Failures tend to cluster: every mistake involves dates, or clauses that point to other clauses, or contracts longer than 60 pages. One such pattern is worth more than a point of score.

• Do you test it with a fixed set of questions after every change? How many questions?
• Where did the questions come from — real users, or written to look good?
• What does it get wrong, and how often? Can I see some failures?
• What does it do when the answer isn’t in the document?
• When you change it, how do you know you didn’t break something that used to work?

Teams with crisp answers — “212 questions from real support tickets, 86% pass, here are the failure types” — build tools that hold up. Teams who answer with another demo are asking you to judge the cook on one plate.