← All playgrounds

How Problems Compound (one report, five problems)

Our AI support assistant serves 20 clients. Every problem from Part II shows up in one ordinary evaluation: small sample, unrepresentative data, a lenient judge, picking the best of several prompts, fixing the test set's misses. Each looks small on its own. Run the evaluation, then compare the report with what production sees.

The simulation, in five lines

Truth: 20 clients with true accuracy from about 56% to 83%, average 70%. Only the simulation knows it.
Data: synthetic requests are 6 points easier than real ones; the pilot client is one of the best clients.
Judge: the uncalibrated LLM judge lets 1 in 4 wrong answers pass, and fails 1 in 50 correct ones.
Variants: all prompt variants are equally good; we keep the one that scores best on the test set.
Fixing: each round fixes 30% of the misses the judge flags on the test set; only 1 in 5 of those fixes helps on new requests.

The evaluation pipeline

These are the choices of a busy, well-meaning team. Leave them as they are for the first pass.

1Test cases

Sample size (U.1)

2Where the test cases come from

Representativeness (U.5)

3Who judges the answers

Biased judge (U.4)

4Prompt variants tried, keep the best

Closed-book overfitting (U.2)

5Rounds of fixing the test set's misses

Open-book overfitting (U.3)

6The final number is measured on

Held-out discipline (U.2, U.3)

1The report

Run the evaluation once, the way the team would. Before clicking: what number do you expect, and would you promise it to a new client?

2What production sees

The simulation knows the true accuracy the 20 clients experience. In real life you would learn it from complaints.

3Where the gap comes from

Turn the problems on one at a time, in the order they happen in the pipeline, and measure how much each adds. Each step is the average of 300 repeated evaluations.

4What the 95% interval is worth

Repeat the whole evaluation 200 times, each time on new test cases. How often does the reported 95% interval contain the truth? Then compare with each problem on its own.

5Try the fixes

Each button changes the pipeline above and adds a row. Which single fix closes the gap?

What to take away

Noise adds; biases stack, and in an evaluation most of them point up, because every choice made by looking at the score keeps the lucky errors. More test cases make the interval narrower around the wrong number. No single fix closes the gap: remove the biases (real data, a checked judge, one final measurement on untouched data), then buy precision. And even then, a new client can land anywhere in the spread between clients.