Source #3 of evaluation error

Open-Book Overfitting

You looked at the data. From that moment the data is training data, and the score it gives you is no longer a measurement.

How to read and interact with this page

Set the dials. Under “Watch the dev score drift away from the truth”, pick the dev set size, the rounds of look-and-patch, the fix generality g, and how you fix. Then press “Run the iteration loop”.
Read the chart. Blue is the dev score you see; orange is the true catch rate in production, which you normally cannot see. The round-by-round log is under the chart.
Split the gain. “Where the climb came from” separates the patches that fixed a pattern from the ones that only taught the prompt the dev traces.
Average over many runs. “Run 200 loops” shows the gap after each round for three levels of g; “Run across dev-set sizes” shows how much a bigger dev set helps.
Replay a real run. In “Let an AI do the climbing”, step through a recorded prompt-optimizer run round by round: dev accuracy on the 40 tickets it sees, held-out accuracy on the 120 it never sees. Nothing calls a model.
Try first: run the loop with g at 0% and then at 100%, and compare the two gaps.

Nobody cheated

Overfitting sounds like a sin: somebody trained on the test set. In practice it happens to careful people doing reasonable things. Two stories you will recognize.

The trace analyzer

You are building an LLM that reads agent traces and flags the ones where the agent went wrong. You collect 40 traces, label them by hand, and run your prompt. It misses a third of the errors. So you do the obvious thing: you open the misses, read them, understand why the model did not flag them, and add instructions to the prompt. "A tool call that returns an empty result and is never retried counts as an error." Run again: better. Open the new misses, patch again. After five rounds the prompt catches 95% of the errors in your 40 traces.

Every step was good engineering. But each instruction you wrote was shaped by a trace you had in front of you. Some instructions capture a real failure pattern; others only describe that one trace. From the dev score you cannot tell which.

The few-shot classifier

You are classifying support tickets with a few-shot prompt. Policy: every time the classifier gets a ticket wrong, add that ticket, with its correct label, to the examples. It is cheap, it feels like learning, and the dev accuracy goes up every single time, by construction: the model is now being shown the answer. Dev accuracy reaches 100%. Production accuracy moved a little, or not at all.

Both stories are the same mechanism. In classical overfitting the model has too much capacity and memorizes the training data. Here the model is frozen; the developer is the fitting agent, and the dev set is the training data. The prompt is where the memorization lives.

Watch the dev score drift away from the truth

We simulate the trace-analyzer loop. In production, failures come in 30 patterns with a long-tailed frequency distribution: a few patterns are common, most are rare. Your first prompt catches each pattern with some probability between 30% and 80%. You draw a dev set of n traces, and each round you run the prompt, look at every miss, and patch the prompt for it.

A patch always handles the trace you looked at. With probability g (how general your fixes are) it also handles the pattern behind that trace, so it transfers to production. With probability 1 − g it is a rule that only matches that trace. You never know which one you just wrote.

The chart will show two lines: the dev score you see, and the true catch rate on production traffic, which you normally cannot see.

Dev set size 40

Traces you labeled and iterate on.

Rounds of look-and-patch 8

Each round: run, read the misses, patch.

Fix generality g 40%

Chance a patch fixes the pattern, not just the trace.

How you fix

Your rule catches that exact trace 95% of the time.

dev score (what you see)true catch rate (production)

round-by-round log

Round	Dev score	Misses you read	Patches that generalized	True rate

Where the climb came from

The dev score rose for two reasons that look identical on the chart: patches that fixed a pattern, and patches that only taught the prompt the traces in the dev set. The simulation knows which is which, so we can split the gain.

Run the loop above first.

The middle bar is a counterfactual you cannot compute in real life: the dev score if the prompt had not memorized any individual trace. The distance from the middle bar to the true rate is a second, quieter effect: even perfectly general fixes only fix the patterns that happened to appear in your dev set. The long tail you never sampled stays broken.

Every round widens the gap

One run is one draw. Here we repeat the loop 200 times with the dev-set size and fixing mode you chose above, and plot the average gap between the dev score and the true rate after each round, for three levels of fix generality.

Does a bigger dev set save you?

Partly. With more traces you see each failure pattern several times, and the tail patterns show up at all, so more of your patches end up fixing patterns rather than individual traces. The memorized traces also become a smaller share of the score. Here is the gap after the full loop, as a function of dev-set size, at the fix generality you set above.

Let an AI do the climbing: a real run

Everything above is a simulation. This section is a real run, recorded once and replayed here; nothing calls a model when you press the button.

The task is the ticket classifier from the lesson: five categories (Billing, Shipping, Returns, Account, Other) for an online clothing shop. We have 160 labeled tickets. 40 are the dev set: the optimizer sees how the prompt does on them. 120 are held out: scored after every round, never shown to the optimizer. The labels come from annotators who, as in real life, disagree on about one ticket in eight.

A small model classifies the tickets with the current prompt. A larger model plays the automated prompt optimizer: it is told to maximize dev accuracy, it sees the dev tickets the prompt got wrong with their expected labels, and it rewrites the prompt. Then we classify again. Each prompt was run twice; the numbers are the average.

dev accuracy (what the optimizer climbs)held-out accuracy

What happened

Follow one rule from a dev ticket to a broken held-out ticket

Dev ticket	"I'm getting order notification emails for someone else's account! Please take my email address off it." Labeled Shipping by the annotator; the house policy says Account. A labeling slip.
Rule the optimizer wrote	"Order or shipment notification emails (including ones sent to the wrong person) are Shipping, even if they are described as a privacy concern."
Held-out ticket	"Why do I keep getting 'your order has shipped' emails when I haven't ordered anything?" Correctly labeled Account. V1 got it right. V3 now says Shipping.

The optimizer did its job: it fixed the ticket in front of it. It had no way to know that the label it was fixing toward was a slip, so it turned one annotator's mistake into a rule. Look back at the dev tickets it was shown in round 1: the ones marked annotator slip are now visible. Four of the seven were slips, and it wrote a rule for every one.

The optimizer saw it coming

In round 1, asked to describe its changes, the optimizer added a third line on its own (left out above so you could see it here):

"Caveat: the rules are fitted to these same 7 misses and some labels look like house conventions, so the gain is in-sample and could cost accuracy on unseen tickets."

It was right, and the loop continued anyway, because the loop only reads one number.

One run, 120 held-out tickets: the size of the drop is uncertain, its direction much less so. Dataset written for this lesson; labels follow a house policy, with one in eight swapped for the most plausible wrong label to mimic annotator disagreement. Classifier and optimizer were Claude models, run in October 2026.

How this differs from the other sources

Multiple hypothesis testing (#4)

You never look at the data. You try N variants, score them all on the same set, keep the best. The winner's score is inflated by selection; the inflation is noise, and it is predictable from N and the sample size.

Open-book overfitting (#3)

You do look at the data, and it changes what you build. The inflation is information leakage, not noise: it does not shrink with more evaluation runs, and it grows with every round you iterate.

Sampling bias (#2)

Your dev set does not resemble production. Open-book overfitting bites even when it does: in the simulation, dev and production are drawn from the same distribution.

Classical overfitting

The model fits the training set. Here the model never changes; the prompt does, and you are the optimizer. The usual remedies (regularization, early stopping) have no handle to grab.

What to do about it

You cannot stop looking at failures; that is how systems get better. What you can do is stop measuring on the data you looked at.

Practice	Why it works
Freeze a held-out set, touch it once	Split before you start iterating. Iterate on the dev set as hard as you like; score on the held-out set only when you are about to ship. The held-out number is the one you report.
Keep two pools: the ones you read, the ones you score	Any item you opened by hand moves into the "read" pool and leaves the scoring pool for good. Keep a log; teams routinely forget which traces they already mined.
Few-shot examples come from a third pool	An example that is in the prompt can never also be a test item. Draw examples from a pool that is never scored, and periodically check that the examples do not quietly resemble the test items.
Refresh the dev set from production	After a few rounds the dev set is exhausted: everything in it is memorized. Retire it and sample a new one. This also re-samples the long tail you have not seen.
Report two numbers	"Dev 95% (iterated on), held-out 74% (fresh)." The gap is itself a diagnostic: large gap means your fixes were trace-specific, and the next round should be spent on generalizing them, not on new misses.
Prefer fixes that name the pattern	Before patching, write down the failure pattern in one sentence and find a second trace that matches it. If you cannot, you are about to memorize a trace.