The scenario: you are building a few-shot LLM classifier that routes support tickets to one of five teams. You write N prompt variants, score every one of them on the same test set of n tickets on one metric of interest, accuracy, and keep the variant with the best score. That is what everybody does, and it is the right way to pick. The trap is in what you then report.
The catch: every score is the true accuracy plus sampling noise. If you measure one variant, its score is unbiased. If you measure N and keep the highest, you kept the one whose noise happened to be the most positive. The winner's score is inflated, and the inflation grows with N and shrinks only with √n. You never looked at the test data, you did nothing wrong, and you are still fooled. This is the Select goal from the lesson, and its bias is called the winner's curse.
How much? If the per-ticket noise has standard deviation σ, the winner sits about (σ/√n) × √(2 ln N) above the truth on average (the expected maximum of N normal draws). With accuracy near 82%, n = 200 and N = 10, that is about +4.2 points.
The fix: select on one set of tickets, then report the winner's score from a different one, tickets it has never seen. If you have the per-ticket results, you can also check directly whether the winner beats the baseline once the selection is accounted for (second tab).
The leaderboard: measured accuracy of each variant (bars, sorted; thin bars: 95% credible interval from a Beta posterior) against its true accuracy (yellow diamonds). Green: the one we pick. Variants are named by their true rank, so V1 is the one we should pick.
How many of the 300 experiments picked each variant. V1 (yellow) is the one we should pick every time.
For each of the 300 experiments: the accuracy we estimated for the variant we picked, minus that variant's true accuracy. The black line (and the grey bar) is zero: an estimate within a quarter point of the truth. The bars sit almost entirely to the right of it.
First row: the names, baseline first. Then one row per ticket, one column per system: 1 = right, 0 = wrong (or any score, higher = better). Every system must be scored on the same tickets. Comma, tab or space separated.
The examples simulate 400 tickets, a baseline at 82% and 8 variants scored on the same tickets. Assumes these are all the variants you tried; survivors of a larger search carry more inflation than the data can show.
This is a bootstrap: it uses only your tickets, assumes nothing about how scores are distributed, and keeps the pairing. The "movement" in run step 2 is the same wobble a single-system bootstrap interval measures; the difference is recording the largest of N instead of the wobble of one. Statisticians call it the max-t method, a bootstrap version of Dunnett's test.