The scenario: You design N prompt variants for a customer support bot. Each variant was crafted from different design principles — you never looked at the test data. You evaluate all N on the same test set (measuring Correctness) and pick the winner.
The question: If you measured just one variant, your estimate of its Correctness would be unbiased — noisy, but centered on the truth. But when you test N and pick the best, the winner's score is biased upward. How much? The Gumbel distribution tells us exactly.
Key distinction from overfitting (#3): In overfitting, you see the data and it shapes your system (information leakage). Here, you never see the data — but selecting best-of-N on the same dataset creates selection bias.
Each variant's measured score is a random variable: the true score plus noise ≈ Normal(0, σ/√n), where σ is the per-item standard deviation and n is the test set size. When you pick the maximum of N such i.i.d. normals, the distribution of that maximum follows a Gumbel distribution (the Type I extreme value distribution).
The expected inflation — how much the best-of-N is above the true score — is:
Compare to a single variant: E[single] − true = 0. The gap between "one measurement" and "best of N measurements" is exactly this Gumbel offset.
Intuition: Ask one person to guess your weight — their guess is unbiased, just noisy. Ask 50 people and keep the closest guess — that guess is biased toward being "too accurate." The more people guess, the luckier the best guesser was.
All variants have the same true quality. Any differences in measured scores come purely from sampling noise.
All N variants have this same true score. The spread comes from sampling noise only.
1. Run Tournament — Creates N variants (all with the same true Correctness) and evaluates each on a test set of n queries. You'll see a bar chart of variant scores and a leaderboard. The winner looks convincingly better — but it's noise.
2. Reveal Truth — Re-evaluates the tournament winner on a fresh test set. The inflated score collapses back toward the true score. Also runs 50 tournaments to show winner instability — a different variant "wins" almost every time.
3. Single vs Best-of-N — Compares the distribution of single-variant estimates (unbiased) against best-of-N estimates (biased up), and checks whether the gap matches the Gumbel prediction (σ/√n)·√(2 ln N).