The scenario: You are evaluating a few-shot LLM classifier that routes support tickets to one of five teams (account, order, refund, payment, delivery). You can vary experimental conditions (model, prompt, temperature, dataset) and, once the numbers are in, look at them by metric (accuracy, judge score, escalation rate, latency, cost) and by slice (ticket category, channel, language).
The catch: by default every condition has exactly the same true quality. Nothing is real. And yet, every time you run the experiment, the tool finds a statistically significant story, complete with a plausible explanation. Roll a few times and a different story is "true" almost every roll.
Two multipliers of looks: each dimension you vary multiplies the number of runs and the tokens you pay for (the curse of dimensionality), and each metric × slice you inspect multiplies the number of comparisons. looks = (cells − 1) × metrics × slices. At α = 0.05, about one look in twenty crosses the line by chance, so you can predict how many "findings" you will collect before you collect them.
The distinction that matters: exploring (many looks, questions not yet precise, output = hypotheses) versus comparing (one hypothesis, decision rule fixed up front, one look). Exploring is not wrong. Shipping on an exploratory finding is. The fix is a loop: explore broadly, articulate a claim, then challenge it on fresh data with every choice fixed beforehand. Or better: pre-register the compare before looking at all.
Optional reality check: switch on a planted real effect. Exploration will find it, but it will also find the fakes, and you cannot tell which is which until you challenge them.
Every "significant" difference in this run, best p first, written up the way it would land in a readout. Improvements first (green), then regressions (orange). Click a story to examine it, or challenge them all at once.
Every dimension multiplies the number of runs, and every run costs tokens: the grid runs every combination of the conditions you vary. The number of looks is what generates stories: each extra cell is compared with the baseline on every metric and every slice.
For your grid, as n goes from 50 to 600 tickets per cell: the expected number of chance findings (blue, left axis) stays flat at 5% of comparisons, while the smallest accuracy difference that looks "significant" (orange, right axis: on all tickets; dashed: on a single category slice) shrinks with √n. The dot marks your current n.
Each bar is one cell vs the baseline cell on the chosen metric within the chosen slice. Green: significantly better. Orange: significantly worse. Error bars: 95% CI of the difference.
A silent simulation of your grid in which nothing is real: every cell has identical true quality. These are six of the differences that still came out "significant", written up the way they would appear in a readout. Change the grid or reshuffle and a different set appears.
Tokens, not calls, are the cost: variant 4 (a step-by-step prompt) returns ~15× the output tokens of variant 2 (a concise one), and Banking tickets are almost twice as long as Retail ones.
Write down the hypothesis and the decision rule before the data exists (uses the conditions, n and reality set on the Explore tab). You get exactly one look. If the target is met, that is evidence; a target chosen afterwards is a description.