Watch the bell curve emerge as you increase the number of coin flips. A visual introduction to the Central Limit Theorem and convolution of independent random variables.
CLT interactiveHands-on bootstrapping: resample from your data, build sampling distributions, compute confidence intervals, and test whether two samples differ. The foundation for many of the uncertainty fixes above.
bootstrap interactiveExplore common probability distributions interactively. Adjust parameters and see how the shape changes.
distributions interactiveRun the same evaluation twice — get different numbers. Set true scores, pick a sample size, and watch the scorecard wobble. Then tame it with beta posteriors, bootstrap CIs, and confidence intervals.
interactive more data helpsEnter successes out of n trials and get a posterior over accuracy: point estimate, mode, and 95% credible interval. Powered by the Beta distribution with selectable priors.
interactive beta posterior #2Your test set is 70% English but production is 40%. The scorecard looks great — until you see what users actually experience. Explore biased vs. representative sampling with adjustable language mixes.
interactive more data doesn't help #4You test 10 prompt variants. The winner scores 4.3. Ship it? All variants had the same true score — the "winner" was just the luckiest. See how selection bias grows with N.
interactive selection bias #5Aggregate says 4.0. But Client A sees 4.5 and Client B sees 3.0. The aggregate hides the story. Includes a Simpson's Paradox demo where the "winning" system actually loses everywhere.
interactive property of the world #6Mean says A wins. Threshold rate says B wins. Worst-case says B wins clearly. The winner changes with every tab click. The data didn't change — the question did.
interactive design decision #7Your January eval says nothing about April. Simulate 12 months of gradual degradation and sudden shocks. See how long it takes to notice without continuous monitoring.
interactive fresh data over time #9GPT-4: 97.6% → 2.4% in 3 months. Same prompt. Same task. The model changed under you. Real documented cases + simulated version timeline.
interactive version discipline ✶Every source applies again at the judge level. See how judge noise widens CIs, judge bias shifts scores, and stacking uncertainties makes measurements unreliable.
interactive cross-cuttingIterate on a judge prompt: watch dev score climb across 4 versions, then see the held-out gap. The gut punch. Uses synthetic customer support data + real OpenAI API calls.
python notebook process discipline #87 "semantically equivalent" prompt variants — reorder examples, change delimiters, rephrase rubric. Same data, same model. Watch scores swing wildly.
python notebook robustness testing