Coins, dice, roulette, poker — compute any probability just by simulating. Three demos: simulate an event, a ratings distribution where whole-experiment statistics become certainties, and estimating claims from one sample (Beta, closed form, bootstrap).
interactive probability by simulationEnter how often an event occurred and get a posterior over the true rate: point estimate, 95% credible interval, and P(rate > T). Powered by the Beta distribution with selectable priors.
interactive beta posteriorEnter S correct out of N and see the three numbers people draw as “error bars”, as values and as bars on one axis. Grow N and watch the SD refuse to shrink while the SE and the 95% CI do.
interactive sampling4 out of 5 correct. Or 8,000 out of 10,000. Enter N and S, ask “how likely is accuracy above 90%?”, and read the answer off the Beta posterior: the curve that is your belief about the true accuracy.
interactive bayesPick a distribution or paste your own sample, then ask any question — P(X > T), P(mean > T), P(median > T) — and get the answer plus a 95% interval for how sure you can be. No formulas.
interactive bootstrapThe five systems the course evaluates repeatedly — from a deterministic two-input function to a multi-stage support pipeline and an LLM summarizer with no single right answer.
course examplesA trained ML classifier over real datasets. Set a test-set size, spin the wheel, and watch accuracy, the confusion matrix, and a Beta credible interval move. Includes a companion tab for noise on multiple metrics.
interactive sampling noiseYour test set is 70% English but production is 40%. The scorecard looks great — until you see what users actually experience. Explore biased vs. representative sampling with adjustable mixes.
interactive more data doesn’t helpPull apart the three things that move a measurement — a systematic offset, spread, and random noise. Adjust the distributions and watch the shape, the mean, and the uncertainty change.
interactive distributionsThe LLM as the thing that judges your AI's answers. An LLM that classifies (real predictions, confusion matrix, the lean that is bias), then the scoring judge: turn the bias and noise dials and watch noise widen the interval while bias slides it off the truth.
interactive judge noise & biasTen HR-assistant answers, five judge prompts, three models, five repeats each: 750 real judgments against a fixed expert verdict. See which judges miss a fabricated number, which reward padding, and how much the same judge disagrees with itself.
interactive judge of judgesNothing is real, yet every run finds a “significant” story with a plausible explanation. Roll again and a different story is true almost every time. See how the grid of conditions and the freedom to pick metric and slice multiply your comparisons, then explore → claim → challenge on fresh data, and compare with a pre-registered test.
interactive pre-registrationYou try 10 classifiers on the same test set and keep the best. Its score is inflated — by how much? All ten had the same true quality; the “winner” was just the luckiest. See the inflation grow with the number of things you try, and shrink back on fresh data.
interactive selection biasIterate on a prompt against a dev set and watch the score climb across versions — then check the held-out set and see the gap. The difference between fitting the data and fitting the world.
interactive process disciplineAggregate says 4.0, but Client A sees 4.5 and Client B sees 3.0 — the aggregate hides the story. Includes a Simpson’s Paradox demo where the “winning” system actually loses everywhere.
interactive property of the worldMean says A wins. Threshold rate says B wins. Worst-case says B wins clearly. The winner changes with every tab click — the data didn’t change, the question did.
interactive design decisionGPT-4: 97.6% → 2.4% in 3 months. Same prompt, same task — the model changed under you. Real documented cases plus a simulated version timeline.
interactive version disciplineYour January eval says nothing about April. Simulate 12 months of gradual degradation and sudden shocks, and see how long it takes to notice without continuous monitoring.
interactive fresh data over timeSee how unrepresentative data, a biased judge, prompt selection and test-set reuse affect the reported score. Compare individual fixes with a fresh, representative test set and checked labels.
interactive combined evaluation errorsEvery source applies again at the judge level. See how judge noise widens CIs, judge bias shifts scores, and stacking uncertainties makes a measurement unreliable.
interactive cross-cuttingSubmit an eval write-up and check whether the evidence supports the decision. The capstone tool that turns a measured result into a ship, canary, or iterate recommendation.
interactive capstoneWatch the bell curve emerge as you add coin flips — a visual introduction to the Central Limit Theorem and the convolution of independent random variables.
interactive CLTHands-on bootstrapping: resample your data, build sampling distributions, compute confidence intervals, and test whether two samples differ. The engine behind many of the uncertainty fixes.
interactive bootstrap