Designing Experiments and Mastering Uncertainty — Part II
Estimators, the Central Limit Theorem, and bootstrapping
"Any measurement, without knowledge of the uncertainty, is meaningless." — Walter Lewin, MIT
"Those who are able to represent what they do not know make better decisions." — Chris Piech, CS109 class at Stanford
The Series (continued)
4 — Random Variables for Random Variables
Distributions of estimators. The Beta distribution as a tool for modeling uncertainty about probabilities. How to assess an AI system and reason about the uncertainty in your measurements. Updating beliefs with data and credible intervals.
5 — Adding Random Variables
Combining independent random variables. The Central Limit Theorem and why sums of random variables tend toward the normal distribution. Practical consequences for AI evaluation.
6 — Bootstrapping
Bootstrapping methods for practical uncertainty estimation. Building sampling distributions from a single dataset, computing confidence intervals, and comparing systems without parametric assumptions.
Interactive Tools
Convolution Playground
Interactive visualization of the convolution of independent random variables. Explore how sums of random variables give rise to new distributions.
Bootstrap Lab
Hands-on bootstrapping playground: resample, build sampling distributions, and reason about uncertainty from a single dataset.
Course Reader
The reference material for this part is the open textbook Probability for Computer Scientists by Chris Piech (Stanford CS109) — Part 4 up to "Bootstrapping" (included).
← Back to Part I
Probability, random variables, and Bayesian reasoning