Session 04

The Science of Experimentation / II

4 hours · Week 4 · A practical guide with running examples from conversational clustering and knowledge base gap detection

Learning Objectives

Use the Beta distribution to express uncertainty about probabilities
Understand why the sum of many IID variables is approximately Normal (CLT)
Use bootstrapping to estimate uncertainty without assumptions about the distribution
Understand why experiment design matters and how experiments serve as decision tools
Convert research questions into specific, falsifiable hypotheses

Part 1: Beta Distribution

Beta distribution: the “distribution of probabilities.” Conjugate prior for Bernoulli/Binomial — observe data, update by addition: Beta(a,b) + h successes, t failures → Beta(a+h, b+t).

Materials

Course Reader: Beta

Part 2: Adding Independent Random Variables

Adding independent RVs: sums stay in the same family (Normal+Normal → Normal, etc.).

Materials

Lab: Convolution of Independent Random Variables
Course Reader: Adding RVs · Sampling

Part 3: Central Limit Theorem

Central Limit Theorem: the average of n IID samples → Normal(μ, σ²/n) for any distribution. Foundation for confidence intervals, standard error, and bootstrapping.

Materials

Course Reader: CLT

Part 4: Bootstrapping — Statistics for the Unconscious Engineer

The bootstrap is a simple but powerful idea: use your data as if it were the population, resample from it with replacement, and measure how much your statistic varies across resamples. No formulas. No distributional assumptions. Just repeated sampling.

Bootstrap Confidence Interval (in 4 lines of Python)

resamples = [np.mean(np.random.choice(data, size=len(data), replace=True))
             for _ in range(10_000)]
ci_low  = np.percentile(resamples, 2.5)
ci_high = np.percentile(resamples, 97.5)

Why it works: By resampling your observed data, you simulate what would happen if you collected many different samples from the same source. The spread of your statistic across resamples estimates its real-world variability.

When to use it:

Your metric has no closed-form standard error (e.g., F1, AUC, median)
Your sample is small or the distribution is skewed
You want to compare two systems: bootstrap the difference
You want a CI without memorizing any formula

The bootstrap doesn’t require you to know the distribution of your data. It requires only that your sample is representative. If your sample is biased, the bootstrap faithfully estimates a biased population—garbage in, garbage out still applies.

Materials

Lab: Bootstrap Playground
Course Reader: Bootstrapping