Bootstrap Hypothesis Testing Lab

The Null Hypothesis (same for all tests)

H₀: Both samples come from the same distribution.
If this is true, the labels “Sample A” and “Sample B” are meaningless — every data point could equally have ended up in either group.

The Permutation Bootstrap Procedure

1. Compute the observed test statistic (e.g. |mean(A) − mean(B)|) on the real data.
2. Pool all data points together — forget which group they came from.
3. Shuffle the pool randomly and split into two fake groups of the original sizes n_A and n_B.
4. Compute the test statistic on these fake groups.
5. Repeat steps 3–4 thousands of times to build the null distribution — what the statistic looks like when group membership truly doesn’t matter.
6. p-value = fraction of shuffled statistics ≥ the observed statistic.

If the observed statistic lands far in the tail of the null distribution (small p-value), we reject H₀.

Bootstrap vs. Welch’s t-test — side by side

This tool runs both on the difference-in-means test so you can compare. Try Exponential or Log-Normal with n=10 to see divergence.

The setup

You compare two prompt variants on the same 10-example eval set. Each example is scored 0/1 by the same judge.

example i :   1  2  3  4  5  6  7  8  9 10
Variant A :   1  1  0  1  1  0  1  1  1  0     mean = 0.70
Variant B :   0  1  1  1  1  0  0  1  1  0     mean = 0.60
diff  d_i :  +1  0 -1  0  0  0 +1  0  0  0     mean = 0.10

The wrong move: two separate CIs and "do they overlap?"

Compute a bootstrap CI for mean(A) and a separate one for mean(B); if they overlap, declare "no difference." This throws away the pairing structure. Example 3 is hard for A and easy for B; example 7 is the reverse. Treating the two samples as independent inflates the noise you compare against, because you're letting "which examples happened to be in sample A" add variance that the paired data does not actually have. Two overlapping per-variant CIs can coexist with a CI on the difference that excludes zero — so the overlap rule misses real regressions.

The right move: bootstrap the per-example differences

1. For each example i, compute d_i = A_i − B_i. (In our table: [+1, 0, −1, 0, 0, 0, +1, 0, 0, 0], observed mean 0.10.)
2. Resample indices i₀, i₁, …, i₉ from {1, …, 10} with replacement — one list of indices, used for both variants. Equivalently: resample the vector of d_i values directly.
3. Compute the mean of the resampled differences d̄*.
4. Repeat B = 10,000 times to get the bootstrap distribution of d̄*.
5. Report the observed d̄ and the 2.5% / 97.5% percentiles of the bootstrap distribution as a 95% CI on the mean difference.
6. Decide: if the CI lies entirely above 0, A is better; entirely below 0, B is better; straddles 0, you can't tell from this data.

Pseudocode

d = [a - b for a, b in zip(A, B)]        # per-example differences
boot_means = []
for _ in range(10_000):
    idx = [random.randrange(len(d)) for _ in range(len(d))]   # resample indices
    boot_means.append(mean(d[i] for i in idx))
ci_low, ci_high = percentile(boot_means, 2.5), percentile(boot_means, 97.5)

Why resampling at the example level matters

The pairing is the signal. Example difficulty is a nuisance variable: hard prompts score low under both variants, easy prompts score high under both. When we resample indices jointly — so A and B are always evaluated on the same bootstrapped examples — that shared difficulty cancels inside each d_i and never enters the variance of d̄*. If you bootstrap A and B independently, you reintroduce that variance and your CI on the difference is wider than it should be, so real improvements fail to show up as significant.

What the observed data says here

The observed mean difference is +0.10 (A − B). With only 10 paired examples and 8 ties, a paired bootstrap CI on this difference will straddle 0 — you don't yet have enough evidence to call A better. The right response is not "ship A" (or "overlap ⇒ ship neither") but collect more paired examples until the CI on d̄ moves off zero in a direction you can defend.