Bootstrap Hypothesis Testing Lab

Generate → Permute → Decide. Play with sample sizes, distributions, and test statistics.

The Null Hypothesis (same for all tests)

H₀: Both samples come from the same distribution.
If this is true, the labels “Sample A” and “Sample B” are meaningless — every data point could equally have ended up in either group.

The Permutation Bootstrap Procedure

  1. Compute the observed test statistic (e.g. |mean(A) − mean(B)|) on the real data.
  2. Pool all data points together — forget which group they came from.
  3. Shuffle the pool randomly and split into two fake groups of the original sizes nA and nB.
  4. Compute the test statistic on these fake groups.
  5. Repeat steps 3–4 thousands of times to build the null distribution — what the statistic looks like when group membership truly doesn’t matter.
  6. p-value = fraction of shuffled statistics ≥ the observed statistic.

If the observed statistic lands far in the tail of the null distribution (small p-value), we reject H₀.

Bootstrap vs. Welch’s t-test — side by side

Bootstrap permutation Welch’s t-test
Assumptions None — works for any distribution, any statistic Approximately normal populations (relaxed for large n by CLT)
How null is built Empirically, by shuffling real data Analytically, using the t-distribution
What it tests Whatever statistic you plug in (means, medians, variance, KS…) Difference in means only
When they agree When data is roughly normal and n is moderate-to-large, p-values are very similar
When they diverge Skewed data, heavy tails, small n, or non-mean statistics — bootstrap is more reliable

This tool runs both on the difference-in-means test so you can compare. Try Exponential or Log-Normal with n=10 to see divergence.

Step 1 — Generate Data

Sample A

Sample B

Step 1: Configure distributions and hit Generate Samples
Step 2: Inspect your data, then hit Run Bootstrap Test