Why your scorecard lies when n is small
You’ve built an AI system (a chatbot, a summarizer, a classifier). To evaluate it, you run it on n test queries and have an LLM judge score each output from 1 to 5 on four metrics: Correctness, Helpfulness, Tone, and Completeness. You average the scores and get a scorecard. That scorecard is your basis for shipping (or not).
The problem: that scorecard is one random sample. Run the same evaluation again with a different set of n queries from the same population, and you’ll get different numbers. At small n, the numbers can be wildly different — different enough to change your decision.
A single evaluation gives you a point estimate, not the truth. The truth is a distribution of possible measurements. Without a confidence interval, your scorecard is meaningless — you don’t know if “4.1” means “somewhere between 3.5 and 4.7” (useless) or “between 4.0 and 4.2” (actionable).
| Method | How it works | When to use |
|---|---|---|
| Confidence interval | mean ± z · (std / √n). Assumes approximate normality (works well for n > 30 by CLT). | Quick sanity check. Always report this alongside any point estimate. |
| Beta posterior | Binarize scores (≥4 = “good”). Count successes k out of n. Posterior is Beta(k+1, n−k+1). The full curve shows your uncertainty about the “good rate.” | When you care about a pass/fail threshold (e.g., “what fraction of outputs are good enough?”). Exact — no normality assumption needed. |
| Bootstrap | Resample your n scores with replacement 5,000 times. Each resample gives a different mean. The spread of these means is your uncertainty. | Works for any statistic (mean, median, percentile) and any distribution. No assumptions. The workhorse method. |
Yes — this is the one source of uncertainty where more data is a clean fix. The standard error of the mean shrinks as 1/√n. Double n, uncertainty shrinks by ~30%. Quadruple n, it halves. The beta posterior tightens, the bootstrap distribution narrows. Try it: drag n from 20 to 200 and watch the uncertainty collapse.
But beware: this only fixes sampling noise. If your sample is biased, your judge is noisy, or your prompt is brittle, more data gives you a very precise wrong answer. Sampling noise is source #1 of 9 — the easy one.
Set the true average score and spread for each metric. Spread controls how much individual scores vary around the mean: high spread means some outputs score 2 and others score 5, even though the average is 3.8. Low spread means most outputs score close to the mean.
Run Evaluation — Draws n random samples from the true score distribution and computes the average for each of the 4 metrics. You'll see scorecard bars, a histogram of sampled scores, and 95% confidence intervals. Each click re-randomizes — the scores jitter around the truth, and that jitter IS the lesson.
Run 100 Evaluations — Repeats the evaluation 100 times and plots the sampling distribution of the mean. Shows how the average itself is a random variable, with a bell-shaped spread around the true score. The CI overlay shows the 95% range across runs.