Estimating Accuracy (Beta distribution calculator)
How to read and interact with this page
- Enter what you observed. N examples in the test set and S of them correct, or click a preset. That is all the page knows.
- Ask a belief question. Pick “above T”, “below T” or “between T1 and T2” and set the threshold.
- Read the curve. It is the Beta posterior: your belief about the true accuracy after seeing the data. The shaded area is the answer to your question; the bar under the curve is the 95% credible interval.
- Grow N. Click 4 / 5, then 80 / 100, then 8,000 / 10,000, and watch the curve sharpen and the interval narrow.
- Change the prior. The first tile shows what you believed before the data, uniform by default. Switch to Custom Beta(8, 2) on 4 / 5 and watch the answer move; then do the same on 8,000 / 10,000 and watch it not move.
Prior belief
Every accuracy equally plausible.
Experiment results
Question
—
Posterior distribution
The Beta posterior
Prior × data = posterior Beta(5, 2). The curve is our belief about the true accuracy.
—95% credible interval (equal-tailed)
“There is a 95% probability the true accuracy is in here”, given the prior. This is a statement about the accuracy, which is what your boss is really asking. The prior matters when N is small; try Custom Beta(8, 2) on 4 / 5, then on 8,000 / 10,000.