Point estimate + credible interval via the Beta distribution
You ran your model on n test cases and it got k correct. The empirical accuracy is k/n, but that's a point estimate: it says nothing about how confident you should be. If k=17, n=20, your accuracy is 85%, but the true accuracy could easily be 70% or 95% — you don't have enough data to tell.
The Beta distribution gives you the full story: not just a single number, but a distribution over plausible values of the true accuracy.
Each test case is a Bernoulli trial (correct / incorrect). The Beta distribution is the conjugate prior for Bernoulli: if you start with a Beta prior and observe binary data, the posterior is also Beta. The math is exact — no approximation, no normality assumption.
Start with prior Beta(α₀, β₀). Observe a occurrences and b non-occurrences. Posterior is:
Beta(α₀ + a, β₀ + b)
That's it. The prior encodes what you believed before; the data updates it.
Beta(1, 1) — uniform. Says "the true accuracy is equally likely to be anywhere in [0,1]." Reasonable default when you have no prior info.
Beta(0.5, 0.5) — Jeffreys prior. Slightly favors extremes (0 and 1). Often used in frequentist-leaning settings.
Beta(α, β) with α + β large — informative. Encodes prior belief with strength proportional to α + β. E.g. Beta(40, 10) means "I believe accuracy is around 80%, with the confidence of 50 prior observations."
Posterior mean = (α₀ + a) / (α₀ + β₀ + n). Your best single-number estimate.
Mode (MAP) = (α₀ + a − 1) / (α₀ + β₀ + n − 2). The peak of the posterior — "most likely value."
95% credible interval. The range in which the true accuracy lies with 95% probability, given the data and prior. Unlike a frequentist CI, this interpretation is the natural one.