Sources #1–9 assumed a perfect judge. You asked "is this response correct?" and got the right answer. In reality, the judge is itself an imperfect system. Every uncertainty source applies again at the judge level.
Compounding: Judge uncertainty doesn't just add to other uncertainties — it multiplies. A noisy judge + sampling noise = much wider CIs than either alone. A biased judge + biased sample = errors that reinforce each other.
Part 1 — Judge Noise: Same judge, same input, different scores each time. See how judge noise widens your confidence intervals.
Part 2 — Judge Bias: A generous vs harsh judge. Same outputs, different scorecards.
Part 3 — Compounding: Stack multiple sources together and see the total uncertainty explode.
The judge scores each item on 1-5, but with some noise. Higher noise = the same item gets different scores each time.
σ=0: perfect judge. σ=1: each item's score can shift ±1 randomly. σ=1.5: very noisy judge.
Run Simulation — Runs a single evaluation using the settings of the currently active tab. On the Judge Noise tab: scores n items with a perfect vs noisy judge and shows the score difference. On Judge Bias: compares a generous vs harsh judge on the same outputs. On Compounding: evaluates with all toggled uncertainty sources active.
Run 100× — Repeats the evaluation 100 times and shows the full distribution of results. On Judge Noise: two histograms (perfect vs noisy) revealing how noise widens the spread. On Judge Bias: two histograms centered on different means. On Compounding: a single histogram showing the total CI — toggle sources on/off and re-run to see the CI explode as you stack uncertainties.
Tabs: Judge Noise = same judge, random variation per item. Judge Bias = generous vs harsh judge on same outputs. Compounding = stack multiple sources (sampling noise, judge noise, judge bias, sample bias) and watch the total uncertainty multiply.