LLM as a Judge
L3 lab · an LLM can be the thing that judges your AI's answers — classifying them or scoring them. In L1 the judge was perfect; a real judge has two defects, bias and noise.
An LLM as a classifier
An LLM can classify, exactly like the trained models in L1 — read a piece of text and emit one
label (POSITIVE / NEGATIVE, or one of several categories). No training: a prompt with
the allowed classes and a few labelled examples. This demo moved here from the L1 lab — same datasets, real
precomputed LLM predictions; spin freely, no API calls.
Because we hold the true labels, we can measure the LLM exactly like L1 measured the trained model: accuracy, per-class precision/recall, the confusion matrix. And the confusion matrix shows the direction of the misses — the bias — visible only because we have ground truth.
Gen-AI classifier on the L1 datasets
How the LLM is prompted (few-shot)
Browse this draw's examples
| Text | True label | LLM's label |
|---|
The judge gives a number — a scorecard
Gen-AI answers are rarely just right or wrong. So a scoring judge rates quality on a scale, usually across several dimensions — a scorecard. Here an AI summarizes a real court case, and a second LLM scores that summary against an expert reference on three dimensions (1–5):
Accuracy · Completeness · Fluency → Overall
| Dimension | What it asks the judge | Example |
|---|---|---|
| Accuracy | are the stated facts correct — no invented or contradicted claims? | 3 |
| Completeness | does it capture the parties, the claims, the outcome? | 4 |
| Fluency | is it clear, well-formed, professional prose? | 2 |
| Overall | computed, not judged: 0.45·A + 0.45·C + 0.10·F, rounded to the nearest ½ | 3.5 |
Overall is a fixed rule, not a fourth judgment. 0.45·3 + 0.45·4 + 0.10·2 = 3.35 → 3.5. Substance (accuracy, completeness) dominates; fluency barely moves it — for a capable model fluency is near-constant anyway. This is the example from slide 11 of the deck.
The two prompts
One LLM call summarizes the case; a second scores that summary against the expert reference. Both are plain prompts — no fine-tuning. These are the actual Jinja templates the build script runs.
1 · Summarizer prompt
2 · Judge prompt (reference-based scorecard)
Honest caveats baked into this dataset: Overall is computed (above),
Fluency is ≈ 5 for nearly every case (a capable model writes clean prose), and the summarizer and the judge
are the same model (gpt-5.5) — a setup prone to self-preference.
Browse the scored cases
Real output from Multi-LexSum, each case under 20k words so the summarizer reads the whole record — no truncation. We keep the 100 most distinctive of 143 scored civil-rights cases — those whose accuracy/completeness sit furthest from the average — and browse them most distinctive first, so you start with the telling ones instead of a run of 4/5s. For each case: the AI summary, the expert reference it was scored against, and the judge's scorecard.
One number — but which cases? Sample noise
You cannot score all the population. You score a sample and report its mean. Draw a different sample and the mean moves — that is sample noise, exactly L1's, now on a 1–5 score. The thin teal line marks the full-set mean (all 100); the amber bar is this draw's mean, per dimension.
The judge's two defects: bias and noise
The scorecard came from one real judge. To see how a judge can be wrong, switch to a frozen pool of answers whose true quality we get to peek at (you never can in real life; the pool's true mean is 3.8 / 5). Draw a sample, let the judge score it, and read the per-answer error two ways: the mean error (the systematic part — the bias) and the RMSE (the typical total miss — bias and noise). The bars below focus on the gap between them: that gap is the noise.
Judge dials
Same judge outputs; only the reporting scale changes. (RMSE below stays on the 1–5 rating scale, where the dials live.)
RMSE — how bias and noise combine (on the 1–5 rating scale, where the dials live)
predicted RMSE = √(bias² + noise²) = –
measured RMSE over the sample: –
Bias and noise add in quadrature: RMSE² = mean error² + noise². So the RMSE bar is never shorter than the |mean error| bar — the extra height is the noise. Slide bias → both bars rise together; raise noise → only RMSE grows (noise averages out of the mean error). The measured RMSE can differ from the predicted √(bias²+noise²) because it's one finite re-roll and ratings clamp at 1–5.
What the judge did to each answer
A sample of the pool. The grey bar is each answer's true quality; the amber bar is the judge's rating. Bias slides every amber bar the same way; noise jitters each one independently — which is why noise mostly cancels in the mean (except where ratings clamp at the 1–5 edges) while bias accumulates.
| Question | AI answer | True quality | Judge's score |
|---|
Three sources compound: sampling noise, judge noise, and bias
Now make it realistic. A real judge returns an integer 1–5 score, and three things move the number you finally report: the judge is noisy (it sometimes flips a score up or down); the judge may be biased (it systematically bumps a fraction of scores in one direction); and you only score a finite sample. The three charts below pull these apart — the scores the judge handed out, the per-answer error (its offset from zero is the bias, its width is the noise), and the bootstrap interval for the mean (where the true average really sits). Watch how bias shifts the error chart off zero — and how the 1–5 ceiling quietly trims it.
Judge dials
Bias is a probability: at +0.30 the judge bumps 30% of scores up by one point. Noise fires random ±1 nudges on each score — the higher you dial, the more of them land, so the scores scramble fast. Both cap at the 1–5 edges.
Same judge outputs; only the reporting scale changes.
1 · Distribution of the scores — the integer 1–5 scores the judge handed out
–
2 · Error on this reporting scale — map(score) − map(true) per answer; the error itself changes when the scale does
–
3 · Bootstrap distribution of the mean — resample the observed scores thousands of times; where could the true mean be?
–
–
Bootstrapping the means
Same noisy, biased judge on a finite sample — but here we focus on one question: given the scores you observed, where could the true mean be? Resample those scores with replacement thousands of times, take the mean each time, and read the spread. Turn up noise or shrink the test set and the interval widens; slide bias and the whole interval slides off the truth — and the bootstrap can't see it.
Judge dials
Bias is a probability: at +0.30 the judge bumps 30% of scores up by one point. Noise fires random ±1 nudges on each score — the higher you dial, the more of them land, so the scores scramble fast. Both cap at the 1–5 edges.
Same judge outputs; only the reporting scale changes.
Bootstrap distribution of the mean — resample the observed scores thousands of times; where could the true mean be?
–
–
–
Judge playbook
The simulator showed the failure modes. In a real eval, use a small trusted slice to measure the judge, then improve the rubric and measure again. This is the practical loop: audit → revise → re-audit → report the remaining uncertainty.
Measure a judge
Before trusting a judge on the full test set, audit it on examples where you have a trusted reference: human labels, gold answers, or an expert score. Keep this slice separate from the final eval if you use it to tune the rubric.
A minimal judge audit
| Check | How to measure it | What it tells you |
|---|---|---|
| Trusted slice | Label or score 50-200 representative examples with humans or references. | The yardstick. Without it, bias is invisible. |
| Bias | For scores: mean(judge - trusted). For labels: confusion matrix by true class. | Whether the judge is systematically generous, harsh, or tilted toward a label. |
| Noise | Re-run the same inputs K times; record score std, label flip rate, or within-1 agreement. | How much the judge wobbles when nothing else changed. |
| Agreement | Labels: accuracy, F1, Cohen's kappa. Scores: MAE, correlation, within +/-1 point. | Whether the judge tracks the trusted scorer closely enough for the decision. |
| Segments | Repeat the above by class, length, language, difficulty, model source, and safety-critical cases. | Where a good aggregate hides a bad judge. |
| Calibration | Fit a simple correction on part of the trusted slice, then test it on a held-out part. | Whether raw judge scores can be mapped onto the trusted scale. |
Calibration does not make the judge objective. It tells you how the judge lines up with a trusted slice, and where it still fails.
Build a better judge
A good judge is a measurement instrument. Treat the rubric, prompt, examples, output schema, and model choice as versioned parts of that instrument.
Rubric design
| Define the construct | Name what quality means for this task; do not mix unrelated goals in one score. |
| Anchor the scale | Say what 1, 3, and 5 mean; include short examples at each level. |
| Separate dimensions | Score correctness, completeness, safety, and fluency separately when they can disagree. |
| Say what not to reward | For example: do not reward length, confidence, formatting, or model identity. |
Prompt and workflow
| Use structured output | Return JSON: score, evidence, failure mode, confidence, and optional abstain. |
| Require evidence | Ask for the criterion or short quote that justifies the score. |
| Blind the judge | Hide model identity; randomize A/B order for comparisons. |
| Version and re-audit | Every prompt/model/rubric change creates a new judge that needs a new calibration check. |
Worked example
Suppose you judge support replies on a 1-5 scale. You build a trusted slice of 120 replies scored by humans, then run the LLM judge on the same replies.
Audit readout
| Readout | Result | Decision |
|---|---|---|
| Human mean | 3.8 | The target scale. |
| Raw judge mean | 4.2 | Bias = +0.4 points; the judge is generous. |
| Mean absolute error | 0.6 | Typical miss is a little over half a point. |
| Within +/-1 point | 88% | Acceptable for aggregate tracking, weak for per-item decisions. |
| Rerun std | 0.25 | Noise is smaller than bias; fix the rubric first. |
| Length correlation | +0.35 | Verbosity bias: long replies get rewarded. |
Next move: add "do not reward length," add anchor examples for terse but correct replies, then re-audit
on a held-out trusted slice. If needed, report both raw and calibrated scores, for example
calibrated = 0.82 * raw + 0.37.
Example rubric skeleton
You are judging whether a support reply resolves the customer's need.
Score 1-5:
1 = wrong, unsafe, or unrelated
3 = partially helpful but incomplete
5 = fully resolves the issue, accurate, concise
Do not reward length. Reward only correctness, completeness, and usefulness.
If the answer lacks enough information, say so.
Return JSON:
{
"score": 1-5,
"evidence": "short criterion or quote used",
"failure_mode": "none|wrong|incomplete|unsafe|unclear",
"confidence": "low|medium|high"
}
Use this repo
The code path. The lab notebook reproduces these numbers on real LLM judges — three rubrics over
the same answers, re-run variance at temperature>0, and the bootstrap on your own scored answers.
Prompts. The prompts/ folder contains strict, lenient, and 1-5
score rubrics you can compare and modify.
Bootstrap calculator. Paste your own scores (0-100) and it reports the 95% credible interval for the true mean and the chance it clears 80 / 90 / 95 / 99 — the same bootstrap as chart 3, on your data.
The lesson. This lab belongs to Lesson 9 — the full lesson page has the video, what we cover, and the skills you unlock.
Next judge audit. Lesson 6 picks this thread back up: verbosity bias, position bias, and judge-vs-human calibration.
The L3 takeaway
- Bias and noise are different defects. Noise is random and mostly averages out; bias is systematic and persists. (One catch from the compounds tab: on a bounded, top-heavy scale like 1–5, heavy noise hitting the ceiling can itself add a small downward bias.) Telling them apart is the first move.
- Both widen your real uncertainty — differently. Noise widens the interval you can measure. Bias adds an offset your interval can't even see, so the truth may lie outside a confident-looking CI.
- They call for different fixes. Reduce noise by averaging repeats, lowering temperature, and writing a crisper rubric. Reduce bias only by auditing the judge against trusted labels — calibrating a "judge of judges" (the thread we pick up in L6).
- Label or number, the same two defects apply. And with no ground truth you can only see noise (re-run consistency) — measuring bias always needs a trusted slice to calibrate against.