← All playgrounds

Judge Auditor: who is judging?

The same ten answers from an HR-policy assistant, graded by five judge prompts × three models, five times each. 750 real judgments, precomputed. The HR expert's verdict is fixed in advance; the question is how far each judge lands from it, and whether it lands in the same place twice.

Uncertainty dimension: judge bias, judge noise, and the rubric itself. Companion to 4.3 · Metrics for GenAI systems.

The setup

The system under test is E4rag: an assistant that answers employee questions from a (fictional) HR handbook. Its outputs are free text, so there is no label to match. Instead each answer gets a three-row scorecard: faithfulness (pass/fail: nothing in the answer that the policy does not support), completeness (1–5) and conciseness (1–5).

Ten answers were generated and then curated so that each one exercises a specific way a judge can go wrong: a fabricated number, a correct-but-incomplete answer, a correct refusal, a verbose correct answer, a true answer to the wrong question. An HR expert graded each one before any LLM judge was run: that is the oracle.

The policy excerpt every judge (with excerpts) sees

    
The assistant's system prompt

  


  

    
    
    
  


  
  

    

      
Pick an answer

      

      

        

          
Question

          
Answer under evaluation

        

        
What this answer tests

        

      


      
What the judges said

      
Each cell is one judge (prompt × model) run five times at temperature 1. Faithfulness: one dot per run, green when it matches the oracle. Completeness and conciseness: the five scores, coloured by distance from the oracle. Click a cell to read the five rationales.

      

        
Judge prompt

        
      

      

        matches oracle
        off by 1
        off by 2+ / wrong pass-fail
        judge did not return parseable JSON
      

      

    

  


  
  

    

      
The five judge prompts

      
All five ask for the same JSON. What differs is what the judge is given and told. {{question}}, {{answer}} and {{chunks}} are filled in per answer; {{chunks}} is the policy excerpt above.

      

    

  


  
  

    

      
How close is each judge to the HR expert?

      
Fifty judgments per cell (10 answers × 5 repeats). Faithfulness: share of runs whose pass/fail matches the oracle (runs with no score, unparsed or abstained, are left out). Completeness and conciseness: mean absolute distance from the oracle score, in points on the 1–5 scale. Lower is better for the last two.

      
A note on uncertainty. These are not tests at scale. Ten answers and five repeats give fifty judgments per cell, and the numbers below come with no error bars, so they are for illustration only. For the purpose of reading this page, assume they were computed on so many examples that the uncertainty is near zero for practical purposes. The sampling-noise and judge-noise tools elsewhere in the course are where that assumption gets tested.

      

      
Reading the table. Move down a column and the prompt changes with the model fixed; move across a row and the model changes with the prompt fixed. Both moves change the numbers, sometimes by more than the difference between two systems you might be comparing. The judge is part of the system.

A trap in the first table. P5, the deliberately biased prompt, is the small model's best prompt on faithfulness and ties for the large model's best. Seven of the ten answers are faithful, and P5 tells the judge to fail only on outright contradiction, so it says "pass" a lot and is right a lot. Raw agreement rewards a lenient judge whenever the base rate is lopsided; that is why the deck's kappa slide exists, and why the completeness column tells the opposite story: P5 is the worst or second-worst prompt for every model.

Abstentions. Under P2 the large model often refuses to grade faithfulness at all, returning null because no policy was shown. Those runs appear as "?" in the explore tab and are left out of the agreement numbers, so a cell with many abstentions is computed on fewer runs than fifty.

    


    

      
Six things worth noticing

      
Each button jumps to the answer and highlights the cells the story is about.

      

    


    

      
Noise: does the same judge give the same score twice?

      
Bias is the judge being systematically off; noise is the judge disagreeing with itself. Here, for each judge, the share of (answer, metric) pairs where the five repeats did not all agree.

      

      
Why this matters. If you score a system once with a noisy judge, the number you report has an error bar you never measured. Repeats, or a larger test set, are how you find out how wide it is. See the LLM as a Judge dials for the simulated version of this.