"Eval" in Software 3.0 (AI Agents)
When there is no right answer — LLM-as-judge, rubric design, and building eval datasets
What's different in AI-powered systems
AI-powered systems generate open-ended outputs: summaries, conversations, decisions, code. Often there is no single "right answer" to compare against.
New challenges
- No ground truth for many tasks — what's the "correct" summary of a meeting?
- Human judgment is expensive and doesn't scale to thousands of test cases
- LLM-as-judge as a scalable proxy — with its own failure modes (position bias, verbosity bias, self-preference)
- Rubric design becomes a first-class engineering skill — vague rubrics produce unreliable judges
- Eval datasets are hard to build — you need representative, diverse, adversarial examples
- Evaluation is continuous — not a one-shot gate but an improvement loop
You often can't write down the expected output. Evaluation itself becomes an AI problem — you use LLMs to judge LLM outputs, and you need to evaluate your evaluator.
LLM-as-Judge
When human evaluation doesn't scale, we turn to LLMs as automated judges. The idea is simple: ask a (typically stronger) model to evaluate the output of another model against a rubric.
Why LLM-as-judge?
- Scale: Evaluate thousands of examples in minutes, not weeks
- Consistency: Same rubric applied uniformly (no annotator fatigue)
- Cost: Orders of magnitude cheaper than human annotation at scale
- Iteration speed: Fast feedback enables rapid development cycles
Known failure modes
- Position bias: Judges prefer the first (or last) response when comparing two outputs
- Verbosity bias: Longer responses are rated higher regardless of quality
- Self-preference: Models prefer outputs that match their own style
- Sycophancy: Judges may agree with confident-sounding but wrong outputs
- Inconsistency: Same judge, same input, different scores across runs
- Anchoring to surface features: Judges key on formatting, structure, or keyword presence rather than actual substance
- Inability to verify facts: Judges cannot reliably detect factual errors or hallucinations — a plausible-sounding but wrong claim often gets a high score
- Leniency / central tendency: Judges avoid extreme scores, clustering around the middle of the scale and compressing real quality differences
- Sensitivity to prompt phrasing: Small changes in rubric wording can cause large, systematic score shifts
- Lack of domain expertise: Judges struggle with specialized content (medical, legal, financial) where errors require expert knowledge to catch
When you still need human evaluators
LLM-based judging is cheap and scalable, but there are situations where human evaluation remains necessary:
- Subtle factual correctness in specialized domains. An LLM judge may lack the domain expertise to catch plausible-sounding but wrong claims — e.g., an incorrect drug interaction in a medical summary, or a wrong legal citation. A domain-expert human evaluator can verify facts the LLM has no reliable way to check.
- Tone, empathy, and cultural appropriateness. In sensitive contexts (crisis support, HR communications, cross-cultural interactions), whether a response "feels right" depends on social and emotional nuance that LLM judges handle poorly. They tend to reward surface-level politeness rather than genuine empathy, and may miss culturally inappropriate phrasing. Human raters who share the audience's context catch these failures.
- Validating the judge itself. You need humans as ground truth to know if your LLM judge is biased — there is no way to bootstrap trust in an automated judge without human agreement data.
- Subtle safety violations. Some safety failures require real-world reasoning and context that LLM judges miss — e.g., an output that is technically polite but manipulative, or advice that is harmful only given the user's specific situation.
- Open-ended creative quality. When "good" is inherently subjective and audience-dependent (marketing copy, creative writing, UX microcopy), LLM judges tend to reward generic fluency over the qualities that actually matter to the target audience.
An LLM judge is itself a system that needs evaluation. You need to measure agreement with human judgments, track inter-run consistency, and watch for systematic biases. The right question is not "LLM judge or human?" but "where in the loop do humans add irreplaceable value?"
Rubric design as engineering
The quality of your LLM judge is bounded by the quality of your rubric. A vague rubric produces noisy, unreliable scores. A precise rubric produces consistent, actionable signal.
What makes a good rubric?
- Specific criteria: "Is the summary accurate?" is vague. "Does the summary contain all key decisions from the meeting?" is actionable.
- Clear scale anchors: Define what a 1, 3, and 5 look like with concrete examples
- Independent dimensions: Score accuracy, completeness, and tone separately — not as one muddled number
- Negative examples: Show the judge what failure looks like, not just success
Rubric design is prompt engineering for evaluation. Invest as much care in your judge prompt as you do in your system prompt.
Building evaluation datasets
Your eval is only as good as the data you evaluate on. A biased or unrepresentative eval set will systematically mislead you.
Dataset design principles
- Representative: Cover the real distribution of inputs your system will see in production
- Diverse: Include edge cases, adversarial inputs, and different user intents
- Versioned: Track changes to your eval set — changing the test changes the score
- Sized appropriately: Too small and your metrics have wide confidence intervals; too large and you waste resources
Eval datasets are a product, not an afterthought. They need curation, versioning, and continuous improvement — just like the system they evaluate.