Choice of Metric Lab

Source #6 — "The winner depends on which question you ask."
What is this playground about? (click to expand)

The scenario: System A vs System B, same 200 queries, same LLM judge. You compute Correctness scores 1-5 for each query. Which system is better?

The twist: The answer depends entirely on which metric you use. Mean? Median? % above threshold? Worst-case? Each tells a different story, and they can disagree. The data doesn't change — the question changes.

The lesson: The metric encodes your values. There's no "correct" metric to discover. You must decide what matters before you measure. Otherwise, you'll unconsciously shop for the metric that confirms your preference.

Setup: Both systems were evaluated on the same 200 queries by the same judge. System A has a higher average but more variance — some great answers, some disasters. System B is more consistent but has a lower average. Click each metric tab to see who "wins."

Metric Tabs (below): Each tab shows a different way to compare the same two systems on the same data.

Mean — Average score. Rewards high highs; a few 5s compensate for 1s. System A likely wins here.

Median — Middle value. Ignores extremes. Shows the "typical" experience.

% ≥ 4 (Good Rate) — How often is the answer "good enough"? Rewards consistency over brilliance.

5th Percentile (Worst Case) — The worst 5% of experiences. Protects against disasters. System B likely wins here.

Std. Deviation — How much scores vary. Lower = more predictable experience.

After clicking 3+ tabs, the 4-Metric Trap panel appears, showing all 4 evaluation metrics at once — revealing that A and B each win on different metrics. Who's better is a product decision, not a data question.