The scenario: You're evaluating an AI customer support assistant. Your test set has queries in several languages — English, Spanish, Italian, German, Japanese. But does the mix of languages in your test set match what real users actually send?
Why it matters: If your system scores 4.5 on English but 2.8 on Japanese, and your test set is 80% English while production is only 50% English, your test scorecard will be wildly optimistic. You'll ship thinking the system is great — and users will disagree.
The lesson: Language is just one axis. Bias can hide in complexity, domain, user type, synthetic vs. real, question length — any dimension where the test set doesn't match production. Always ask: "Does my test set look like production?"
Step 1 — Feel it: See the scorecard from a biased test set. Decide whether to ship. Then reveal what production actually looks like.
Step 2 — Explore: Adjust the test set mix and production mix with sliders. Watch the gap appear and disappear.
Step 3 — Fix it: See per-segment breakdowns and stratified sampling. The fix isn't more data — it's representative data.
These are the system's real quality scores in each language segment. In practice you wouldn't know these — the playground lets you set them to see the effect.
| Segment | Correct. | Helpful. | Tone | Complet. |
|---|
1. Evaluate (Biased Test) — Runs the evaluation using your current test mix (set in the sliders above). You'll see per-segment scorecards and a pie chart of the test composition. The overall score looks good because the test is dominated by the segments where the system is strong.
2. Reveal Production Reality — Shows what each segment's real production traffic looks like — often very different from the test mix. Displays a side-by-side pie chart comparison (test vs production), a gap summary table, and highlights which segments are over/under-represented. The overall production score may be much lower.
3. Fix It — Stratified — Re-runs the evaluation using stratified sampling: each segment is weighted by its production proportion. The new score reflects reality, not your test set's bias. Shows the corrected scorecard and how much the gap closed.