The big idea: Your system performs differently in different domains. The aggregate scorecard hides this. This is not a measurement problem — it's a property of the world. No amount of better measurement or more data can fix it. Only improving the system itself can reduce the gap.
The scenario: You build an AI customer support assistant and sell it to three enterprise clients. Your internal test scorecard says Correctness 4.0 — looks solid! But each client sends a different mix of queries (finance, medical, legal), and the system handles these domains very differently.
What the three steps show:
Step 1 — Aggregate Scorecard: You see the internal evaluation — a single scorecard weighted by your internal test mix (which happens to be finance-heavy). The numbers look good. Leadership approves. You deploy.
Step 2 — Deploy to Clients: Each client has a different query mix. Client A (finance-heavy) sees great scores. Client B (medical-heavy) sees terrible scores. Client C (mixed) is somewhere in between. The system didn't change — the clients' realities are different. You also see a per-domain breakdown showing the real story: the system is great at finance and struggling with medical. The aggregate hid this.
Step 3 — Simpson's Paradox: Two systems (A and B) are tested on the same 200 queries. System A wins the per-query average. But when you look per-domain, System B is better at medical and legal — it just loses slightly on finance, which dominates the test set (80% of queries). If you weight domains equally or by production traffic, B wins. The "winner" reverses depending on how you aggregate. This is Simpson's Paradox: a trend in the aggregate that reverses when you break the data into subgroups, caused by a confounding variable (domain composition) hidden inside the average.
On the left panel you can see the true scores per domain (how the system actually performs in each language/domain — in practice you wouldn't know these) and the client query mixes (adjustable with sliders — drag them to see how different mixes create different client experiences from the same system).
Connection to Source #2 (Sampling Bias): Source #2 is about your test set having the wrong mix. Source #5 is about reality itself being heterogeneous. Even with a perfectly representative test set, the aggregate hides that some segments are failing. The fix isn't better sampling — it's per-segment reporting and improving the system on weak domains.
Does more data help? No. More data gives you a tighter estimate of each domain's score, but doesn't reduce the variance between domains. The gap is real. Only system improvements (better medical handling, domain-specific prompts, routing) can reduce it.
How your system actually performs in each domain. Finance: great. Medical: struggling. Legal: decent.
| Domain | Correct. | Helpful. | Tone | Complet. |
|---|
Each client sends a different mix of queries. Adjust to see how the same system creates different experiences.
1. Aggregate Scorecard — Shows the internal evaluation, weighted by your finance-heavy test mix. A single scorecard with 4 metrics. The numbers look solid — leadership approves.
2. Deploy to Clients — Reveals what three different clients actually experience: Client A (finance-heavy) sees great scores, Client B (medical-heavy) sees terrible scores, Client C (mixed) is in between. Also shows a per-domain breakdown exposing the real story behind the aggregate.
3. Simpson's Paradox — Compares two systems (A and B) on the same 200 queries. System A wins the per-query average, but System B wins per-domain and production-weighted averages. The "winner" reverses depending on how you aggregate — a classic aggregation paradox caused by domain composition as a confounding variable.