Climbing on Noisy Labels
The same wrong labels do three different things depending on where they sit. In the training set the climb learns them. In the validation set they mislead your checks. In the test set they move the number you report.
How to read and interact with this page
- Each section puts the wrong labels in one set, and the other sets stay clean, so you see what each set’s labels do on their own. The fourth section puts one labeler’s habit in all three.
- Blue, purple and grey are scores you can see; orange is the truth: how often the router is right on new tickets, which you normally cannot see.
- “Run one climb” draws a new world each time. The buttons that run 200 or 500 climbs use fixed seeds, so they always give the same numbers.
- Try first: in the first section, run 200 climbs at every rate with the reviewer off, then read the table.
The router and its labels
We route support tickets to one of three teams: Billing, Access or Network. The data comes in three sets: 200 training tickets, 100 validation tickets and 100 test tickets. Each ticket’s label is the team that solved it, and sometimes that was the wrong team.
Tickets come in 30 patterns, a few common and many rare. The first prompt routes each pattern right 60% to 95% of the time, about 78% overall. Then an optimizer climbs on the training set. Each round it runs the prompt, reads every miss (a ticket where the answer differs from the label) and writes a rule for it.
A rule always fixes the ticket it was written for. With probability g, the fix generality, it is also a rule for the whole pattern, and it moves the pattern halfway toward the team in the label. If the label was right, the pattern improves on new tickets. If the label was wrong, the rule sends the pattern to the wrong team.
A wrong label comes in two kinds. Random slips are spread over tickets, and each goes to one of the other two teams. A labeler’s habit gets a few patterns wrong every time, always the same way.
Training · 200
The optimizer reads its misses and writes rules.
Validation · 100
Only scores come back. They check the climb and pick a strategy.
Test · 100
One look at the end gives the number you report.
1 · Wrong labels in training: the climb learns them
Here validation and test are clean, and some training labels are wrong. Each wrong label the optimizer reads looks like a miss. Patching that miss writes the label’s mistake into the prompt.
round-by-round log
| Round | Training score | Misses read | of which wrong labels | Rules from right labels | Rules from wrong labels | Relabeled | True accuracy |
|---|
| Wrong labels | First misses that are label errors | Training score after | True accuracy after | Real gain | Rules from wrong labels | Real gain, labels checked |
|---|
Averages over 200 climbs of 8 rounds on 200 training tickets, at the fix generality and kind of error set above. “First misses” are the misses read in the first round, before any patch. “Labels checked” means a reviewer looks at each miss before it is patched and catches 4 out of 5 wrong labels.
2 · Wrong labels in validation: the check misreads
Now training and test are clean, and some validation labels are wrong. Nobody reads validation tickets and no rule is written from them, so the climb itself is unaffected. What changes is what validation tells you about the climb.
A gap that is not overfitting
We score validation after every round to check the climb. A training score that pulls away from validation is the usual sign of overfitting. Wrong validation labels pull the two apart on their own.
Picking a strategy
Before the climb we fix four strategies: four optimizers whose rules generalize 10%, 30%, 50% and 70% of the time. Each one climbs on the clean training set. Validation scores the four final prompts once, and we keep the one that scores highest. Wrong labels shrink every difference validation sees, while chance moves a 100-ticket score as much as before.
| Wrong validation labels | Picks the truly best strategy | Average cost of the pick | A real 5-point gain shows as |
|---|
The cost of a pick is how many points of true accuracy it loses against the truly best of the four. A real difference between two prompts shows on noisy labels as that difference times 1 − 1.5 × (share of wrong labels), with three teams and slips spread evenly.
3 · Wrong labels in test: the number and its band are off
The climb is over and the prompt is frozen. We score it once on the test set and report the score with a 95% interval. Here we repeat that one look a thousand times, each on fresh test tickets with the same share of wrong labels.
| Wrong test labels | Average reported score | Off by | Interval contains the truth |
|---|
The interval is the 95% Wilson interval for the reported share. It accounts for which tickets ended up in the test set, not for wrong labels.
4 · One labeler for all three sets: a habit passes every check
Random slips in validation and test drag the score down. A habit does something worse when the same labeler labeled all three sets. The climb learns the habit on training, and validation and test reward it, because their labels share it.
What to do, set by set
| Wrong labels in | What they do | What you see | What to do |
|---|---|---|---|
| Training | Misses that are label errors become rules. The climb learns the noise and gains less. | Nothing: the training score climbs just as high. A clean validation set shows the smaller gain. | Before patching a miss, decide whether the router or the label is wrong, and relabel. The first misses you read hold the most label errors. |
| Validation | Every difference shrinks and the score drops. The climb itself is untouched. | A training–validation gap that looks like overfitting. Picks closer to a coin flip. | Label validation twice and settle disagreements. Estimate its error rate before reading the gap. |
| Test | The number moves, usually down. The interval ignores it. | A number off by more than its ±band, and a band that is no wider than before. | Use carefully settled labels. Audit a sample to estimate the error rate, and report it with the score. |
| All three, one labeler | A habit is learned on training and rewarded on validation and test. | Every score agrees, and every score is too high. | Have a second, independent labeler label validation and test. Audit the labels team by team. |
How the simulation works, and what it leaves out, is in MODEL.md.