"Eval" in Software 1.0 (traditional) and 2.0 (ML)
What's similar and what changes when outputs become probabilistic
Evaluation in traditional software
In traditional software, expected outputs are known. You write a function, you know what it should return, and you write a test that checks.
- Unit tests: assert that
add(2, 3) == 5 - Integration tests: verify that components work together
- Regression tests: ensure new changes don't break old behavior
Evaluation is deterministic and binary — pass or fail. The hard part is coverage (did you test enough cases?), not measurement (is this output "good enough?").
You can always write down the expected output. If you can't, the spec is incomplete — not the test.
Evaluation in ML systems
Machine learning introduced probabilistic outputs. A classifier doesn't return "the right answer" — it returns a distribution over labels. Evaluation shifts from "is this exact output correct?" to "how good is this model across a test set?"
Key concepts
- Train/test split — evaluate on data the model hasn't seen
- Metrics — precision, recall, F1, AUC, RMSE — each captures a different aspect of quality
- Overfitting — good on training data, bad on new data
- Cross-validation — robust estimation of model performance
- Statistical significance — is this improvement real or just noise?
For a thorough treatment of ML evaluation methodology, see:
- MLVU: Evaluation — full lecture on methodology, metrics, and pitfalls
- Methodology lecture (annotated PDF)
"Correct" becomes a distribution. You need held-out data, aggregate metrics, and statistical reasoning. But you still have ground truth labels to compare against.