Session 03

The Science of Experimentation / I

Iterating in the Dark · 4 hours · Week 3

Learning Objectives

Design proper evaluation pipelines and recognize common pitfalls
Understand how evaluation challenges change in Software 3.0

Part 1: “Eval” in Software 3.0

Slides: Eval in Software 3.0 — What’s Different (pptx)

How evaluation changes when we move from classical ML to LLM-powered systems: vibe eval → golden datasets → LLM-as-judge. The visibility, culture, and action problems of uncertainty. Why “eval” should be called “estimation of a random variable.”

Part 2: Evaluating ML Models

Slides: Model Evaluation — Experiments, Statistical Testing, and Metrics (PDF)

Covers four topics:

Experiments — train/validation/test splits, overfitting, the multiple testing trap, cross-validation, hyperparameter search
Statistical testing — true metric vs. sample estimate, confidence intervals, standard error, error bars (what do they mean?), bootstrapping
Metrics — regression (MSE, RMSE, bias-variance), classification (class & cost imbalance, confusion matrix, precision, recall)
PR, ROC, and AUC — ranking classifiers, precision-recall tradeoff, ROC curves

Reading: Eval in ML — shortened reading (PDF — adapted from MLVU, VU Amsterdam)

← Previous: Probability · Next: The Science of Experimentation / II →