Five systems — E1 to E5 — that the course evaluates over and over, introduced in Lesson 3, The Dimensions of Experiment Design. They form a complexity ramp: from a deterministic two-input function, where evaluation looks like unit testing, to systems where even “correct” is a judgment call. The lesson walks E2 and E4 through the design dimensions in depth.
E1Host health anomaly score in monitoringCompute a single anomaly-severity score for a host from its CPU and memory deviations — the Euclidean distance from “no deviation at all”, then threshold it to fire an alert.
The degenerate case: a pure deterministic function. Exact outputs and near-zero run-to-run uncertainty once the specification and inputs are fixed — evaluation looks like unit testing. (The specification, the alert threshold, and input coverage can still be uncertain.)
def deviation_score(cpu_dev: float, mem_dev: float) -> float:
if not 0.0 <= cpu_dev <= 1.0:
raise ValueError(f"cpu_dev must be in [0, 1], got {cpu_dev}")
if not 0.0 <= mem_dev <= 1.0:
raise ValueError(f"mem_dev must be in [0, 1], got {mem_dev}")
return (cpu_dev**2 + mem_dev**2) ** 0.5deviation_score(0.0, 0.0) == 0.0 # perfectly normal
deviation_score(0.6, 0.8) == 1.0 # clean 3-4-5
deviation_score(1.0, 1.0) == math.sqrt(2) # both metrics maxed outThe same function, ported to JavaScript — including the two ValueError branches. Try a value outside [0, 1].
(cpu_dev, mem_dev) pairEntries marked derived follow from the function itself; the lesson’s goals table does not cover E1.
E2ML classifier for routing tickets to groupsA classification task: classify a support ticket into ONE of five classes — account, order, refund, payment, delivery.
A trained model. The function is learned from data, so uncertainty now lives in the data (sampling, labels), the training run, and the measured score — no longer in the code alone.
Source: Bitext Customer Support dataset on HuggingFace.
Predictions precomputed with the deck’s baseline — TF-IDF + logistic regression trained on the 2,000-ticket cached Bitext subset. Confidence is the model’s top-class probability — watch it drop on the ambiguous tickets.
E3GenAI few-shot classifier for routing tickets to groupsThe same classification task as E2 — five classes, same dataset — but solved by prompting an LLM with a few examples per class instead of training a model.
Three tunable parts — the prompt, the few-shot examples, and the model — so the boundary of the system under eval becomes a choice: are you evaluating the prompt, the examples, the model, or all three together?
Source: Bitext Customer Support dataset on HuggingFace.
Outputs precomputed with the LLM via OpenRouter — 5 few-shot examples per class, temperature 0 — on the same tickets as E2. Where the two disagree, ask: which one is right, and how would you know?
E4Self-solve via clusteringAnswer support requests without a human: cluster historical requests (which come with their solutions), attach a solution to each cluster, then answer a new ticket by matching it to a cluster and extracting that cluster’s solution.
A multi-stage pipeline with both training-time and inference-time components — and no clean ground truth for “the right solution”.
Clusters computed with k-means (k = 6) on TF-IDF over the same Bitext subset; each card shows real member tickets — the {{…}} placeholders are genuine dataset artifacts. Cluster names and resolutions are hand-written for this page (illustrative). Note the giant catch-all cluster: k-means owes you no tidy themes.
Matching = nearest centroid, precomputed. Watch how many natural-sounding tickets land in the catch-all cluster — the corpus vocabulary is templated, the tickets aren’t.
E5Text summarizationSummarize a text into a short summary a reader can act on.
Open-ended output: there is no single right answer, and many different summaries can all be good. Scoring needs a rubric, a human rater, or an LLM judge — the scorer becomes part of the experiment.
Summaries precomputed with the LLM, temperature 0. The source texts were written for this page. Is each summary faithful? Would you score it 4 or more — and on what scale?