Optimizing in the Dark: What Everybody Gets Wrong — but Not You

An experiment is successful if it reduces uncertainty—whether that means making a concrete decision or simply deepening your understanding of a system, a dataset, or a phenomenon.

Decision-oriented experiments answer “what should we do?”:

• Choose between Approach A vs B
• Ship a change (because it improves the north-star and respects guardrails)
• Iterate (because it improves one slice but fails on another)
• Stop (because gains are too small relative to cost/complexity)

Understanding-oriented experiments answer “how does this work?” or “what is going on?”:

• Characterize — how does the system behave under different conditions?
• Discover — what patterns or structure exist in the data?
• Explain — why does the system fail on this slice? What drives the variance?
• Calibrate — how reliable is this metric / oracle / judge?

Both types benefit from the same rigor—clear questions, careful measurement, and honest reporting. The difference is what you do with the answer: act on it, or learn from it.

Building a system is not the same as evaluating it. You can write elegant code, design clever prompts, and wire together powerful LLMs—but none of that tells you whether your approach works, why it works, or when it breaks.

Good experiment design lets you make defensible claims:

• “Our interaction strategy converges in fewer turns than the baseline.”
• “Embedding similarity improves recall of KB gaps at fixed precision.”
• “This improvement persists across datasets/oracles and isn’t a seed artifact.”

2.1 Start with a Research Question (Broad)

A research question frames what you want to learn.

2.2 Convert to a Testable Hypothesis (Falsifiable)

A hypothesis is a specific claim with measurable outcomes.

Conversational clustering hypotheses:

• H1: Pairwise “should X and Y be together?” questions converge in fewer turns than full-clustering proposals.
• H2: Modeling contradictory feedback (memory + resolution) improves final clustering quality vs treating feedback as independent.
• H3: LLM-oracle behavior matches key human-oracle distributions (feedback types, convergence turns) well enough to be a valid simulator.

KB gap detection hypotheses:

• H1: Embedding-based link prediction finds more valid missing triples than rule-based path mining at fixed precision.
• H2: Structural + textual fusion outperforms either alone.
• H3: Performance degrades gracefully as KB completeness increases (fewer true gaps).

2.3 Make Hypotheses “Specific Enough to Fail”

Include:

• Target metric(s)
• Expected direction and magnitude
• Failure condition: what result would falsify the hypothesis?

3.2 Why This Matters

Without a pre-spec, it’s tempting to:

• Switch your “primary” metric to whichever one looks best
• Add conditions after seeing partial results
• Report a subset of runs that tell a cleaner story
• Redefine “convergence” to favor your method

None of this is intentional dishonesty—it’s how human pattern-matching works. The pre-spec protects you from yourself.

Think of evaluation as a ladder:

1. Offline (fastest): fixed dataset + labels/judges
2. Replay: run on logged inputs with a stable scoring harness
3. Shadow: run on real traffic, no user-visible impact
4. Online A/B: randomize users/sessions; measure user outcomes

Use cheaper rungs to iterate and detect regressions, and expensive rungs to validate real-world impact.

5.1 Unit of Analysis and Randomization

Pick the “thing” that gets assigned to conditions. The right choice depends on where your independent variable acts:

For conversational clustering, session-level is usually the right unit: each session is one full conversation between the system and an oracle about one dataset.

5.2 Controls and Confounds

A confound is anything that differs across conditions besides your intended change:

• Different LLMs (or different temperatures)
• Different prompt budgets or context windows
• Different compute/time limits
• Different preprocessing or chunking
• Different random seeds (run-to-run variance)

5.3 Threats to Validity

Include a threat assessment in every write-up. Use these checklists to self-audit:

6.1 What Role Does Data Play?

6.2 Dataset Characterization (Mini “Dataset Card”)

For each dataset, document:

• Domain, size, time range
• How it was collected
• Known biases / missingness
• What “labels” mean and how reliable they are
• Privacy constraints and redactions

6.3 Splits and Leakage (Especially with LLMs)

Controls:

• Maintain a strict separation: dev vs final evaluation
• Use time-based or group-based splits when appropriate
• Snapshot your KB and record the version/date

6.4 Synthetic vs Real Data: Tiered Evaluation

Synthetic setups accelerate iteration but can mislead. A good tiering strategy:

7.1 Build a Metric Tree

Use this template for every experiment. Fill in all six rows before running.

METRIC TREE
============

North-star (the one metric that defines success):
  [What ultimately matters. One metric.]

Driver metrics (levers that move the north-star):
  [2–4 metrics that contribute to the north-star.]

Diagnostics (explain *why* drivers move):
  [Metrics that help debug, not decide.]

Guardrails (must not regress):
  [Hard constraints. Experiment fails if these get worse.]

Slices (must report separately):
  [Subgroups where performance may differ.]

Budgets (operational constraints):
  [Tokens, latency, cost, human time.]

7.2 Intrinsic vs Extrinsic Metrics

• Intrinsic: evaluates output in isolation (NMI, ARI, Hits@10, MRR)
• Extrinsic: evaluates usefulness downstream (deflection rate, triage time)

Interactive clustering often needs oracle-based / extrinsic metrics because there may be no single ground truth.

7.3 Define Metrics Precisely (Formula + Unit + Aggregation)

Write:

• Unit: per session / per dataset / per oracle
• Aggregation: mean, median, p95
• Meaningful difference: what Δ matters?

7.4 Add Operational Metrics for LLM Systems

For every experiment that uses LLM calls, track:

• Tokens in/out, calls/session
• Latency (p50/p95)
• Cost estimate
• Failure rates (timeouts, tool errors)

8.1 Baselines You Should Always Have

• Trivial floor: random / all-in-one / all-separate
• Reasonable non-interactive: k-means on embeddings; one-shot LLM clustering
• Your system (full)
• Ablations (next section)

9.1 What Is an Ablation?

Remove or simplify one component to estimate its contribution. The term comes from medicine (removing tissue to understand its function). In systems work, it means: what happens when I take this piece out?

Ablations answer:

• What matters
• What is redundant
• What is expensive but useless
• Where improvements actually come from

9.2 How to Design Ablations

1. Draw a block diagram: each block is a candidate ablation.
2. Replace one block with a neutral/simple version.
3. Change one thing at a time.
4. Measure with the same harness and data.

9.3 Interaction Tests (When Components May Synergize)

If you suspect A and B interact, run a small factorial:

• baseline
• +A
• +B
• +A+B

This reveals synergy vs redundancy.

9.4 Cost Ablations (Highly Recommended)

Ablate expensive components first and plot quality against cost. This is often the real story—a component might contribute a 2% quality improvement at 40% of the total cost, which changes whether you’d actually use it.

10.1 Define the Session Contract

A session contract specifies the rules of engagement so that every run follows the same structure and results are comparable.

Without a contract like this, you’ll find that run-to-run differences in session structure make it impossible to tell whether outcome differences are due to your intervention or due to protocol drift.

10.2 Define the Instrumentation Schema (What Gets Logged)

Decide what to log before running. If you forget to log something, you can’t recover it.

{
  "session_id": "exp03-dataset02-oracle_llm-seed42",
  "condition": "strategic_selection",
  "seed": 42,
  "dataset_id": "imdb_500",
  "oracle_type": "llm_oracle",
  "oracle_persona": "detail_oriented_analyst",
  "model_id": "gpt-4-0613",
  "temperature": 0.2,
  "max_tokens": 800,
  "prompt_hash": "a3f8c1d2"
}

{
  "turn_number": 4,
  "system_action": "pairwise_question",
  "displayed_items_count": 2,
  "displayed_clusters_count": 0,
  "oracle_response_type": "correction",
  "oracle_response_length_tokens": 34,
  "oracle_response_time_ms": 1200,
  "llm_calls": 2,
  "tokens_in": 3400,
  "tokens_out": 280,
  "latency_ms": 1850
}

{
  "total_turns": 9,
  "convergence_reason": "satisfaction_stable",
  "final_satisfaction": 4,
  "final_pairwise_f1": 0.81,
  "total_tokens": 31200,
  "total_cost_usd": 0.41,
  "total_duration_s": 142
}

10.3 Convergence Criteria

Pick one and justify:

• Oracle declares “done”
• No cluster changes for N turns
• Satisfaction ≥ threshold and stable
• Pairwise disagreement < ε (if you can compute it)

10.4 Avoid Order Effects and Carryover

If the same human oracle sees multiple conditions:

• Randomize condition order
• Counterbalance across participants
• Avoid showing the same dataset twice if learning carryover matters

10.5 Measure Cognitive Load (Don’t Just Claim It)

Proxies:

• Items shown per turn
• Words shown per turn
• Response time
• Self-reported effort (1–5)
• Dropouts / fatigue signals

11.1 Run Multiple Repetitions

LLMs are stochastic; clustering can be seed-sensitive. Run each condition ≥ 3–5 times (more if variance is high).

Report:

• Mean ± std and/or median + IQR
• Confidence intervals for differences (bootstrap is robust)

11.2 How Many Runs Do I Need? (Power and MDE Intuition)

Practical approach using bootstrap (no formulas needed):

1. Run a small pilot (3–5 seeds) of your baseline on one dataset.
2. Compute your primary metric for each seed. Note the standard deviation.
3. Use this rule of thumb: to detect an effect of size Δ, you need roughly N ≥ (2 × std / Δ)² runs per condition.
4. Alternatively, bootstrap your pilot data: resample with replacement, compute the CI width for the mean. If the CI width is larger than the effect you care about, you need more runs.

11.3 Choose Tests That Match Your Design

• Paired tests (paired t / Wilcoxon) when runs are matched by dataset/oracle
• Bootstrap CI for metric differences as the default
• Report effect size (not just p-values)

11.4 Multiple Comparisons: Why Testing Many Things Inflates False Positives

If you run 20 experiments and use p < 0.05 as your significance threshold, you’d expect about 1 “significant” result purely by chance—even if none of your interventions actually work.

Practical mitigations:

• Pre-specify primary comparisons (see Section 3). Before running, write down which 2–3 comparisons are your main claims.
• Hold out a frozen test set. Use dev data to screen many ideas. Only evaluate your final 2–3 candidates on the held-out test set.
• Validate winners on fresh data. If variant B “won” in your main experiment, run it once more on a new dataset or new oracle set.
• Apply formal corrections when needed. Bonferroni divides your significance threshold by the number of tests. Benjamini-Hochberg controls the false discovery rate and is less conservative. For most student projects, pre-specifying and validating is sufficient.

12.1 Why Error Analysis Matters More Than Aggregate Metrics

An aggregate metric like “0.71 F1” tells you how well your system does. Error analysis tells you why it fails and what to try next. Without error analysis, improving a system is guesswork—you’re trying random things and hoping the number goes up.

12.2 How to Do Error Analysis (Step by Step)

1. Collect failures. Gather all cases where the output was wrong or the oracle was unsatisfied.
2. Sample and inspect. Sample 30–50 representative cases. Read them carefully—the actual items, the oracle’s words, the system’s choices.
3. Categorize. Group failures into types. Don’t decide categories in advance—let them emerge from the data, then consolidate.
4. Quantify. Count how many failures fall into each category. This is your error taxonomy.
5. Prioritize. The largest category is usually your best next intervention.

13.1 Calibrate Against Human Labels

Before trusting LLM-judge or LLM-oracle outputs at scale, validate on a small human-labeled set:

• Have humans perform the same task on 50–100 cases.
• Run the LLM on the same cases.
• Compute agreement: Cohen’s kappa, correlation, or accuracy on the human-labeled subset.
• If agreement is low, your LLM proxy is not measuring what you think.

13.2 Test Self-Consistency (Reliability)

Run the same LLM-judge on the same inputs multiple times (3–5 repetitions). Measure how often it gives the same answer. If self-consistency is low (e.g., the LLM rates the same clustering 3/5 one time and 5/5 another time), then your “metric” has high measurement noise—and you need either more repetitions (averaged) or a better prompt.

13.3 Known LLM-Judge Biases

13.4 Practical Mitigations

• Always calibrate on a human-labeled subset before trusting LLM judgments at scale.
• Run self-consistency checks and report the variance.
• Randomize presentation order in any pairwise LLM evaluation.
• For LLM oracles: compare feedback type distributions (correction rate, redirect rate) against human data. If the LLM oracle almost never disagrees, it’s too easy.

Run these before trusting results:

• Random label test: if you randomize labels/feedback, performance should collapse.
• Permutation invariance: renaming cluster IDs should not change metrics.
• Leakage scan: duplicates across splits; near-duplicate items.
• Budget sensitivity: if you halve token budget, does quality degrade smoothly?
• Canary cases: a few hand-crafted examples that must behave predictably.

16.1 The Optimization Loop

1. Establish baseline + instrumentation
2. Run error analysis → pick the biggest bottleneck
3. Generate 2–5 targeted variants
4. Screen offline → promote to shadow → confirm online (if applicable)
5. Lock in, add regression tests, repeat

16.2 Error Taxonomies (Turn Failures into Next Steps)

Conversational clustering failures:

• Poor question selection (wasted turns)
• Memory loss / contradiction mishandling
• Oracle overload (too much info)
• Unstable clusters (flip-flopping)

KB gap detection failures:

• Retrieval / candidate generation misses
• False positives due to entity ambiguity
• Relation-specific weaknesses
• Evaluation leakage (easy negatives)

See Section 12 for how to build and use these taxonomies in practice.

17.1 Prompt Changes Are Interventions

Changing a system prompt can shift behavior across your entire test set. Treat every prompt edit as a candidate that must be evaluated:

• Save the exact prompt text (version it in git, or at minimum log it with a hash and timestamp)
• Run the full evaluation harness, not just a few spot checks
• Compare against the previous prompt version, not just your memory of how it worked

17.2 LLM Version Updates Can Silently Break Things

Model providers update endpoints without notice. A pipeline that worked last month may behave differently today.

Mitigations:

• Pin model versions in experiments (e.g., gpt-4-0613, not gpt-4)
• Include a few canary inputs with known-correct outputs; check these before trusting a full run
• Log the model identifier returned in API responses, not just what you requested

17.3 Determinism Is Harder Than in Classical ML

Even at temperature 0, LLM outputs can vary across API calls due to batching, quantization, and infrastructure changes. This means:

• Variance across runs is real and must be measured, even with “deterministic” settings
• Caching exact LLM outputs for ablation studies (so you isolate your change from LLM variance) is a valid and recommended practice
• When comparing two prompts, run them on the same inputs in the same session if possible

19.1 What to Include in the Write-Up

• Hypothesis + decision
• Pre-spec (or label as exploratory)
• Dataset card(s)
• Metrics (definitions + why they match goals)
• Baselines + fairness budgets
• Ablations
• Variance + CIs
• Qualitative examples (2–3)
• Limitations + failure cases
• LLM-judge calibration results (if applicable)

19.2 Reproducibility Checklist

• Exact model names/versions (pinned, not floating)
• Prompts (in full, in an appendix) + prompt hashes
• Temperature, top_p, max_tokens
• Seeds + number of repetitions
• Dataset versions / KB snapshots
• Code commit hash
• Logging schema for runs (see Section 10.2)
• Cost per run (tokens + dollars)

Experiment design is not bureaucracy—it’s how you turn exploration into knowledge.

1. Start from a decision and a pre-specified, falsifiable hypothesis.
2. Measure what you care about with a metric tree and guardrails.
3. Compare against fair baselines and run ablations to understand causality.
4. Do error analysis to understand why, not just how much.
5. Treat variance as first-class—run enough seeds and report CIs.
6. Calibrate your instruments (especially LLM judges and oracles).
7. Treat every prompt and chain change as an intervention that needs evaluation.
8. Log everything; version everything; report everything—including failures.