How to read this doc

This course grades projects on what they discovered, not what they built, unless otherwise agreed. The build is instrumental — "just enough to prove or study what's needed." A polished system with no defensible findings fails. A minimal system that produces a clean, honest empirical claim succeeds.

The scaffolding is organized in three parts:

• Part A — Problem framing and study design (the core). What you're asking, why it's worth asking, how you'll know the answer.
• Part B — Just-enough engineering (the floor). The minimum apparatus required to run the study reproducibly. Engineering above this floor is not rewarded — time spent here is time stolen from Parts A and C.
• Part C — Analysis & reporting (the delivery). The analysis, the report, the story.

Each component below has the same structure:

• What it is — one-line definition.
• Why it matters — the pedagogical / scientific reason.
• Minimum acceptable — what a project must have.
• Nice to have — what strong projects add on top. (In Part B, "nice to have" is genuinely optional — don't invest here unless the study requires it.)
• Common failure mode — the anti-pattern we see most often.

Grading lives in deliverables.md §4, not here. This document defines what each component is; the rubric defines how it scores.

PART A — Study design (the core)

1. Research Questions and Versioned Study Plan

What it is. The evolving record of what you are trying to learn: the questions the project is asking, written down early, and updated as the project teaches you what the right questions actually are. A living document, not a locked-in contract.

Why it matters. Real study questions evolve. Initial framings turn out to be too broad, or ask the wrong thing, or depend on an assumption that doesn't hold once you see the data. This is normal and healthy — a project that discovers it was asking the wrong question and reframes it has learned something. What is not healthy is reframing silently, so the reader of the report cannot tell what was planned vs. what emerged. The discipline is to keep questions and their changes explicit, dated, and versioned, so the reader can always reconstruct the trajectory.

This replaces the heavier "pre-registration" norm used in purely confirmatory science. We want students to think hard about the question upfront, commit to an initial version, and then evolve it honestly — documenting each change with a short rationale.

Minimum acceptable:

• A docs/study_plan.md (or equivalent) committed by the end of Sprint 1, containing:
• A short, dated v0 with the initial research question(s), in plain language, and the tentative headline claim. "Tentative" is allowed; "falsifiable enough to tell if we're getting somewhere" is required.
• A rough idea of the primary outcome you'd measure.
• A "what we do not yet know" list of open questions.
• A changelog at the top of the document: every version bump (v0, v1, v2, …) is a dated entry with a one-paragraph rationale — what changed and why. New version = new commit. Old versions are preserved in git history; the current version sits at the top of the file.
• By the time the main experiment is run, the study plan is on some version (often v2 or v3) and the final report references it: "The headline claim tested here was introduced in v2 (date), after v1's pilot showed X."
• Claims in the final report are traceable to a specific study-plan version. Exploratory analyses that emerged after seeing results are labeled as such — not as confirmatory.

Nice to have:

• Secondary questions with explicit multiple-comparison hygiene (Bonferroni, Holm, BH) declared in whichever version introduces them.
• A small-scale pilot (N=5–10) run between v0 and v1 to sanity-check the pipeline and sharpen the question.
• A short "lessons from v_N-1 → v_N" reflection in the final report — the evolution itself is often an interesting finding.

Common failure mode. The study plan exists only as the final report's introduction, reverse-engineered after the results came in. No git history of the question. Reader cannot tell what was planned vs. what emerged from looking at the data. The opposite failure is also real: a team freezes v0 and refuses to update it even when evidence suggests a reframe — clinging to a dead question to "stay honest." The honest move is to update the plan and document the change.

2. Related Work / State of the Art (versioned)

What it is. A focused, living survey of how others have studied the same phenomenon — committed early, then evolved as the team's understanding of the field deepens. Like the study plan, this is a dated, versioned document, not a one-shot writeup at the end.

Why it matters. KM is a mature field — clustering, extraction, evaluation-without-ground-truth, survey methodology, graph construction, synthetic data fidelity all have decades of literature. A project that ignores this either reinvents known results or repeats known mistakes. The related-work section is what makes your methodology informed rather than improvised. Doing it early forces the design choices to be informed by the literature; doing it iteratively lets the survey sharpen as the team discovers which threads actually matter.

Minimum acceptable:

• A docs/related_work.md committed by the end of Sprint 1, containing:
• A short, dated v0: 5–10 anchor references (papers, tools, methods, datasets) the team already knows are adjacent, with a one-line take on each (what they did, what we take from them).
• An explicit "what we don't yet know about the field" list — threads the team suspects exist but hasn't surveyed yet.
• A changelog at the top of the document: each version bump (v0, v1, v2, …) is a dated entry with a one-paragraph rationale — what got added / removed / reframed and why. The changelog evolves in step with the study plan: when a study-plan version bump reframes the question, the related-work document usually needs a corresponding version bump.
• By Sprint 2, a v1 that expands the anchor set to the final target (≈ 8–15 engaged citations) and adds a positioning paragraph: "prior work X measured Y using method Z; we differ in…".
• A short note on methodological precedents in whichever version introduces them — e.g., "we use the PANAS scale because it's the established instrument; we chose bootstrap CIs following X et al."
• The final report's related-work section (≥ 1.5 pages) is rendered from the current version of related_work.md — same citations, same framing, no last-minute additions that weren't part of the iterative process.

Nice to have:

• A comparison table of the 3–5 most-adjacent prior works (what they did, what they measured, what's different about your contribution), evolved across versions.
• A short "lessons from v_N-1 → v_N" reflection — we dropped thread A after realizing it's solving a different problem; we added thread B after the pilot surfaced a measurement issue.
• An appendix with a broader bibliography for readers who want to go deeper.

Common failure mode. Related work written in the last week of the project, citing whatever vaguely matches — used to justify choices already made, not to inform them. The opposite failure: a v0 bibliography frozen at Sprint 1 that never evolves, even after the project's methodological neighbors have clearly shifted.

3. Study Design

What it is. The blueprint of the investigation: conditions, factors, controls, sample size, what's measured, what's held constant, how conclusions will be drawn.

Why it matters. The study design is where most of a KM project's intellectual work lives. A sloppy design cannot be rescued by clean code or nice plots. Committing to the design before running lets the analysis be confirmatory and the results trustworthy.

Minimum acceptable:

• A study-design document (docs/study_design.md) committed early: factors / conditions / levels, what is manipulated vs observed, what is held constant, what the primary outcome is, what the decision rule is.
• Sample-size justification. Back-of-envelope power calculation, or a principled argument for why N is enough given the expected effect. Feasibility-driven N ("we can afford 20 raters") is acceptable if accompanied by an explicit note on what that N can and cannot detect.
• Controls and counterfactuals. At least one control condition or baseline against which the headline claim is compared. A study with no baseline cannot support a causal claim.
• What this design rules out. An explicit list of alternative explanations the design does not eliminate (so the reader doesn't have to reverse-engineer the limitations).

Nice to have:

• Randomization plan (how order, assignment, prompt variants are balanced).
• Stratification or blocking variables.
• A small "design review" with Fabio before the main experiment runs.

Common failure mode. "We ran some conditions and compared them." No pre-specified factor structure, no baseline, no written reasoning for why this comparison answers the question.

4. Quality / Measurement Operationalization

What it is. The precise definition of "better" and the procedure for measuring it. Before you can say X improves Y, you need a rigorous specification of Y.

Why it matters. Quality in KM systems is multi-dimensional and task-dependent. Treating it as a single scalar ("accuracy") is the most common experimental-design error in the field. Operationalizing the construct before data collection is what separates a study from a demo. If it's defined after looking at results, the study is circular.

Dimensions of quality to consider (pick what's relevant, not all):

• Correctness / task success — did the system produce the right answer? Binary or graded.
• Faithfulness / grounding — are claims supported by sources? (Hallucination's opposite.)
• Calibration — when the system says it's confident, is it right more often?
• Construct validity — does your proxy measure what you say it measures?
• Inter-rater / inter-judge reliability — do two raters (or judges) agree above chance?
• Robustness — does the measure hold under paraphrases, adversarial inputs, distribution shift?
• User-facing utility — does the output advance the user's goal? (Requires a user study or proxy.)
• Cost / efficiency — tokens, latency, human-minutes per unit of quality. Usually a secondary metric.

Measurement strategies:

• Rule-based / deterministic — exact match, regex, schema validation. Cheap, narrow.
• Programmatic graders — code that checks a property (compiles, passes tests, sums to target).
• LLM-as-judge — a separately-configured model scores outputs. Cheap at scale but judge-biased; must be validated against human labels on a subset.
• Human evaluation — gold standard for subjective dimensions; expensive; requires rater training and inter-rater reliability reporting.
• User / proxy-user metrics — behavioral signals from real or simulated users.
• Established instruments — psychometric scales (PANAS, SWLS, Cantril Ladder, etc.) when the construct already has validated measures in the social-science literature.

Minimum acceptable:

• A written quality spec (docs/quality_spec.md) committed before the main experiment: which dimensions matter, how each is operationalized, which is the primary outcome.
• Every metric reported in the paper is listed in that spec, with its measurement method.
• If LLM-as-judge is used, it is validated against a human-labeled subset of ≥ 30 items; judge-vs-human agreement (Cohen's κ or equivalent) is reported.
• If human rating is used, inter-rater reliability is reported on a calibration subset.
• Confidence intervals on every quantitative quality claim.

Nice to have:

• Multi-dimensional quality reported as a table or radar chart, not collapsed to a single scalar.
• A "quality under distribution shift" analysis testing robustness.
• Independent re-annotation of a subset by a second rater to quantify measurement noise.

Common failure mode. "We measured accuracy." Accuracy of what? How graded? By whom? On which split? With what CI? Also: redefining the metric after seeing results — garden-of-forking-paths dressed up as primary analysis.

5. Data: Provenance, Ethics, Exploration

What it is. Where the data comes from, what its properties are, and under what terms you are allowed to use it.

Why it matters. KM is the science of extracting understanding from data. Skipping the provenance / ethics / exploration step means you are studying something you don't understand, possibly shouldn't be using, in ways the reader cannot replicate.

Minimum acceptable:

• Provenance. Where the data came from (public dataset, scraped, self-collected, LLM-generated). License and terms of use. Citation.
• Ethics. If any data is personal or identifiable, a short ethics note: what consent applies, what you anonymized, what you will not publish. If humans are rating or being surveyed, a similar note for them. Default to public data; escalate anything involving real private chats / emails / personal data to Fabio before collecting.
• Exploration. A short data-exploration artifact (notebook or report section): size, key distributions, null rates, duplicates, obvious quality issues. The reader should be able to understand the data without rerunning your pipeline.
• Held-out split declared up front. A portion of the data is set aside before analysis and used only once, for the final evaluation.

Nice to have:

• A data-card (in the style of Gebru et al.) included in the repo.
• A hashed manifest of the exact data used, so a reviewer can confirm "same data" vs "similar data."
• An exploratory-analysis section in the final report that informed the study design.

Common failure mode. "We used dataset X" with no exploration and no awareness of its known biases. Conclusions turn out to rehearse the artifacts of the data.

PART B — Just-enough engineering (the floor)

Every item in this part is a floor, not a goal. Investment above minimum acceptable is discouraged unless the study concretely requires it. A team that writes a second LLM harness while the related-work section is half a page is failing the course.

6. Minimum Viable Build

What it is. The smallest executable artifact that lets the study run, end-to-end, reproducibly.

Why it matters. The build is a means. Every feature that doesn't serve the study is a distraction from the study. Scope discipline is itself graded.

Minimum acceptable:

• A git repo that another person can clone and run. A README.md with exact setup steps and the one command (or 3-step sequence) that reproduces the headline result on a toy example.
• A working system (CLI, notebook, or small web app — whichever fits) that actually executes the experimental pipeline end-to-end.
• Config-driven experimental conditions. Each condition is a config, not a forked script.
• Declared module structure (src/, scripts/, notebooks/, prompts/ if LLMs are used).
• Seed control on every run; seed is recorded in the log.

Nice to have (only if the study requires it):

• A lightweight UI for human raters / oracles (spreadsheet-backed is fine; don't build React unless you have to).
• Typed interfaces between modules (pydantic / dataclasses).

Common failure mode. Building a "proper system" with auth, caching, multi-user support — while the experiment itself runs on N=12 and has no study-plan history. Engineering performed in place of science.

7. Prompts, Configs, and Instruments as Versioned Artifacts

What it is. Every prompt, every config, every rating rubric, every survey instrument lives as a named file under version control — not embedded in code, not pasted into chat histories.

Why it matters. Prompts and configs are the single biggest source of experimental variance. If they're embedded strings you can't diff them, review them, or report exactly what you ran. Survey wording shifts results — version it like code.

Minimum acceptable:

• prompts/ directory with one file per named prompt (if LLMs are involved).
• configs/ directory with one file per experimental condition.
• Survey instruments, rating rubrics, rater instructions all committed as files.
• Changes to any of these are git commits with a message, not silent in-place edits.
• The file hash (or commit SHA) of the prompt / config / rubric used is recorded in every run log.

Nice to have:

• Explicit variable substitution in prompts ({persona_description} rather than f-strings).
• A one-line diff in the report showing exactly how a prompt changed between v1 and v2 of an experiment.

Common failure mode. Prompts embedded as triple-quoted strings inside long functions. Survey wording edited in-place. The reader of the report cannot tell exactly what the subject saw.

8. Logging, Reproducibility, and Resilience

What it is. Every run leaves a structured, replayable trace. Failures fail loudly rather than silently truncate the data.

Why it matters. No logged run = no claim. This is the audit trail for your science. Silent failures are worse than loud ones: a study that "ran on N=200" but actually completed 137 runs (because the API rate-limited you and the loop kept going) has confidence intervals that are wrong.

Minimum acceptable:

• One structured record per run-step (JSONL or Parquet); one run = one file or one indexed batch.
• Every record includes: run_id, seed, config_hash, model_and_version (if applicable), timestamp, inputs, outputs, errors.
• Retry with exponential backoff on transient failures (rate limits, timeouts).
• Clear distinction between transient errors (retry) and permanent errors (fail loudly with the offending input logged).
• A post-run integrity check: did every intended run complete? Report the answer to the user before analysis begins.
• Headline results are regenerated from the committed logs by a committed analysis script.
• Fresh clone → documented command → headline result reproduces.

Nice to have:

• A log-viewer / slicing CLI.
• Cost-per-run recorded alongside outputs.
• A replay.py that re-executes a single run deterministically from its log.

Common failure mode. try: except: pass around the LLM / API call. Experiment "succeeds" with silently-missing data. CIs report what the surviving data says, not what you intended to measure. Also: print()-based logs scattered across notebooks — impossible to replay.

PART C — Analysis & reporting (the delivery)

9. Analysis and Statistical Rigor

What it is. The quantitative backbone of the findings. Confidence intervals on every claim. Multiple-comparison hygiene. Honest separation of confirmatory from exploratory analyses.

Why it matters. The headline claim is only as good as the statistics behind it. Point estimates without CIs are assertions, not evidence. Multiple comparisons without correction inflate false-positive rates. Exploratory analyses presented as confirmatory are fiction.

Minimum acceptable:

• Confidence intervals on every quantitative claim in the report. Bootstrap (or appropriate parametric equivalent) is the default.
• Confirmatory first, exploratory clearly labeled. The primary analysis — the one tied to the current study-plan version's headline claim — is reported first, with its result stated cleanly. Secondary or post-hoc analyses come after and are labeled as such.
• Multiple-comparison correction applied (and declared in the relevant study-plan version, Bonferroni / Holm / BH) whenever multiple primary claims are tested together.
• All figures and tables in the report are regenerated by committed analysis scripts from committed logs.
• Honest treatment of null results: a well-executed null earns full credit and is reported as a finding, not buried.
• If the question was reframed between study-plan versions, the report says so explicitly — "v1 asked X and we pivoted; the analysis below reflects v2's question."

Nice to have:

• Sensitivity analyses: results under alternative reasonable choices (e.g., different outlier thresholds, different aggregation methods).
• Effect-size estimates alongside p-values, not p-values alone.
• A "negative results" subsection.

Common failure mode. Point estimates without uncertainty. Post-hoc analyses presented as if they were planned from v0. Cherry-picked comparisons. p-values with no correction.

10. Reporting and Communication

What it is. The report and the presentation — the channels through which the work becomes legible to other people.

Why it matters. Everything else exists to enable this. A clean study with an unclear report fails to land. A clear report on shaky scaffolding misrepresents the evidence. Both the content and the communication are graded.

Minimum acceptable:

• Technical report, research-paper-style, 8–15 pages: intro · related work · study design (with a short "how the question evolved" paragraph referencing the study-plan versions) · method · results (with CIs) · discussion · limitations · ethics · references.
• Report structure mirrors the current study-plan version: primary analysis first, secondary / exploratory clearly marked.
• Explicit "what we can and cannot conclude" section.
• Contributions per team member stated (so the reader knows who did what).
• Final presentation (12 min solo / 20 min pair): question → method → findings → limitations → individual contributions.
• All figures regenerate from committed scripts + committed logs.

Nice to have:

• A short (1-paragraph) plain-language summary at the top.
• Public-facing artifacts: a blog post, a short video, a tweet-length TL;DR.
• A "replication pack" directory with everything needed to rerun headline figures in one command.

Common failure mode. A report that restates the README. Slides that restate the report. "We built a system that works" in place of a defensible quantified finding.