Project Ideas — Brainstorm

Orientation: study-and-analysis first. Every project tests an empirical claim; the build exists to enable the study. "Just enough engineering to prove or study what's needed" — polish beyond that is out of scope (and penalized if it crowds out the study).

Design constraints for every idea:

• Executable by a student team in the course timeframe
• Students source their own datasets (scoping is part of the project)
• Evaluation strategy that doesn't require expensive ground truth
• Connects to course material (experimentation, clustering, graphs, LLMs, measurement, extraction)
• "Interesting" — non-trivial, produces a defensible empirical claim

Three dimensions every project must cover:

1. Learning — understanding a method, a domain, a paper, or a literature thread. Embedded in every project as a real related-work survey and methodological grounding; never the deep deliverable on its own.
2. Building — a minimum viable working artifact: pipeline, tool, or reproducible notebook that runs end-to-end. The build is instrumental to the study, not the product.
3. Experimentation & analysis — the depth of the project. Hypothesis-driven, with a versioned study plan that evolves honestly as the project learns. Proper stats: CIs, power thinking, multiple-comparison hygiene. Must produce a claim with quantified uncertainty.

Every project has all three. Teams invest their depth in Experimentation & analysis — never in Building, never in Learning alone. In groups, individual members specialize so contributions are distinct and gradable.

Team size: 1 to 3 students. Scope should scale accordingly:

• Solo: tightest scope. Headline hypothesis only. One extraction method, one condition set.
• Pair: one owns Study design + analysis; the other owns MVB + data. Shared writeup.
• Trio: one primary + one secondary hypothesis, or one primary plus a stronger control arm. Three clearly-distinct gradable contributions (e.g., method / runtime / evaluation + analysis).

Universal components — every project includes these, no exceptions.

The authoritative spec is universal_scaffolding.md. The 10 required components are organized into three parts:

Part A — Study design (the core, where depth is rewarded):

1. Research Questions and Versioned Study Plan
2. Related Work / State of the Art (versioned)
3. Study Design
4. Quality / Measurement Operationalization
5. Data: Provenance, Ethics, Exploration

Part B — Just-enough engineering (the floor): 6. Minimum Viable Build 7. Prompts / Configs / Instruments as Versioned Artifacts 8. Logging, Reproducibility, and Resilience

Part C — Analysis & reporting (the delivery): 9. Analysis and Statistical Rigor 10. Reporting and Communication

Idea 1 — Conversational Clustering

One-liner: Let a user cluster a dataset by talking to the system instead of tuning K, distance metrics, and features by hand.

Possible framing:

• User says "group these customer reviews by complaint type" → system proposes clusters, shows examples, user says "split cluster 3, merge 5 and 7, ignore sentiment" → system re-clusters.
• LLM acts as the interface between user intent and clustering hyperparameters / feature selection / cluster labeling.

Open questions:

• What's the dataset? (text / tabular / mixed)
• Is the novelty in the interaction loop or in the algorithm underneath?
• How do we evaluate it without ground truth?

Connects to: Session 6 (clustering), Session 7 (LLM), Session 11 (evaluating without ground truth)

Idea 2 — Conversational Graph-Based Data Generation

One-liner: Generate synthetic datasets by conversing with an LLM that builds and walks a knowledge graph.

Possible framing:

• User describes a domain ("hospital patient records with comorbidities")
• System co-constructs a schema graph (entities, relations, distributions)
• Synthetic records are sampled by walking the graph
• Iterate: "make the age distribution skew older", "add correlation between X and Y"

Open questions:

• Use case: testing ML pipelines? Teaching? Privacy-preserving data sharing?
• How do we ensure realistic joint distributions, not just plausible marginals?
• Evaluation: fidelity vs a real dataset? Utility for downstream tasks?

Connects to: Sessions 8–9 (graphs), Session 7 (LLM)

Idea 3 — AI Happiness (studying happiness of AI agents)

One-liner: Treat AI agents as the subjects of study and investigate whether/when/how they exhibit something like happiness — and what correlates with it.

Framing options (need to pick):

• Behavioral: Put LLM agents in varied environments/tasks. Probe self-reports, persistence, engagement, task preference. Cluster by "affective state". Look for causes.
• Interventional: Manipulate conditions (prompts, tools available, peer agents, success/failure) → measure "happiness" proxies → causal analysis.
• Comparative: Different models / sizes / personas — who's happier? Are there correlations with capability, alignment training, etc.?
• Meta: Can an agent truthfully self-report happiness, and how do you validate that?

Why it fits KDD course:

• Heavy on experimentation, confidence intervals, multiple comparisons (Sessions 4–5).
• Requires operationalizing a fuzzy construct into measurable proxies — classic KDD move.
• Clustering and embedding analyses of behavior logs (Sessions 6–7).

Core sub-problems:

• Operationalization: what proxies? (self-report on a Likert; preference choices; persistence time; language sentiment; refusal rate; asking for tasks to stop / continue)
• Construct validity: do the proxies agree? Do they correlate with anything externally meaningful?
• Confound control: LLM "happiness" is partly training-artifact, partly prompt-induced. How do we separate?

Risks / traps:

• Anthropomorphism — need careful framing: "we measure X which we operationally label 'happiness'", not metaphysical claims.
• Sample size and variance across runs will bite. Good session-5 practice.

Connects to: Sessions 4–5 (experimentation, power, multiple comparisons), Sessions 6–7 (embeddings, clustering of agent behaviors)

Idea 4 — Conversational Graph Extraction from Unstructured Data

One-liner: Build a knowledge graph from a corpus (docs, emails, chats, web) through an interactive conversation — where the schema itself (what to extract, how to structure it) is discovered through dialogue rather than specified up front.

The core premise: Figuring out what you want to extract from a corpus and how to structure it as a graph is the hard part. Users rarely know the right ontology before they see the data — they discover what matters through iteration. A one-shot "design the schema, then extract" pipeline forces a commitment that is usually wrong or incomplete. The research question is whether conversational schema discovery produces a more useful / more stable / more complete graph than the traditional plan-then-execute workflow. This parallels Idea 1 (Conversational Clustering): in both cases the user doesn't know what they want until they see it, and the system's job is to efficiently converge on their implicit intent.

Possible framing:

• User points at a corpus → LLM surfaces candidate entities/relations from a sample and proposes an initial ontology
• User refines schema in chat ("merge 'company' and 'organization'", "add 'acquired_by' relation", "I don't care about dates here, only who said what to whom")
• System re-extracts with the updated schema; user spot-checks boundary cases; schema evolves
• Iterate until the schema stabilizes or a stopping criterion fires
• Could layer in: entity resolution, confidence scoring, provenance

Open questions:

• How do you measure schema quality when there is no ground-truth ontology? (The schema itself is part of the output — evaluation-without-ground-truth applies here exactly as it does in Idea 1.)
• What does the conversation contribute beyond what a careful one-shot prompt would produce? (Must compare against a non-conversational baseline.)
• How do you detect convergence — when should the system stop asking?
• Does the method produce a reusable schema that works on new corpus slices, or does the schema overfit to what the user happened to see during the session?
• Dataset? (news articles, scientific papers, legal docs, course materials — prefer domains where the "right" ontology is non-obvious.)

Connects to: Session 7 (entity extraction), Session 9 (knowledge graphs), Session 11 (evaluating without ground truth) — methodologically shares a lot with Idea 1.

Cross-cutting observations

• Conversational is a recurring theme (1, 2, 4). There may be a unifying thread: "the LLM as a steering wheel for a KDD pipeline".
• Ideas 1 and 4 share a deep structure: the user doesn't know the right clustering / the right ontology up front, and conversation is the tool for converging on their implicit intent. The measurement and study-design questions are nearly isomorphic — convergence rate, cognitive load per turn, stability of the converged artifact, reusability on new data.
• Graphs show up in 2 and 4.
• Idea 3 (AI happiness) is the most methodologically unusual — survey/psychometric methodology applied to AI agents as subjects.

Learning / Building / Experimentation per idea

Each project has all three. Experimentation is mandatory and is always where the depth goes (never Learning, never Building). The table below labels the natural locus of effort for each idea — but in every case, the build is minimum-viable and the experiment is the star. In groups of 2–3, individual students specialize on sub-components for distinct gradable contributions.

Idea 1 — Conversational Clustering

Group split (3 students): Lit-review + evaluation-design owner / MVB builder + prompt engineering / Experiment runner + stats.

Idea 2 — Conversational Graph-Based Data Generation

Group split: Survey & fidelity-metrics owner / Pipeline builder / Downstream-utility experimenter.

Idea 3 — AI Happiness (interventional)

Group split: Construct-definition & psychometrics owner / Harness builder / Stats & causal-analysis owner.

Idea 4 — Conversational Graph Extraction from Unstructured Data

Group split: Schema-discovery methods researcher + MVB builder / Evaluation-design + ground-truth (or partial-benchmark) curation / Experiment runner + stats.