Designing Large Scale AI Systems

Learn to design agentic AI systems and the harnesses that support them. Master the key conceptual abstractions they need. Take the right architectural decisions that turns into reliable and effective systems.

CS Master Course | 2027

Sessions

Fifteen sessions plus the semester project.

  1. Foundations: How ML and LLMs work

    • Basics of ML training and inference
    • Basics of LLM training and inference

  2. Hello AI World

    • One call: messages in, response out, with tokens, cost and latency.
    • Streaming: the response arrives token by token. Time to first token vs. total latency.
    • Multimodal: images and voice alongside text, in the same call.

  3. Prompting and Reasoning

    • Introduction to prompting: instructions, structure, roles and output formats.
    • In-context learning: teach the model a task with examples in the prompt.
    • Reasoning: step-by-step thinking and reasoning models.
    • Test-time compute: spend more compute at inference, and what it buys in quality, cost and latency.

  4. AI conversations: Short-Term Context and Memory

    • Build a chat on a stateless API.
    • How conversation length affects cost and response quality.
    • Managing Context and State

  5. Tools and the Agent Loop

    • Tools: the model can choose and take actions
    • The loop: propose, execute, feed back. Stop conditions and budgets.
    • Combine the model, instructions, tools and loop into an agent with a defined scope.

  6. Observability

    • Record model calls, tool executions and loop steps in a trace.
    • One trace: find the failing step.
    • Use traces to map execution paths, measure cost per task and audit runs.

  7. Context and Harness Engineering

    • Build the harness around a given model: the loop, tools, context assembly and budgets.
    • Context engineering: at each step, assemble what the model needs. Instructions loaded on demand, retrieved knowledge (index, agentic search, long context), tool results, compacted history.
    • How retrieval-augmented generation (RAG) affects answer quality: chunking, citations, and tradeoffs between cost, freshness and control.

  8. Exposing and Consuming Functionality: APIs and MCP

    • Consuming: tools you didn't write. Third-party APIs and MCP servers, discovery, and trusting someone else's descriptions.
    • Expose your system as a tool for others, with clear inputs, outputs and error handling.
    • What the protocol standardizes (tools, resources, prompts) and what it leaves to you: authentication, permissions, versioning, quality.

  9. Security: Trust Boundaries

    • Data can act as instructions: injection through anything the agent reads.
    • Least privilege per tool, sandboxes, approval gates.
    • Identify threats based on what the agent can access and do.

  10. Understanding the Key Abstractions: Designing and Using Frameworks

    • Designing: what a framework must provide for the model call, the conversation, tools and the loop, traces, the harness and interfaces. Build a minimal one from your own code.
    • Using: map each construct of an existing framework to an abstraction. Port your agent and compare the two versions.
    • Framework limitations: hidden prompts, leaky abstractions, delayed model support and dependence on framework-specific tracing.

  11. Evaluating Agents

    • Design evaluation experiments, drawing on the companion course.
    • What agents add: score the path, not only the answer. Simulated users. Reliability over repeated runs.
    • Turn recorded traces into evaluation cases to test agent improvements.

  12. Orchestration: Workflows, Agents, and Agents of Agents

    • Decide which steps to control in code and which to delegate to the model.
    • Composing: subagents, orchestrators, councils. An agent as a tool.
    • Context isolation vs. multiplied cost. Splitting a task changes behavior. When one agent is enough.

  13. Building Agents Without Losing Control

    • Control during development: keep your code under control and know where you want human supervision.
    • Control in execution: decide how much autonomy to leave the agent and where to put points of control.
    • Runs of hours: checkpoints, retries, idempotent side effects.
    • Approvals, clarifications, handoffs. Feedback as data.
    • Cost and latency budgets at system level.

  14. Governance

    • Questions you must be able to answer: who did what, why, with which model and prompt version.
    • Risk tiers, human oversight, documentation. Regulation as a design input.
    • Implement governance controls through traces, versioning and approval gates.

  15. Shipping into Existing Products

    • Versioning model, prompts, tools and policies together.
    • Model migrations, monitoring in production, maintaining a large AI system.
    • Structuring the project and the team.