Hello Software 3.0

Learning objectives

By the end of this lesson, you will be able to:

1. Make a basic chat completion API call and understand its parameters
2. Stream responses token-by-token for better user experience
3. Build a voice pipeline using Whisper (STT) and TTS APIs
4. Generate images with DALL-E and understand prompt revision
5. Understand why testing AI systems is fundamentally different from testing traditional software

What is "Software 3.0"?

The term comes from Andrej Karpathy, who described an evolution in how we write software:

• Software 1.0 — explicit rules written by programmers (if/else, loops)
• Software 2.0 — behavior learned from data (neural networks trained on examples)
• Software 3.0 — behavior specified in natural language (LLMs following prompts)

In Software 3.0, the "code" is the prompt. You're about to write your first lines.

Setup

If you haven't already, set up the repository:

# Clone and enter the repo
git clone https://github.com/reliableai/ai-design-2026-pub.git
cd ai-design-2026-pub

# Install uv (Python package manager) if needed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dependencies (this creates a .venv and installs everything)
uv sync

# macOS only: portaudio is needed for the voice demo (script 3)
brew install portaudio

# Set up your API keys (one-time)
cp .env.example .env
# Edit .env and add your OPENAI_API_KEY

A shared API key will be provided during the lab. Add it to your .env file. All scripts automatically load keys from .env via python-dotenv.

Cost awareness

API calls cost real money. Even though you're using a shared key, it's good practice to understand what things cost. See OpenAI pricing for current rates.

Alternative: OpenRouter

For text-based examples (chat, streaming), you can use OpenRouter instead of OpenAI. OpenRouter provides a unified API to access models from OpenAI, Anthropic, Meta, Google, and others—often at lower prices.

from openai import OpenAI  # Same SDK!

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="openai/gpt-4.1-mini",       # Or "anthropic/claude-sonnet-4"
    messages=[...],                     # Or "meta-llama/llama-4-scout"
)

The OpenAI Python SDK works with OpenRouter—just change the base_url and use an OpenRouter API key. This lets you easily try models from different providers. Get yours at openrouter.ai/keys.

Text in, text out

The Chat Completions API is the foundation. Send messages, get a response.

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()  # Load OPENAI_API_KEY from .env
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[
        {"role": "user", "content": "Explain Software 3.0 in one sentence."}
    ],
    temperature=0.7,
)

print(response.choices[0].message.content)

Run it:

uv run python labs/01_hello_world/1_chat.py

The response object

The API returns a structured response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "gpt-4.1-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Software 3.0 refers to..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 32,
    "total_tokens": 47
  }
}

The usage field tells you how many tokens were consumed—this determines cost.

Watch tokens arrive

With streaming, you receive tokens as they're generated instead of waiting for the full response. This dramatically improves perceived latency.

stream = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "Write a haiku about APIs."}],
    stream=True,  # Enable streaming
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Run it:

uv run python labs/01_hello_world/2_streaming.py

Streaming vs non-streaming

The total time may be similar, but the user experience is vastly different.

Speak and be spoken to

Voice interaction combines three APIs into a pipeline:

Each step is a separate API call. The LLM doesn't "hear" you—it reads your transcribed words.

# 1. Record audio from microphone
audio = sd.rec(int(5 * 16000), samplerate=16000, channels=1)
sd.wait()

# 2. Transcribe with Whisper (Speech-to-Text)
transcript = client.audio.transcriptions.create(
    model="whisper-1",
    file=audio_file,
)

# 3. Get LLM response
response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": transcript.text}],
)

# 4. Convert to speech (Text-to-Speech)
speech = client.audio.speech.create(
    model="tts-1",
    voice="nova",  # Options: alloy, echo, fable, onyx, nova, shimmer
    input=response.choices[0].message.content,
)

Run it:

uv run python labs/01_hello_world/3_voice.py

You'll have 5 seconds to speak. The script records, transcribes, thinks, and speaks back.

Describe, create

The Images API with DALL-E 3 generates images from text descriptions.

response = client.images.generate(
    model="dall-e-3",
    prompt="A robot teaching a classroom of humans, digital art style",
    size="1024x1024",      # Also: 1024x1792, 1792x1024
    quality="standard",    # Also: "hd" (more detail, higher cost)
    n=1,                   # DALL-E 3 only supports n=1
)

image_url = response.data[0].url
revised_prompt = response.data[0].revised_prompt

Please don't run this one — image generation is expensive (~$0.04 per call). The instructor will demo it in class.

How do you test this?

In traditional software, testing is straightforward: sort([3,1,2]) must return [1,2,3]. There is exactly one correct answer. Same input, same output, every time.

Now try testing the prompt you just ran in Section 1:

def test_explanation():
    response = ask_llm("Explain Software 3.0 in one sentence.")
    assert response == ???  # What goes here?

Run it three times. You might get:

• "Software 3.0 is a paradigm where programs are written in natural language prompts rather than code."
• "It refers to the shift from explicit programming to instructing AI models with plain English."
• "Software 3.0 describes systems where behavior is specified through natural language rather than trained on data."

All three are correct. All three are different. No assert catches this — and unlike a formatting issue you could parse around, these are genuinely different valid answers. The problem isn't surface-level variation. It's that for any non-trivial question, "correct" is a spectrum, not a point.

What we CAN test

LLM-as-judge: testing AI with AI

For the voice pipeline, we can't easily verify correctness. Instead, we ask another LLM:

def test_voice_response_is_sensible(user_input, assistant_response):
    judgment = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{
            "role": "user",
            "content": f"""User said: "{user_input}"
Assistant replied: "{assistant_response}"

Is this a sensible, relevant response? Answer YES or NO."""
        }],
        temperature=0,
    )

    assert "YES" in judgment.choices[0].message.content.upper()

This is circular (using the thing we're testing to evaluate itself), but it's often the best we can do. We'll explore this deeply in Phase 2: Evaluation.

Running the tests

This lesson includes a test suite demonstrating these approaches:

# Run fast tests (skip expensive image generation)
uv run pytest labs/tests/test_lesson_01.py -v -m "not slow"

# Run all tests including image generation (~$0.08)
uv run pytest labs/tests/test_lesson_01.py -v

Run an LLM on your laptop

In class you used cloud APIs (OpenAI, OpenRouter). For homework, you'll run a model locally on your own machine using Ollama. No API key needed, no cost, no internet required once the model is downloaded.

Step 1: Install Ollama

Download and install from ollama.com/download. It's available for macOS, Linux, and Windows.

# macOS / Linux — or just download from the website
curl -fsSL https://ollama.com/install.sh | sh

Verify it's working:

ollama --version

Step 2: Pull a model and chat

Download a small model that runs well on most laptops (requires ~2 GB of disk and RAM):

# Download the model (~2 GB)
ollama pull llama3.2

# Start chatting in the terminal
ollama run llama3.2

Try asking it the same question from class: "Explain Software 3.0 in one sentence."

Type /bye to exit the chat.

Step 3: Use the OpenAI SDK with Ollama

Here's the key insight: Ollama exposes an OpenAI-compatible API. The same Python code you wrote in class works with local models — just change the base_url:

from openai import OpenAI

# Point the SDK at your local Ollama server
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # Required by the SDK, but Ollama ignores it
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Explain Software 3.0 in one sentence."}
    ],
    temperature=0.7,
)

print(response.choices[0].message.content)

The only differences from the cloud version: base_url points to localhost, and the model name matches what you pulled with Ollama.

Step 4: Complete the notebook

Open the homework notebook and work through the exercises:

uv run jupyter lab labs/01_hello_world/homework_01_ollama.ipynb

The notebook walks you through:

1. Connecting to Ollama with the OpenAI SDK
2. Comparing cloud vs local responses (quality, speed)
3. Streaming from a local model
4. Trying different models and system prompts

From calls to conversations

You've made four types of API calls. Each was stateless—the model has no memory of previous interactions. Every call starts fresh.

In the next lesson, we'll build agents: programs that maintain conversation history, make decisions, and use tools. The single API call becomes a building block in a larger loop.

Exercises

1. Temperature experiment: Modify 1_chat.py to use temperature 0, then 1.5. Run each 3 times. How do the responses differ in consistency?
2. Measure TTFT: In 2_streaming.py, measure time-to-first-token vs total time. Print both. What's the ratio?
3. Voice variations: Change the TTS voice in 3_voice.py. Options: alloy, echo, fable, onyx, nova, shimmer. Which sounds most natural?
4. Prompt revision: Generate images with "A cat" vs a 50-word detailed prompt. Compare DALL-E's revised prompts. When does it add more vs less?
5. Test the tests: Run test_exact_match_fails 10 times. Does it always fail? Always pass? Sometimes both? What does this tell you?

API documentation & resources

OpenAI API reference

• Chat Completions API — the core text generation endpoint
• Streaming — receive tokens as they're generated
• Audio Transcription (Whisper) — speech to text
• Text-to-Speech — text to audio
• Image Generation (DALL-E) — text to image
• Models overview — available models and their capabilities
• Pricing — current API costs

Python SDK & tools

• openai-python — official Python client library (works with OpenRouter too)
• python-dotenv — load environment variables from .env files
• OpenRouter — unified API for multiple LLM providers

Background reading

• 1hr Intro to LLMs — 1 year old and still very useful overview of how large language models work
• Software 2.0 — Andrej Karpathy's original essay (Software 3.0 extends this)
• OpenAI Text Generation Guide — comprehensive guide to chat completions

Code for this lesson

• lesson1_demo.ipynb — single notebook demo for class walkthrough
• labs/01_hello_world/1_chat.py — basic chat completion
• labs/01_hello_world/2_streaming.py — streaming responses
• labs/01_hello_world/3_voice.py — voice pipeline
• labs/01_hello_world/4_image.py — image generation
• labs/tests/test_lesson_01.py — test suite demonstrating testing approaches
• homework_01_ollama.ipynb — homework: local LLM with Ollama