Managing Context

The central challenge

Every time you call an LLM, it starts from scratch. There is no server-side session, no implicit memory. The model only sees the messages array you send in that request — nothing more.

This has profound implications for cost, latency, and architecture. This lesson walks through five strategies, each solving a problem the previous one creates:

Each call is independent

The Chat Completions API is stateless by design. Two separate calls share nothing:

# Turn 1: tell the model your name
r1 = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "My name is Alice."}],
)
# Turn 1: "Nice to meet you, Alice!"

# Turn 2: ask for your name — but we DON'T send turn 1
r2 = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "What is my name?"}],
)
# Turn 2: "I don't have access to your name."

The model can't answer because it literally never saw the first message.

Provider-managed state: the Responses API

OpenAI's Responses API can keep conversation state server-side. Chain turns using previous_response_id:

resp1 = client.responses.create(model="gpt-4.1-mini", input="My name is Alice.")

resp2 = client.responses.create(
    model="gpt-4.1-mini",
    input="What is my name?",
    previous_response_id=resp1.id,   # server remembers turn 1
)
# "Your name is Alice."

Convenient, but it's a black box — you don't control what's kept or trimmed. For the rest of this lesson, we'll manage context ourselves.

How ChatGPT works under the hood

The fix is simple: keep a messages list and append every exchange. On each turn, send the whole thing.

history = []

# Turn 1
history.append({"role": "user", "content": "My name is Alice."})
r1 = client.chat.completions.create(model=MODEL, messages=history)
history.append({"role": "assistant", "content": r1.choices[0].message.content})

# Turn 2 — the model now sees BOTH messages
history.append({"role": "user", "content": "What is my name?"})
r2 = client.chat.completions.create(model=MODEL, messages=history)
# "Your name is Alice."

It works — but on turn 50, you send 50 turns of history. On turn 500, you send 500 turns.

The quadratic problem

If each turn adds ~300 tokens, the input tokens on turn N are roughly 300N. The cumulative input tokens across all turns grow as ~150N².

Bounded cost, lossy context

The simplest fix: only send the last N turns. Cost is now bounded regardless of conversation length.

WINDOW_TURNS = 2
window = full_history[-2 * WINDOW_TURNS :]  # last 2 exchanges

# Ask something that requires early context
messages = window + [{"role": "user", "content": "What is my name?"}]
r = client.chat.completions.create(model=MODEL, messages=messages)
# "I don't have that information."  ← name was in turn 1, outside the window

What is a "turn"?

A turn is one user message plus all subsequent assistant and tool activity until the next user input. When trimming, always cut at turn boundaries — never split a user message from its response.

Turn 1:  user("Hi")  →  assistant("Hello!")
Turn 2:  user("Look up X")  →  tool_call("search")  →  tool_result(...)  →  assistant("Here's X")
Turn 3:  user("Thanks")  →  assistant("You're welcome!")
                              ↑
             With window=2, keep turns 2-3. Turn 1 is dropped entirely.

Compress old context, don't drop it

Instead of a separate summarization step, ask the model to do both in one call:

1. Answer the user's question
2. Update a running memory summary

Each turn sends: [system prompt with current memory] + [user message]. The model returns both an answer and updated memory as structured JSON.

PROMPT_TEMPLATE = Template("""You are a helpful assistant.

CURRENT MEMORY:
{{ memory }}

USER MESSAGE:
{{ user_message }}

Return JSON: {"answer": "...",
  "updated_memory": {"facts": [...], "preferences": [...], "goals": [...]}}""")

# Each turn: render template → call API → parse JSON → update memory
memory_text = json.dumps(memory)
prompt = PROMPT_TEMPLATE.render(memory=memory_text, user_message=user_message)
r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_object"},
)
parsed = json.loads(r.choices[0].message.content)
answer = parsed["answer"]
memory = parsed["updated_memory"]  # carry forward to next turn

How it works

[SYSTEM]  You are a helpful assistant.
          CURRENT MEMORY: {"facts": ["Name: Alice", "Location: Trento"], ...}
          ... return {"answer": "...", "updated_memory": {...}} ...

[USER]    What is my name and what do I do?    ← only the current turn

One API call per turn. Bounded tokens. No growing history. The memory acts as lossy compression of the entire conversation.

Memory stays bounded

Running an 8-turn conversation with memory, prompt tokens stay roughly flat:

Compare this to section 2 where tokens grew linearly per turn and quadratically in total.

Two complementary approaches

The OpenAI Agents SDK formalizes two short-term memory strategies that can be combined:

Synthetic message injection

When summarizing, the old conversation prefix is replaced with a synthetic pair:

[Synthetic user]:      "Summarize the conversation so far."
[Synthetic assistant]:  "{structured summary of turns 1-7}"
[Real turn 8]:          user("What metrics should I use?")
[Real turn 9]:          user("Remind me: what's my project?")
                         ↑
          Last N turns remain verbatim; older turns are compressed into the summary.

Persisting across sessions

Everything above is short-term memory — it dies when the conversation ends. Real applications need to remember users across conversations.

The extract → save → load → inject pattern

# End of session 1: extract facts
extracted = extract_long_term_facts(session1_conversation)
save_user_memory("marco", extracted)
# {"facts": ["PhD student in Trento", "Studies GNNs", ...],
#  "preferences": ["short, direct explanations"]}

# Start of session 2: inject into system prompt
loaded = load_user_memory("marco")
messages = [
    {"role": "system", "content": f"""You are a helpful assistant.

LONG-TERM MEMORY (from previous sessions):
Facts: {loaded['facts']}
Preferences: {loaded['preferences']}

Use this memory naturally."""},
    {"role": "user", "content": "What do you know about me?"},
]
# "You're Marco, a PhD student in Trento studying GNNs for drug discovery..."

This is the pattern behind ChatGPT's "Memory" feature. It requires user identity — you need to know whose facts to load.

Who does what?

When to use which strategy

Key design questions

1. How long are your conversations? — 5 turns? Full history is fine. 100+ turns? You need windowing or summarization.
2. Does old context matter? — If each turn is independent (e.g., code completion), a small window suffices. If users reference early decisions, you need summarization.
3. Do users return? — One-off interactions don't need long-term memory. Returning users benefit enormously from it.
4. What should memory forget? — Not all information deserves to be remembered. Selective memory (project facts vs. chitchat) is often better than total recall.

Key points

1. LLMs are stateless. Context is your responsibility — you send it, or it doesn't exist.
2. Full history doesn't scale. Cost and latency grow quadratically with conversation length.
3. Memory is lossy compression. You choose what to keep and what to lose. The summarizer decides what matters.
4. Short-term vs long-term. Short-term resets with the conversation; long-term persists across sessions and requires user identity.
5. Design for your constraints. Window size, summary format, persistence strategy — all tradeoffs with no universal right answer.

uv run python labs/03_context/1_stateless_agent.py
uv run python labs/03_context/2_stateful_agent.py
uv run python labs/03_context/3_agent_with_memory.py
uv run python labs/03_context/4_agent_with_long_term_memory.py