Using AI Frameworks
When and why to use frameworks — LangChain as a case study, plus how memory, context management, and observability shape the decision.
We spent seven weeks building from scratch. A single LLM call (L1). A stateless chat loop (L2). A stateful agent with four different memory strategies (L3). Tool calling (L4). Eval harnesses (L5–L8). Monitoring (L9). Councils and complex systems (L10–L13). Every abstraction we could have imported, we built by hand first. That was on purpose.
Now the question this lecture answers: should you throw away what you built and adopt LangChain? Or LlamaIndex, or LangGraph, or one of the twenty other frameworks that promise to turn your 200-line agent into 15? The honest answer is "it depends," and the point of this lecture is to give you the language to say what it depends on, rather than arguing from taste.
We will not try to cover every framework. We'll treat LangChain as the case study because it's the one most students will encounter first, compare it to LlamaIndex and LangGraph where those are genuinely different rather than cosmetically different, and spend real time on the two places where framework choice actually bites in production: memory and context management, and observability.
Four things you're really buying
"LangChain lets you build an agent in 10 lines" is a demo statement, not an engineering statement. The real value of a framework, the thing you're paying for in complexity and lock-in, falls into four categories. Be honest about which ones you actually need.
The four categories of value
- Abstractions. Prompt templates, retrievers, chains, memories, agents, tools. Names for patterns you would have built anyway. Value = shared vocabulary + tested implementations of the boring cases.
- Integrations. Adapters for 50+ model providers, 30+ vector stores, 100+ loaders. Swap OpenAI for Anthropic by changing one import. Value = you don't write the adapter, someone else does, and they keep it working.
- Observability hooks. Callback systems that emit every LLM call, every tool invocation, every retrieval, into LangSmith / Langfuse / Phoenix / OpenTelemetry. Value = tracing without instrumenting your code by hand.
- Ecosystem. Cookbooks, Stack Overflow answers, tutorials, hires who already speak the vocabulary. Value = network effect. Real, and often underrated in favour of «it's easy to roll my own».
What you are not buying
- Not correctness. Frameworks do not make your prompts better, your retrieval more relevant, or your evals more rigorous. All the Phase 2 content (L5–L9) still applies.
- Not performance. Every abstraction adds latency, every callback adds overhead, every retry layer hides cost. The «one-liner» is often six LLM calls.
- Not stability. LangChain has had multiple major API rewrites (classic Chains → LCEL → LangGraph for agents) in two years. Your code will rot unless you pin and upgrade deliberately.
A framework is a bet on ecosystem, not on lines of code. If the ecosystem's integrations and observability save you weeks, the tax on abstractions and breaking changes is worth paying. If you only need two integrations and control your own tracing, you are buying the tax without the benefit.
A 2×2 is enough to orient
There are roughly thirty serious frameworks. You do not need to learn them; you need to know which quadrant each sits in so you can find them when the problem appears.
How to read the quadrants
- LangChain / LangGraph. The Python default. LangChain for composing chains with LCEL; LangGraph for stateful graphs with explicit nodes, edges, and checkpointing. New agentic systems should start at LangGraph, not classic LangChain agents.
- LlamaIndex. Data-first. Better defaults for chunking, multi-document querying, query planning. «LlamaIndex is a framework for your data; LangChain is a framework for your pipeline.»
- CrewAI / AutoGen. Multi-agent collaboration patterns out of the box. Useful if your problem genuinely decomposes into roles; overkill otherwise.
- OpenAI Agents SDK / PydanticAI. Minimalist, typed, no graph engine. You write the loop; the framework gives you tool calling, retries, and structured output. Debugger-friendly.
- DSPy. Different axis entirely — it compiles prompts against a metric. You don't write prompts, you write modules; DSPy optimizes the prompt. Powerful for benchmark-driven work; niche for product work.
Start from the shape of your problem, not the popularity of the framework. Orchestration-heavy agentic loop ⇒ LangGraph or an Agents SDK. Document-heavy retrieval ⇒ LlamaIndex. Role-decomposed workflow ⇒ CrewAI. Benchmark-driven prompt tuning ⇒ DSPy. If none of those apply strongly, the honest answer is that you probably don't need a framework yet.
One agent, one tool-calling loop, four implementations
The running example for this lecture is a course-assistant agent with three tools, multi-turn memory, and a real tool-calling loop. The tools are deliberately different shapes so the agent has to choose:
The three tools
search_syllabus(query, k=3)— retrieval over the syllabus corpus. Unstructured input and output.get_office_hours(day=None)— deterministic structured lookup. Returns{"slots": [...], "email_fallback": ...}.compute_grade(project, homework, exam, participation)— pure function, weighted sum.
These three live in tools_shared.py as plain Python. Each framework wraps them in its own type system (OpenAI schema, @tool, FunctionTool, ToolNode), but the underlying code is shared. That keeps the comparison honest.
First, the from-scratch agent. Every moving part is visible: OpenAI's tool-call schema, an explicit dispatcher, a hand-written memory with summary + window compression, structured trace events. This is the standard from L3 + L4, written once more so the framework versions have something to compare to.
# agent_scratch.py (abridged -- see the file for the full version)
from openai import OpenAI
from tools_shared import OPENAI_TOOL_SCHEMAS, TOOL_DISPATCH
def run_turn(memory, user_msg, trace, max_steps=5):
client = OpenAI()
memory.add({"role": "user", "content": user_msg})
for step in range(max_steps):
r = client.chat.completions.create(
model="gpt-4.1-mini",
messages=memory.render(), # summary + recent window
tools=OPENAI_TOOL_SCHEMAS,
tool_choice="auto",
)
msg = r.choices[0].message
memory.add(msg.model_dump(exclude_none=True))
trace.emit("llm_end", tokens_in=r.usage.prompt_tokens,
tool_calls=len(msg.tool_calls or []))
if not msg.tool_calls:
return msg.content
for tc in msg.tool_calls:
args = json.loads(tc.function.arguments or "{}")
result = TOOL_DISPATCH[tc.function.name](args)
trace.emit("tool_end", name=tc.function.name)
memory.add({"role": "tool", "tool_call_id": tc.id,
"content": json.dumps(result)})
return "[max_steps]"Now the LangChain version. The loop is still there — we wrote it by hand on purpose, so the mapping to the scratch version stays one-to-one — but the tool schemas, the dispatch table, and the observability hook disappear into framework primitives.
# agent_langchain.py (abridged)
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
import tools_shared as ts
@tool
def search_syllabus(query: str, k: int = 3) -> list[str]:
"""Retrieve top-k relevant snippets from the course syllabus."""
return ts.search_syllabus(query, k)
# ... same for get_office_hours, compute_grade ...
TOOLS = [search_syllabus, get_office_hours, compute_grade]
def run_turn(session_id, user_text, tracer, max_steps=5):
llm = ChatOpenAI(model="gpt-4.1-mini",
callbacks=[tracer]).bind_tools(TOOLS)
history = get_history(session_id) # InMemoryChatMessageHistory
history.add_message(HumanMessage(content=user_text))
for _ in range(max_steps):
ai = llm.invoke([SystemMessage(content=SYSTEM), *history.messages])
history.add_message(ai)
if not ai.tool_calls:
return ai.content
for tc in ai.tool_calls:
result = TOOLS_BY_NAME[tc["name"]].invoke(
tc["args"], config={"callbacks": [tracer]})
history.add_message(ToolMessage(content=str(result),
tool_call_id=tc["id"]))What changed between the two versions
- Gone: OpenAI tool-schema JSON (generated from the function signature by
@tool); the manualTOOL_DISPATCHtable (replaced byTOOLS_BY_NAME); the ad-hocTraceclass (replaced byBaseCallbackHandler— the hook LangSmith / Langfuse / Phoenix all subscribe to). - Still there: the loop. The max_steps guard. The decision about what goes into the system prompt. The memory strategy. These are the things a framework cannot decide for you.
- Coupled to: LangChain's message types, tool format, and callback interface. When you later want LlamaIndex's retriever or LangGraph's checkpointer, you're re-typing.
Before you ship any agent, run it once with langchain.debug = True (or with a callback that prints every LLM and tool event). If you cannot immediately recognize every call the framework made, you don't yet understand your own agent, and you will not be able to debug it when it silently burns 60x the expected tokens in production.
Memory is a design decision, not a feature
In L3 we built four memory strategies by hand: full history, sliding window, running summary, and vector-retrieved. Each one is a different answer to the same question: when the context window isn't big enough, what do you throw away, and who decides? Frameworks give you labelled boxes for each strategy. They don't answer the question for you; they just make it easier to pick a box and harder to see inside it.
With a tool-calling agent the stakes get higher. Every turn of the loop (assistant message, tool_call, tool result) is a message in the history. A four-turn conversation with two tool calls per turn is already twelve messages, and tool results are often long JSON blobs. The agent's memory budget is dominated by tool I/O, not by user turns. If you don't actively compress, your second-turn prompt will be 10× the first.
Mapping L3 strategies to LangChain memory primitives
- Full history ↔
ConversationBufferMemory. Keeps every turn. Grows without bound. Cheapest to implement, most expensive to run. - Windowed ↔
ConversationBufferWindowMemory(k=N). Keeps last N turns. Bounded cost, but any fact older than N turns is gone. - Summarised ↔
ConversationSummaryMemory. Each turn, an LLM re-summarises history. Bounded cost, but every summary is a lossy compression and every summary call is another LLM call. - Summarised + windowed ↔
ConversationSummaryBufferMemory(max_token_limit=...). Recent turns verbatim, older turns summarised. The sensible default when you actually need long conversations. - Vector-retrieved ↔
VectorStoreRetrieverMemory. Past turns are embedded and retrieved by relevance to the current query. Scales to unlimited history; retrieval quality becomes the bottleneck.
If you came from L3, this list should feel familiar — you built all five. The framework's contribution is the names, not the strategies. In LangChain ≥ 0.3, the classic Memory classes are considered legacy; the modern API is RunnableWithMessageHistory wrapping a chain with a pluggable message store (in-memory, Redis, Postgres). The shape is the same: you still pick the compression strategy yourself.
LangGraph's model — memory as first-class state
LangGraph takes a different stance. State is an explicit typed dict on the graph, and a checkpointer persists it after every node. Threads (conversations) are just checkpoint streams keyed by thread_id. You can stop a conversation, restart it three days later on a different machine, and get the same state.
- Checkpointers shipped today: in-memory, SQLite, Postgres, Redis.
- Time travel: you can roll back to any checkpoint and re-run from there — invaluable for debugging agent failures.
- Human-in-the-loop: interrupt at a node, wait for user input, resume. Enabled because state is externalized.
The trade: you commit to expressing your agent as a graph. That is either «finally, an honest model of control flow» or «I just wanted to write a while loop,» depending on the problem.
# A sketch of the same chat agent with LangGraph checkpointing.
from typing import TypedDict, Annotated
from langgraph.graph import StateGraph, END
from langgraph.graph.message import add_messages
from langgraph.checkpoint.sqlite import SqliteSaver
class State(TypedDict):
messages: Annotated[list, add_messages]
def chat_node(state: State) -> State:
reply = llm.invoke(state["messages"])
return {"messages": [reply]}
graph = StateGraph(State)
graph.add_node("chat", chat_node)
graph.set_entry_point("chat")
graph.add_edge("chat", END)
app = graph.compile(checkpointer=SqliteSaver.from_conn_string("memory.db"))
# Same thread_id = same conversation. Different thread_id = fresh context.
config = {"configurable": {"thread_id": "user-42"}}
app.invoke({"messages": [("user", "Remember my dog is named Juno.")]}, config)
app.invoke({"messages": [("user", "What's my dog's name?")]}, config)Picking a memory strategy is picking which failure mode you are willing to ship. Full history fails with latency and token-budget explosions. Window fails by forgetting. Summary fails by distortion. Retrieval fails by relevance miss. Every production agent has one of these failure modes; the only question is whether you chose it or the framework's default did.
Tracing is not optional; the question is who owns it
In L9 we said: the failures of AI systems are semantic, probabilistic, and slow to surface. A trace — the full call graph of every LLM, tool, retriever, and memory read/write for a single request — is the smallest observability artifact that lets you reason about those failures. If you cannot produce a trace for last Tuesday's 3am complaint, you cannot debug it.
For tool-calling agents this matters double. A single user turn can fan out to three LLM calls and four tool calls, each with its own latency and cost, and most agent bugs («it called the wrong tool», «it looped forever», «it ignored the tool result») are invisible in the final assistant message. You can only see them in a trace.
Frameworks do not relieve you of needing observability. They change the answer to the question who emits the trace, and to whom. Four tiers, in increasing order of ecosystem commitment:
Tier 1 — Hand-rolled
Your monitoring.py from L9. structlog, a trace_id, a @traced decorator. Full control, zero dependencies, zero ecosystem. Works perfectly until you want to visualize 10,000 traces, correlate across services, or run LLM-as-judge on sampled traffic — then you're rebuilding a trace store.
Tier 2 — Framework callbacks
LangChain emits events through BaseCallbackHandler: on_llm_start, on_llm_end, on_tool_start, on_retriever_end, and so on. You subscribe and do whatever you like — log to stdout, push to your own store, forward to a vendor. This is the hook that every higher tier builds on.
from langchain_core.callbacks import BaseCallbackHandler
class CostCallback(BaseCallbackHandler):
def __init__(self):
self.tokens_in = self.tokens_out = 0
def on_llm_end(self, response, **kwargs):
usage = response.llm_output.get("token_usage", {})
self.tokens_in += usage.get("prompt_tokens", 0)
self.tokens_out += usage.get("completion_tokens", 0)
cb = CostCallback()
chain.invoke("...", config={"callbacks": [cb]})
print(cb.tokens_in, cb.tokens_out)Tier 3 — Vendor-integrated tracing
- LangSmith — first-party for LangChain / LangGraph. Enable with one env var (
LANGCHAIN_TRACING_V2=true), get traces, eval runs, prompt versioning, and dataset management in the same UI. Paid for teams, commercial. - Langfuse — OSS, self-hostable. Framework-agnostic via decorators (
@observe) and OTel ingestion. Strong alternative if you want a vendor UX without the vendor lock-in. - Phoenix (Arize) — OSS, strong on RAG tracing visualization and eval loops. Good default when RAG is the bulk of your system.
- Logfire (Pydantic) — OTel-native. Strong Python DX, strong typing, pairs naturally with PydanticAI.
- Helicone — proxy-based. Point your model client at the Helicone proxy; no code changes, logs everything. Easiest to adopt, hardest to extend.
Tier 4 — Open standards (OpenTelemetry)
The OpenTelemetry GenAI semantic conventions define a standard vocabulary for LLM traces: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.finish_reasons, etc. Libraries like OpenLLMetry (Traceloop) auto-instrument OpenAI, Anthropic, LangChain, LlamaIndex and others to emit OTLP spans. Any OTLP-compatible backend (Langfuse, Phoenix, Datadog, Grafana Tempo, Honeycomb) ingests them.
This is the tier you want if you already run OTel for the rest of your stack, or if you explicitly do not want your tracing to be coupled to any single vendor.
from traceloop.sdk import Traceloop
Traceloop.init(app_name="rag-demo") # auto-instruments LangChain, OpenAI, etc.
# Every chain.invoke() now emits OTLP spans to your configured backend.
# Same code, same traces, swap backends with one env var.Pick your observability layer before your framework, not after. If your organization standardizes on OTel, that rules out anything whose tracing story is a proprietary callback with no OTel exporter. If you don't care and want fastest time-to-traces on LangChain, LangSmith is the honest answer. Observability is where "which framework" most often becomes "which framework's vendor do I want to depend on."
The same agent in LlamaIndex and LangGraph
The notebook contains the full runnable versions; here is the shape of each so you can recognize the pattern.
# agent_llamaindex.py (abridged) -- the framework owns the loop
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import FunctionTool
from llama_index.core.memory import ChatMemoryBuffer
import tools_shared as ts
tools = [FunctionTool.from_defaults(fn=ts.search_syllabus,
name="search_syllabus", description="..."),
FunctionTool.from_defaults(fn=ts.get_office_hours,
name="get_office_hours", description="..."),
FunctionTool.from_defaults(fn=ts.compute_grade,
name="compute_grade", description="...")]
agent = FunctionAgent(tools=tools, llm=Settings.llm, system_prompt=SYSTEM)
memory = ChatMemoryBuffer.from_defaults(token_limit=2000)
await agent.run(user_msg="When are your office hours?", memory=memory)
await agent.run(user_msg="And what's the late-homework rule?", memory=memory)The tool loop, the message bookkeeping, and the multi-turn memory are all inside FunctionAgent.run. The trade is that when you want to interpose between a tool decision and its execution — for a guardrail, a budget check, a confirmation prompt — you are reading framework source.
# agent_langgraph.py (abridged) -- explicit graph, checkpointed state
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode, tools_condition
from langgraph.checkpoint.sqlite import SqliteSaver
class State(TypedDict):
messages: Annotated[list, add_messages] # add_messages = append reducer
llm = ChatOpenAI(model="gpt-4.1-mini").bind_tools(TOOLS)
def agent_node(state): return {"messages": [llm.invoke([SYSTEM, *state["messages"]])]}
graph = StateGraph(State)
graph.add_node("agent", agent_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.set_entry_point("agent")
graph.add_conditional_edges("agent", tools_condition,
{"tools": "tools", END: END})
graph.add_edge("tools", "agent")
app = graph.compile(checkpointer=SqliteSaver.from_conn_string("agent.db"))
app.invoke({"messages": [HumanMessage(content="...")]},
config={"configurable": {"thread_id": "session-1"}})LangGraph earns its complexity when control flow has real branches — tool result → check confidence → maybe re-plan → maybe ask a clarifying question. The compiled graph also gives you time-travel debugging for free: app.get_state_history(config) lists every checkpoint, and you can resume from any of them. That is the killer feature for stateful agents you actually have to debug in production.
The framework decision, made honestly
Use a framework when
- You need to support multiple model providers and/or vector stores from day one.
- You want observability to «just work» and are happy to adopt the framework's vendor (LangSmith, LlamaIndex + Phoenix, etc.).
- You're prototyping and optimizing for velocity over clarity — you'll rewrite once the product shape is known.
- Your team is large enough that a shared vocabulary is worth the tax.
- You need a pre-built integration (Slack, Gmail, Notion, 100+ loaders) and don't want to maintain it yourself.
Avoid a framework when
- Your core loop is small and stable — a three-step agent does not need a graph library.
- You have hard latency or cost budgets and cannot afford hidden retries or callback overhead.
- You need to reason about the exact token sequence the model sees (regulated domains, safety-critical systems).
- You cannot tolerate a breaking change every six months. Frameworks are young and their APIs move.
- Your observability is already OTel and the framework's tracing story is proprietary-callback-only.
The framework tax — a checklist before you commit
- Can I print the exact prompt that was sent to the model, without reading framework source?
- Can I step through the retry and fallback logic in a debugger?
- Can I pin a version and not be forced to upgrade for at least 6 months?
- Is my observability independent of the framework (OTel) or coupled to it (proprietary callbacks)?
- If this framework were abandoned tomorrow, how many engineer-days to rip it out?
Four «yes» answers out of five, and the framework is probably a net positive. Two or fewer, and you're buying tax without benefit.
You already built most of what LangChain offers, by hand, in seven weeks. That is not wasted work — it is the only way to have an informed opinion about what a framework is hiding. Choose a framework the way you'd choose a dependency for any other system: because the ecosystem saves you more than the abstraction costs you, because you can read its source when it misbehaves, and because your observability story does not live inside its walls.
One week, one rewrite, one decision memo
The assignment
- Pick one component of your running project (the retrieval step, the planning loop, the memory layer).
- Rewrite it with LangChain, LlamaIndex, or LangGraph. Keep both versions in git under separate entry points.
- Measure: lines of code, p50 and p95 latency, cost per call on a fixed eval set, and a 3-line qualitative note on debuggability («can I see the exact prompt?»).
- Add LangSmith or Langfuse tracing to the framework version. Share three interesting traces.
- Write a one-page decision memo: would you ship the framework version? Why? Apply the checklist from §7.
There is no «right answer» to this homework. The point is that after doing it, you will have an opinion you can defend.