Mastering Individual Calls to LLMs

01 — AI as a Component

AI is embedded inside your solution — a model call in the pipeline. You design around its strengths and limitations: prompts, guardrails, evaluation, fallback logic.

02 — AI as a Coding Agent

AI helps you write, review, and refactor code. It accelerates development but the output is traditional software — deterministic, testable, fully under your control.

03 — AI as a User

AI consumes and produces information through your system. It calls your APIs, reads your docs, and acts on your data. You need to design for this new kind of consumer.

System Message

Purpose: Defines the model's persona, tone, constraints, and output format.

Convention: You resend it every call, but keep it the same across turns. The API is stateless.

Tip: Put guardrails and formatting rules here so they can't be overridden by user input.

User Message

Purpose: The actual task, question, or input data for this specific request.

Changes: Every turn. Each new request or follow-up goes here.

Tip: Include few-shot examples, context, and delimiters in the user message for best results.

Low Temperature (0.0 – 0.3)

Behavior: Picks the most likely token almost every time. Output is almost consistent and repeatable.

Best for: Classification, data extraction.

High Temperature (0.7 – 1.5)

Behavior: Flattens the probability distribution, giving less-likely tokens a real chance. Output varies across runs.

Best for: Creative writing, brainstorming, dialogue, exploration, generating diverse options.

Direct Instruction

The baseline. Clear, literal instructions for simple tasks.

Summarize for a VP:
- exactly 5 bullets
- include 1 risk + 1 recommendation
Text: [paste]

If output drifts, add an output clamp.

Role Prompting

Shapes tone, vocabulary, depth. Does NOT improve factual correctness.

Act as a senior SRE.
Return:
1) Findings (major)
2) Missing evidence
3) Follow-ups (top 5)

Gemini takes personas very seriously — test for conflicts.

Scope Discipline

Set explicit boundaries. "Fix this function" without scope may trigger a full refactor.

Implement EXACTLY this and nothing else. If ambiguous, pick simplest valid interpretation.

Structured Formatting

Delimiters prevent the model from confusing instructions with data. Separate instructions from data. Put long data first, instructions last.

The technique

• Zero-shot: No examples. Relies entirely on the model's training.
• One-shot: One input–output pair shows the pattern.
• Few-shot (3–5): Multiple examples lock in format, tone, and edge cases.

Best practices

• Diverse examples: Cover typical cases AND edge cases.
• Consistent format: Use identical delimiters (Input: / Output:) in every example.
• Order matters: Place the most similar example last, closest to the actual task.

Why it works

LLMs generate tokens sequentially. When you force intermediate reasoning steps, each subsequent token is conditioned on better context.

Key insight: The model "thinks" by writing. No writing = no thinking. CoT gives the model scratch space.

CoT variants

• Zero-shot CoT: Add "Let's think step by step" to trigger reasoning.
• Structured CoT: Prescribe steps: "First analyze X, then evaluate Y, finally conclude Z."
• Few-shot CoT: Provide examples that include reasoning traces, not just answers.

UI: Best-Effort

Extract into JSON with exactly these keys:
party_name, jurisdiction,
effective_date, termination_summary

If unknown, set null.
No extra keys.

API: Enforced

OpenAI: text.format with strict: true → guaranteed shape

Anthropic: Prefilling + tool-use schemas → enforced structure

Answer → Review → Re-answer

A single-call self-critique: Draft your answer, critique it for errors and gaps, then revise. All three steps in one API call — no orchestration needed.

Especially powerful for factual tasks, math, and any output that can be objectively verified.

Evaluation-First

Define success before proposing solutions. The prompting equivalent of test-driven development.

Before proposing, do this:

1. Define success metrics
2. Define baseline + thresholds
3. Propose 3 interventions
4. Predict impact + validation plan

The loop

1. Call 1: Generate initial answer.
2. Call 2: "Here is your previous answer. Rate it 1–10 on [criteria]. What would you improve?"
3. Call 3: "Apply those improvements and produce the final version."

Repeat calls 2–3 until quality plateaus or budget is reached.

When to use

• High-stakes output: Legal drafts, medical summaries, customer-facing content.
• Latency-tolerant: Batch processing, async workflows where extra calls are acceptable.

Common failures

• Rate limits (429): Back off exponentially. Use jitter to avoid thundering herds.
• Timeouts: Set aggressive timeouts (30–60s). Retry with the same prompt; non-determinism means a retry may succeed.
• Malformed output: JSON didn't parse? Retry once. If it fails again, fall back to a simpler prompt or model.

Robustness patterns

• Retry with backoff: 1s → 2s → 4s with random jitter. Max 3 retries for most use cases.
• Model fallback: If GPT-4o fails, fall back to GPT-4o-mini. Graceful degradation beats total failure.
• Validation layer: Always validate output before using it. Schema checks, length checks, safety checks.

Stateless agent

Each call is self-contained. Send a system message + user message, get a response. No history, no context. Simple, cheap, predictable.

# Every call starts fresh
response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"},
    ],
)

Stateful agent

You maintain the conversation history and resend it with each call. The model sees the full context and can reference earlier turns.

# You manage the conversation history
history = [
    {"role": "system", "content": "You are a helpful assistant."},
]

def chat(user_message):
    history.append({"role": "user", "content": user_message})
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=history,
    )
    assistant_msg = response.choices[0].message.content
    history.append({"role": "assistant", "content": assistant_msg})
    return assistant_msg