LLM Evolution Lab

Source #9 — "You didn't change anything. The model did."
What is this playground about? (click to expand)

The scenario: Your system uses GPT-4 via API. Same prompt, same test set, same code. But between March and June, the model provider updates the weights behind the same endpoint. Your scores shift — sometimes catastrophically.

Why it's unique to AI: Traditional software doesn't change behavior when you don't touch the code. LLM-based systems do. The model behind your API is a dependency you don't control.

The lesson: Pin model versions. Run regression tests on a schedule. Treat every version change as a new deployment.

97.6% → 2.4%
GPT-4 accuracy on prime number identification
March 2023 → June 2023. Same prompt. Same task.
Chen et al., "How Is ChatGPT's Behavior Changing over Time?" (2023)
arxiv.org/abs/2307.09009
Let that sink in. A task that GPT-4 got right 97.6% of the time in March dropped to 2.4% three months later. No code changed. No prompt changed. The model was updated behind the same API endpoint.

How would you have caught this if you weren't re-running evals every week?

Simulated: Your System's Version History

A realistic timeline of what happens to a production system that depends on a model API. Each event is something that has actually happened to real teams.

Jan 15, 2025
Initial deployment — gpt-4-0125-preview
Correctness: 4.2. Passed all regression tests. Team is confident.
Feb 20, 2025
Routine re-evaluation
Correctness: 4.1. Within noise bounds. No action needed.
−0.1
Mar 12, 2025
Provider announces model update
OpenAI updates gpt-4 endpoint. Changelog mentions "improved instruction following" and "reduced hallucinations." Sounds good, right?
Mar 18, 2025
Post-update evaluation
Correctness: 3.6. Helpfulness: 3.2 (was 4.0). The "improved instruction following" changed how the model handles ambiguous queries — our prompts relied on the old behavior.
−0.6 Correctness, −0.8 Helpfulness
Mar 19, 2025
Emergency: pin to snapshot version
Team switches to gpt-4-0125-preview (pinned). Scores recover to 4.1. But the pinned version has a deprecation date of June 30.
+0.5 (back to baseline)
Apr–May 2025
Prompt re-engineering for new model
Team spends 3 weeks adapting prompts to the new model behavior. Correctness on new model: 4.0. Close enough. Ship it before the deprecation.
Jun 30, 2025
Deprecation deadline
Old model version is retired. If the team hadn't adapted, they'd be forced to switch cold — with no time to test.
Aug 5, 2025
Silent regression
No announcement, but routine eval shows Tone dropped from 4.3 to 3.7. Investigation: a minor model patch changed formatting behavior. JSON outputs now include markdown fences that break the parser.
−0.6 Tone

Model Version Roulette — Simulation

Click "Simulate" to see how scores change across 6 model version updates over a year. Each version is a real model update — some improve things, some break things. You don't know in advance which is which.

Simulate Model Updates — Generates a randomized 12-month timeline with 6 model version changes. Each version randomly shifts scores up or down for each metric (some updates help, some hurt). You'll see a line chart of Correctness across versions. Each click re-randomizes — the unpredictability itself is the point.

Documented Real-World Cases

SourceFindingImpact
Chen et al., 2023 GPT-4 prime number identification 97.6% → 2.4% in 3 months
Chen et al., 2023 GPT-3.5 code generation (LeetCode) 52.0% → 10.0% between March and June
Deepchecks, 2024 Production analysis across companies 22.9% of model updates cause prompt regressions
Deepchecks, 2024 Worst observed regressions >20% accuracy drop from single model update
Industry reports Model deprecation forcing migration gpt-4-0613 → gpt-4o: subtle behavior changes

How to Fix It

StrategyHow it works
Pin model versionsUse dated snapshots (gpt-4-0125-preview) not aliases (gpt-4). The alias can change under you.
Scheduled regression testsRun your eval suite weekly on a fixed benchmark. Catch drops early, before users do.
Maintain a fallbackIf the new version regresses, roll back to the pinned version while you adapt prompts.
Version-as-deploymentTreat every model version change as a new deployment requiring full re-evaluation.
Multi-provider strategyIf one provider regresses, you can switch. Requires provider-agnostic prompt design.

Does more data help? No. More data tells you more precisely what this version does on this date. When the version changes, your measurements are stale. The fix is continuous monitoring and version discipline, not more data.