Model Drift

Model drift is the degradation of an AI system's real-world performance over time — either because the underlying model itself changed (a provider ships an update behind what looks like the same stable name) or because the data the model encounters in production has shifted away from what it was originally built and evaluated on.

A system that scores well on launch day is not guaranteed to keep scoring well, and the failure can happen without a single change on your own side. Two different mechanisms travel under the same "drift" name, and it's worth telling them apart, because the fix for each is different.

How it works

The first mechanism is model drift proper: the model's weights or serving configuration changed underneath what looks like a stable reference — a floating alias like "latest" now points at a different version than it did last month, so the exact same prompt behaves differently even though nothing in your own code moved. The second is data drift, or concept drift: the model itself is genuinely unchanged, but the world it's operating on moved — user behavior shifted, a product catalog changed, an attacker's tactics evolved — so the same model is now facing inputs meaningfully different from what it was tuned or evaluated against. Either way, the system degrades quietly, with no error message and no failed deploy to point to.

Why it matters

Without a scheduled, repeated eval run, drift is invisible until someone downstream notices real failures — support tickets, wrong answers reaching users, a metric that's been sliding for weeks before anyone looked closely enough to notice. That's a far more expensive way to find out than catching a score drop in a scheduled test. Pinning an explicit model version instead of trusting a floating alias, and re-running the same eval suite on a fixed cadence rather than only at launch, are what convert "we assume it still works" into "we checked, and it does."

A worked example

A team runs a fixed 500-example eval suite against their production model every month. Month one: 92.4% pass rate. Month two, with no code changes on their side: 84.1% — an 8.3-point drop, a 9% relative regression. Investigation traces it to a provider-side model update served behind the same "latest" alias the team had been calling. Pinning to an explicit model version, and re-running the same eval suite before switching to any newer version on purpose, turns that kind of surprise into a decision the team makes deliberately instead of one that happens to them.

How Miatz teaches it

Learners set up a small scheduled eval suite against a live model endpoint and watch it run unattended across sessions, so a drift event — a real score change with no code change behind it — is something they catch happening rather than a hypothetical they're told about. The lesson sits directly on top of llm-evals: drift detection is simply the same eval discipline run on a calendar instead of only at ship time.