Prompt Regression Testing

Prompt regression testing is the practice of rerunning a fixed suite of scored test cases against every prompt edit, model swap, or pipeline change before it ships, comparing the new score against the previous baseline and blocking the change if quality drops — the same discipline as a software regression suite, applied to prompts instead of code.

A prompt edit that fixes the one failing example a developer noticed can silently break ten other cases nobody rechecked, because a prompt has no type system and no compiler to catch that kind of regression. The only way to know is to actually rerun a broad set of cases and look at the scores, every time, not just when something feels risky.

How it works

A prompt regression suite needs three things in place before it is useful: a fixed dataset of representative inputs, ideally pulled from real production cases and past failures rather than invented from scratch; a scoring method for each case, whether exact match, a rubric graded by an LLM judge, or a code-level check like schema validity; and a stored baseline score from the currently deployed prompt to compare every candidate change against.

The workflow that makes this a regression test rather than a one-off eval is automation and a gate: the suite reruns automatically on every proposed prompt change, ideally in the same CI pipeline that runs code tests, and a drop in aggregate score beyond some tolerance blocks the change from shipping rather than merely getting logged for someone to notice later. Without the gate, a regression suite is just a report nobody is required to read before merging.

Why it matters

Prompts change constantly and quietly — a tweaked instruction, an added rule, a new example swapped in — and each change is a real deployment with real risk, even though it often does not go through the same review rigor as a code change. Prompt regression testing is what catches the case where an improvement for one scenario is a regression for five others, before a customer finds it instead of a test suite.

A worked example

A team's support-triage prompt has a regression suite of 150 saved tickets with known-correct labels, and the current prompt passes 142 of them, a 94.7 percent baseline. An engineer adds a new instruction to handle a category of ticket the prompt was missing, reruns the full suite, and the new prompt passes only 136 of 150, 90.7 percent, a 4-point drop. The new instruction fixed the category it targeted but broke formatting on several previously passing tickets in an unrelated category. The regression gate blocks the merge; without it, the fix would have shipped alongside a quieter, broader breakage that would only surface once customers hit it.

How Miatz teaches it

Prompt regression testing is where the eval-suite habit from the LLM evals module becomes a hard gate rather than a nice-to-have: capstone projects require a stored baseline score and a passing regression run as a literal condition of merging any prompt change, mirroring exactly the kind of CI gate a serious engineering team holds its own code to in production.