Prompt regression testing: did that change make things worse?

You change a prompt, swap a model or reword a tool description. You try a few examples and they look fine. What those examples cannot tell you is whether something that used to hold now fails more often.

What counts as a prompt regression?

Any change that makes a behavior your app used to get right fail more often: a rule broken, a tool skipped, a reply in the wrong language.

The change need not be to the prompt. A model swap counts, whether the old model was deprecated, a cheaper one cuts the bill, or a newer one scores worse on your own cases. So do a reworded tool description, a new retrieval index and a new step in the router. OpenAI’s hosted Evals becomes read-only on 31 October 2026 and shuts down on 30 November 2026 (OpenAI), so some teams are moving their regression suites this quarter as well.

Why doesn’t re-running a few examples work?

Because the same prompt gives different replies from one try to the next, so a handful of tries cannot separate a real change from noise.

If 2 of 10 conversations fail, the true failure rate is anywhere from 6% to 51% (the 95% Wilson interval). A change that moves a rule from failing one conversation in five to one in three cannot show up in ten tries before and ten after. An average across all questions hides even more: it can hold steady while one rule that never failed starts failing.

How do you test a change properly?

Replay the exact failures, run cases the change was not aimed at, and decide with a count and a test.

1. Keep the failing conversations. Save the user’s side of each conversation that failed, with the reply that failed and the account state it ran under.
2. Replay them word for word after the change. Same user turns, same account state, same starting data. Before and after then compare the same cases, which takes the luck of drawing new cases out of the comparison.
3. Decide with an exact test. “3 of 3 failed before, 0 of 3 after” can be tested. When the counts are too small to decide, say “cannot tell yet” and replay more, rather than calling it fixed.
4. Run a held-out share. Cases the change was not written against show what it broke elsewhere. Keep them out of reach of whoever, or whatever, made the change.
5. Read rule by rule. Compare each check before and after, not the overall pass rate.
6. For a model swap, run everything on both. The whole suite on the old model and the new one, compared rule by rule, on your own key.

What does that look like with Cortad?

Cortad keeps every version of your AI’s behavior, so each change is measured against the one before it, and it tells you what else changed or regressed, not only the thing you meant to fix. Its verify replays a finding’s failing conversations word for word against your edit and says one of four things, with the counts behind it.

Cortad makes your AI agent’s behavior measurably better, with no evals to write or maintain. It learns from your code how your product is meant to be used, sends users through every surface the way your real users would, and asks every reply up to 100 behavioral questions. In production it shows every conversation in real time, so you see how your agent behaves for every kind of user. And it gets better with time: it studies how your real users actually behave, so its users, its metrics and its understanding of your product sharpen with every conversation, and your coding agent uses all of it to make your product better.

Verify of finding:1 at apps/api/src/agent/prompt.ts:41.
The failure is gone on its own trials: none of 3 replays failed, against 3 of 3 trials in the run it was found in. Held out, the same question in situations you cannot see: 1 of 2 trials failed before, 0 of 2 replayed after.

Verify of finding:1 at apps/api/src/agent/prompt.ts:41.
Cannot tell yet: 1 of 3 replays failed, against 3 of 3 trials in the run it was found in; 1 more clean replay would show it failing less often. Replaying the 3 failing trials again now, round 2.

Verify of finding:4 at apps/api/src/agent/system.ts:12.
The failure stayed: 4 of 4 replays failed, against 4 of 4 trials in the run it was found in; the same question fails on 3 of 18 trials elsewhere in your app, so these still stand out beyond chance. The replay of trial t-0b19 said at reply 1: "Sure! Let's solve this together."

The fourth answer, “shows less often but is not gone”, comes with the counts and a quoted reply that still failed. These examples are from the tool’s own reference; a trial is one simulated conversation and a situation is one kind of user with one goal.

After the fixes, a second run measures the whole app again, and each check is paired with the previous run on the conversations both played:

The next run is compared with this one, run 8f2a1c4e: each finding's question is paired on the trials both runs play, reply by reply where both read the same replies, so its move is counted, not guessed.

To test a model swap, change the model in your code and run again. Cortad always uses the model and key your app uses; it never swaps them for its own.

How often should I run it?

After a change to prompts, tools, models or retrieval, before you merge.

• A verify replays only one finding’s failing conversations and a few held-out ones, so it is the step to repeat while you iterate on a fix.
• npx cortad stick adds a hook for Claude Code and Codex that names the prompt and tool files that changed after each edit, so your agent knows when a check is due.

What do other tools offer for comparing two versions?

Most compare a candidate with a baseline row by row, and none of the tools below documents a test of whether the difference is more than chance. As of 30 Sep 2026: