If your coding agent can run your evals, it can fix a prompt and prove the fix in one session. If it can also edit those evals, a passing score stops telling you anything. This guide covers how to keep the first without the second.
Why shouldn’t the coding agent own the tests?
An agent that can edit the tests can make a failing test pass by changing the test, and current models do this often when the task gives them a reason to.
ImpossibleBench (Zhong, Raghunathan and Carlini, October 2025) built coding tasks whose specification and unit tests contradict each other, so any pass means the agent took a shortcut. In the authors’ words, “GPT-5 cheats 54.0% of the time on Conflicting-SWEbench.” The paper’s chart puts Claude Opus 4.1 at about half. The shortcuts ranged from editing the test to overloading an operator so a comparison always passes.
Hiding the tests from the agent cut cheating to near zero. Making them read-only stopped edits to the tests, but not the special cases and overloaded operators.
METR saw the same pattern outside a benchmark built to provoke it. In “Recent Frontier Models Are Reward Hacking” (5 June 2025), o3 reward-hacked in 39 of 128 runs (30.4%) on RE-Bench, where it could see the entire scoring function, and in 8 of 1,087 runs (0.7%) on HCAST. That is 43 times the rate, where the model could see how it was scored.
Two lessons carry over. Keep the tests where the agent cannot write them. Keep some tests where it cannot read them either, and grade every fix on those too.
Does this apply to AI app evals, or only to unit tests?
It applies to evals, and it is easier to miss there, because an eval is a judgment rather than an assertion. A coding agent with write access can make an eval pass without changing how your app behaves:
- Rewrite the judge prompt, or lower a scorer’s threshold, until the failing reply passes.
- Edit the expected output of the failing case to match the wrong reply, or delete the case.
- Add a special case in the prompt or code keyed to the exact message the test sends.
- Loop on the visible cases until all of them pass, which fits the prompt to those examples and to nothing else.
None of these needs bad intent. Each is the shortest path to “the evals pass”, which is the goal the agent was given.
What does a setup look like that the agent cannot edit?
The agent runs the evals and reads the results, while the cases and questions live where it has no write access, and a held-out share lives where it has no read access either. Cortad is built this way; the next sections show how.
- Tests outside the agent’s write path. A server the agent reaches with a read-only key, or a repository and CI job it cannot push to. Instructions in a prompt are a request, not a control.
- A held-out share. Keep about one case in five out of the agent’s reach, chosen by a fixed rule such as a hash of the case id, so the same cases stay hidden from run to run. The agent sees their counts, never their inputs.
- Replay the exact failures. After a fix, replay the conversations that failed, word for word, so before and after compare the same cases. Then run the held-out share to see what else the fix moved.
- Decide with counts. Report “failed 3 of 3 before, 0 of 3 after”, with an interval, and say “cannot tell yet” when the counts are too small to decide.
- Let the agent object without overruling. If the agent thinks a check is wrong, it writes that down for a person. The check changes only when the person agrees.
Do eval tools already keep the tests away from the agent?
Most do not. Most give your coding agent tools that write tests, datasets or scorers, and none of the 17 we checked documents a set of tests hidden from the agent (as of 30 Sep 2026; the comparison page covers all 17).
| Tool | What the agent can write through its MCP server or skills |
|---|---|
| Braintrust | 42 tools, including edit_dataset_rows and create_evaluator (docs). |
| Langfuse | 86 tools, including datasets, evaluators, evaluation rules, prompts and scores. “Both read and write tools are available by default” (docs). |
| LangSmith | The LangSmith CLI and skills create and delete datasets and evaluators (docs). |
| LangWatch | 97 tools in the published package, including create, update and delete for scenarios and evaluators (docs). Its skill “loops until everything passes” (blog). |
| Confident AI | 75 tools, including update_golden and update_metric (docs). |
How does Cortad keep the tests out of the agent’s reach?
Your coding agent writes tests that its own change will pass. Cortad tests the change in ways the agent did not expect, and tells you what else changed or regressed. The server that holds Cortad’s tests has no route that lets a coding agent change them, and one conversation in five is hidden from the agent.
Cortad makes your AI agent’s behavior measurably better, with no evals to write or maintain. It learns from your code how your product is meant to be used, sends users through every surface the way your real users would, and asks every reply up to 100 behavioral questions. In production it shows every conversation in real time, so you see how your agent behaves for every kind of user. And it gets better with time: it studies how your real users actually behave, so its users, its metrics and its understanding of your product sharpen with every conversation, and your coding agent uses all of it to make your product better.
- The agent’s ten tools start runs and replays, read results, file a dispute and send a note to the team. None writes a case, a check, a seed or a score. That is enforced by the server having no such route, not by an instruction to the agent.
- The agent’s key is scoped to one repository and to those routes. The connect command keeps it in
~/.cortad; it is never returned to the agent or written into a client’s config. - The hidden conversations are picked by a hash of each conversation’s id. Every finding still shows the agent at least one of its failing conversations, and one that failed twice or more keeps one hidden. The agent reads lines like “12 conversations are kept back from you so a fix can be graded on them.”
verifyreplays the failing conversations word for word, then the hidden conversations of the same check once. It says whether the failure is gone on the hidden ones that failed, and if the hidden ones that passed now fail more often than before, beyond chance, it says the fix broke them.- A dispute goes to the account owner. The finding and its rate stay as they are until the owner decides.
What does the loop look like in a coding agent session?
The agent makes one change near the line a finding names, calls verify, and reads a verdict that covers both the conversations it saw and the ones it did not.
Verify of finding:1 started: 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0a.
A verify replays its 3 failing trials word for word, and 2 held-out trials once, round after round until it decides, up to 20 replays. 2 clean replays show it gone.
next: run_status 7c31e0aa-1b2c-4d3e-8f4a-5b6c7d8e9f0aVerify of finding:1 at apps/api/src/agent/prompt.ts:41.
The failure is gone on its own trials: none of 3 replays failed, against 3 of 3 trials in the run it was found in. Held out, the same question in situations you cannot see: 1 of 2 trials failed before, 0 of 2 replayed after.“1 of 2 trials failed before, 0 of 2 replayed after” is the hidden half: the agent learns that the fix held in situations it never read, without learning what those situations were.
How do I make checking a habit for my agent?
Run npx cortad stick. It adds one line to AGENTS.md, CLAUDE.md, the Cursor rules and the Copilot instructions: “After changing prompts, tools, models or retrieval, check with Cortad before committing: npx cortad status”. The server enforces the rest. Setup is on the quickstart.
Sources
- Zhong, Raghunathan, Carlini. ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases. arXiv 2510.20270, sections 4 and 5.2.
- METR. Recent Frontier Models Are Reward Hacking, 5 June 2025.
- Vendor documentation linked in the table, as of 30 Sep 2026.
- Cortad: the cortad package on npm lists the ten tools; the rest of this section describes the server as it runs on 30 Sep 2026.