The usual first test is to paste the system prompt into a playground and try a few questions. That tests how a model reads one prompt. Your users reach something else: an endpoint that builds the prompt from their account, calls tools, retrieves passages and remembers the last turn.
Why isn’t testing the prompt enough?
Because the reply your users get is assembled by your app, and a prompt tested on its own skips every step your app adds.
These are examples from the reference output of Cortad’s own tool, run against a tutoring app. None of them shows up when the prompt is tested alone:
- A router sent a refund question to a model call with no tool, and the reply stated a refund policy the product does not publish.
- A search tool returns an empty list when its index is down, so the model cannot tell “nothing found” from “search failed”.
- The student’s name was written into the system prompt, and markup from that prompt (
</student_profile>) leaked into a reply. - The model client was created with no timeout, so one slow reply held the request open.
Fix in the routing: the reply was written at apps/api/src/agent/answer.ts:58 with no tool after apps/api/src/agent/route.ts:22 chose where the ask went; the fix is that choice, so this ask reaches a step with the tool.What should a pre-launch test cover?
Every endpoint that reaches a model, called the way your own client calls it, by signed-in users in the states your real users will be in, for more than one turn.
- Every endpoint that reaches a model. List them from the code, not from memory. The endpoint nobody remembered is where the untested prompt lives.
- Called the way your client calls it. Same route, same body shape, same sign-in. A test that calls the model SDK directly skips your route handler and everything in it.
- Signed-in users in different states. Free and paid, new and returning. Some rules apply to one state only, such as “Tell a parent what the plan costs only from the pricing page.”
- More than one turn. Send a follow-up in the same conversation. Memory, summaries and tool results from the first turn only matter on the second.
- Each rule your prompts state. If the system prompt says “Never state a refund policy the product does not publish”, some test should ask about refunds and check that sentence.
- What actually ran. Record which tools were called and which passages reached the model for each reply. A good answer that skipped the tool it needed is luck, not behavior.
- A user who pushes back. A simulated user who rephrases after a refusal, or presses for a discount, finds rules that hold only on the first ask.
- Routes that should refuse. Send a request with no sign-in to each route that needs one. It should be refused.
- A copy of your data. Run against a copy of the database, so test users and conversations do not end up in the real one.
How many conversations do I need before I trust a rate?
Enough that the interval around the rate is narrow enough to act on. With a handful of conversations it is not.
| Failed | The true rate could be anywhere from |
|---|---|
| 1 of 3 | 6% to 79% |
| 2 of 10 | 6% to 51% |
| 9 of 12 | 47% to 91% |
| 10 of 50 | 11% to 33% |
| 20 of 100 | 13% to 29% |
Report every rate with its count and its interval. A rule that broke once in three tries could break in most conversations or in almost none. Cortad prints both on every finding, and marks a finding settled only when there are enough conversations to say, at 95% across all of a run’s questions together, that it breaks in at least one conversation in five.
Where do the test users come from if I have no users yet?
From your app’s own code and purpose. Cortad builds user profiles that use your product the way you mean it to be used: across every surface, doing the things your product exists for, and calling your tools along the way.
- The rules your prompts state become the questions.
- The journeys your product exists for, such as a billing question or a first lesson, become the goals of those users.
- The account states your code branches on become who those users are.
If you already have evals, Cortad learns from those too. After launch, it studies how your real users actually behave, so its users get closer to them and its metrics sharpen with every conversation, and your coding agent uses that understanding to make your product better.
What should block a launch?
Anything your users would hit on their first try should block, and Cortad names which findings those are.
- Block: an endpoint that crashes or returns a server error, a route that answers with no sign-in when it should refuse, prompt text or markup leaking into replies, and a reply in the wrong language when a rule says which language to use.
- Wait: a rate whose interval still spans both “rare” and “common”. Run more conversations before deciding.
How does Cortad cover all of this?
Every item above comes from one command in your repository, npx cortad <code>. Your coding agent sends one real request to each endpoint, and you press Run. The quickstart has the details.
Cortad makes your AI agent’s behavior measurably better, with no evals to write or maintain. It learns from your code how your product is meant to be used, sends users through every surface the way your real users would, and asks every reply up to 100 behavioral questions. In production it shows every conversation in real time, so you see how your agent behaves for every kind of user. And it gets better with time: it studies how your real users actually behave, so its users, its metrics and its understanding of your product sharpen with every conversation, and your coding agent uses all of it to make your product better.