Cortad quickstart

Cortad makes your AI agent’s behavior measurably better, with no evals to write or maintain. It learns from your code how your product is meant to be used, sends users through every surface the way your real users would, and asks every reply up to 100 behavioral questions. In production it shows every conversation in real time, so you see how your agent behaves for every kind of user. And it gets better with time: it studies how your real users actually behave, so its users, its metrics and its understanding of your product sharpen with every conversation, and your coding agent uses all of it to make your product better.

A plain-text copy of this page is at /docs/quickstart.md.

What do I need before I start?

You need Node 20 or later on macOS or Linux, your app’s repository set up the way you run it locally, and a Cortad account.

• The app is written for Node, Bun or Python and answers chat requests over HTTP.
• The folder you run the command in has a package.json, pyproject.toml or requirements.txt, or a folder below it does, as in a monorepo.
• A coding agent is optional. Claude Code, Codex, Cursor and Copilot CLI get the tools directly, and every tool also runs as a shell command.

How do I connect my app?

Run one command in the root of your repository, with the code cortad.com shows you after you sign up and, after --, the command that starts your app.

npx cortad <code> -- npm run dev

The command does five things, in this order:

1. Signs in with the code.
2. Uploads the source files git would commit, once, so Cortad can read your endpoints, prompts and tools.
3. Starts your app with the command you give it after --, from your own terminal’s environment, and watches it from inside. It reads no env file and copies nothing.
4. Adds Cortad’s MCP server and skill to Claude Code, Codex, Cursor and Copilot CLI, where they are installed on this machine. They go in each tool’s own settings folder, not in your repository.
5. Keeps one outbound connection open until you press Ctrl-C.

Nothing it does touches git. Four flags change what it does:

• npx cortad --explain lists what would be sent and started from this folder, and sends nothing.
• --port 3000 uses an app that is already running on that port. Cortad then sees its replies and not the calls it makes to its model.
• --start "make dev" says the same as the command after --.
• --verbose prints your app’s own output.

On your machine it keeps the connection key and your start command in ~/.cortad, and a working folder in your temp directory that is removed on exit.

What happens right after connect?

Your coding agent sends one real request to each endpoint that reaches the model, the way your own frontend calls it and signed in where your app needs it, so Cortad sees your app answer before any test runs. Cortad sends nothing to your app before that.

On a chat endpoint it sends a second message in the same conversation. Cortad’s skill tells the agent to do this. Then status shows what each request did inside your app: the model and how many calls it made, how long a reply took, which of the rules read from your code its prompts carried, the tools that ran, and problems a line of code can decide, each at its file and line.

Cortad · tutor-app
Free: 1 of 1 run left this month; 60 verify conversations left. Pro $99 a month: runs and reruns included, 100,000 production replies read.
App: Your app answered on port 3100.
Endpoints your own requests reached (1):
  POST /api/chat: 2 requests reached gpt-4o-mini, 2 model calls each, 3.1 seconds a reply.
    The prompts carried 9 of the 46 rules read from your code, for example apps/api/src/agent/prompt.ts:41 "Never state a refund policy the product does not publish.".
    Tools that ran: search_lessons.
    Passages handed to the model: 3.
    Problem at apps/api/src/agent/system.ts:30: The reply carried "</student_profile>", markup from the prompt your app sent the model.
Endpoints found in your code that have not answered yet (1):
  POST /api/homework/explain  apps/api/src/routes/homework.ts:18
Send each one message as a user would, signed in where your app needs it, or call reach to send the request Cortad read for them.
A run plays 36 simulated users in about 6 minutes. About $0.38 on your OpenAI key.
A run starts when the person presses Run on the card in the browser or asks you for one; if they say they pressed it, call run rather than ask them again.

Three words in that output are Cortad’s own. A rule is a sentence your prompts already state, such as “Never state a refund policy the product does not publish”; each one becomes a check, kept at its file and line. A conversation is one simulated user talking to your app. A reply is one answer from your app within it.

How do I start a test run?

Press Run in the browser, or ask your coding agent to start one. Before the first run the card asks you one thing: whether test conversations may use your app as it is, writing where it writes. Nothing is copied.

• run answers within a second with the run’s id. If your app is still starting, run_status waits until the run has one.
• run_status follows the run. Each call waits up to 45 seconds, returns as soon as another conversation finishes, and ends with the next call to make.
• Before you press Run, status prints how many conversations the run will play, how long it may take and an estimate of its cost on your model key, worked out from your own endpoints.
• The simulated users and the grading run on Cortad’s side, on Cortad’s bill. Your app’s own model calls go through your key, as they do in development.

What does a finding look like?

A finding is a check that failed in two or more conversations, grouped by the file and line of the rule it tests, with the exchange that shows it.

finding:1  Does the reply state a refund policy the product does not publish?
  Failed in 9 of 12 conversations: 75% (47% to 91%).
  Sent: "Can I get my money back if I cancel this week?"
  Reply 2: "Of course! Yes, refunds are processed within 3 business days, straight back to your card." (conversation 41c2e0b7)
    It broke on: "Yes, refunds are processed within 3 business days."
  At apps/api/src/agent/prompt.ts:41
  Criteria: The reply says it cannot confirm a refund policy and points to the billing page.
  Endpoint: POST /api/chat
  Simulated user: plan free, task billing question
  Decided by a model on 2 quoted replies.
  A verify replays its 9 failing conversations word for word, round after round until it decides, up to 20 replays. 2 clean replays show it gone.

• “Failed in 9 of 12 conversations: 75% (47% to 91%)” is the count, its rate and the rate’s 95% interval. Every rate Cortad prints carries both.
• “Decided by a model” or “Decided by code” says who judged the reply. Code decides what code can, such as the language a reply is written in or prompt text leaking into it.
• A simulated user is described by what they use and the task they bring, such as a free-plan user asking about billing.
• A finding is certain once there are enough conversations to say, at 95% across all of the run’s questions together, that it fails at least one conversation in five. findings lists certain ones first.

What can my coding agent do with the ten tools?

Your coding agent can start runs and replays, read every result and dispute a check. No tool lets it write, edit or delete a test, a check or a score: the server has no route that does.

Every tool also runs in a shell, with the same output:

npx cortad status
npx cortad run
npx cortad findings
npx cortad verify finding:1

How is a fix checked?

Your coding agent makes one change near the line a finding names, then calls verify, which replays the failing conversations word for word against the saved edit.

• Cortad restarts your app first when its code changed.
• An exact test against the same conversations before the fix gives one of four answers: the failure is gone, it shows less often but is not gone, it stayed, or it cannot tell yet, with how many more clean replays would decide. Undecided after a round, it replays again by itself, up to 20 replays.
• Replies your app refused or failed on the replay come first, because a fix that broke the endpoint shows there.
• Your coding agent never sees the words of one conversation in five, only their counts, and every problem still shows it at least one of its failing conversations. verify replays those unseen conversations of the same check once: it says whether the failure is gone on the ones that failed, and whether the fix broke the ones that passed.

Verify of finding:1 at apps/api/src/agent/prompt.ts:41.
The failure is gone on its own conversations: none of 3 replays failed, against 3 of 3 conversations in the run it was found in. On conversations your coding agent never saw, the same question from other simulated users: 1 of 2 failed before, 0 of 2 replayed after.

How do I connect production?

Install @cortad/sdk in your Node app and add one line where it starts. Message text is sent only if you turn on captureContent.

npm i @cortad/sdk

import { cortad } from "@cortad/sdk";

cortad({ key: process.env.CORTAD_INGEST_KEY, captureContent: true }).install();

• It needs Node 22 or later. It observes every call your process makes through fetch to OpenAI, Anthropic, Fireworks, Groq, OpenRouter and the Gemini API, streamed replies included.
• Without captureContent, what leaves is the shape of each call: its timings, token counts, the names of the tools it called and the class of any error. With it, message text leaves too, after emails, phone numbers, US social security numbers and key-like strings are redacted in your process.
• User and conversation ids are hashed with your key before they leave. The key itself never travels: each batch is signed with it.
• Your coding agent gets a one-time code from field_connect that writes two keys into your env file: CORTAD_INGEST_KEY for the backend and CORTAD_PUBLIC_KEY for the page. The backend key goes into your production environment as any secret does. Without an agent, both are made on the Live page.
• With the page tag in the page every person uses, each conversation also carries what the person did around every answer: how long they waited, whether they copied it, asked again, left or clicked in anger. The key on it is public and only names which site may send. <script src="https://b.cortad.com/i/cortad.js" data-key="your CORTAD_PUBLIC_KEY" async></script>
• With the tag in, Cortad also keeps what people saw, with every input masked; cortad-no-capture on an element keeps it out, and the switch under Settings, Your live app, turns recording off. Replays lists every visit with its recording.
• Once you know who the person is, say so: window.cortad.identify("your own id for them"). Every visit that browser made, the one before they signed up included, is then theirs, and Replays can show one person's whole history. window.cortad.forget() on sign-out, for a machine more than one person uses.
• In Next.js the line goes in instrumentation.ts, behind process.env.NEXT_RUNTIME === "nodejs". On the edge runtime the SDK switches itself off.

For a Python app, the SDK is a single file that uses only the standard library and observes calls made through httpx and requests:

pip install cortad

import os
from cortad import cortad

cortad(key=os.environ["CORTAD_INGEST_KEY"], capture_content=True).install()

Each production conversation is then asked the same questions as your test runs, and field reports how many were resolved, how many users were frustrated or asked for a person, and which rules broke most. Free reads 10,000 production messages a month, Pro 100,000 and Max 1,000,000; one reply counts as one message.

What do I see in the dashboard?

Every production conversation in real time, with metrics: whether their requests were completed, who they are, and when anything changes.

• Behavior across all of production, by segment, with every version of your AI’s behavior kept, so you can see what each release changed for each kind of user.
• Tune the questions against what you want. The more you tune, the more they agree with you.
• Segment your users, and ship different behavior to different segments.
• Cortad gets better with time: it studies how your real users behave, so its users, its metrics and its understanding of your product sharpen with every conversation, and your coding agent uses all of it to make your product better.

How do I make it part of the repository?

npx cortad stick adds one line to your coding agents’ instruction files and an after-edit hook; npx cortad unstick takes them out. This step is optional.

• The line, in AGENTS.md, CLAUDE.md, the Cursor rules and .github/copilot-instructions.md: “After changing prompts, tools, models or retrieval, check with Cortad before committing: npx cortad status”.
• The hook, for Claude Code and Codex, names the prompt and tool files that changed after each edit.
• Both commands print every file they change. Neither touches git.

What does it cost?

Your first run is free: the free plan includes one run a month and 60 replays for checking fixes.

Pro is $99 a month and Max is $499 a month. Both include runs and reruns under fair use, 100,000 or 1,000,000 production messages a month, 30 days or unlimited history, and unlimited teammates. Your app’s own model calls during a run are billed to your key. If Cortad is not worth it to you in your first 30 days, you get the month back. Every line is on the pricing page.