What makes Cortad different?
Cortad makes your AI agent’s behavior measurably better, with no evals to write or maintain. It learns from your code how your product is meant to be used, sends users through every surface the way your real users would, and asks every reply up to 100 behavioral questions. In production it shows every conversation in real time, so you see how your agent behaves for every kind of user. And it gets better with time: it studies how your real users actually behave, so its users, its metrics and its understanding of your product sharpen with every conversation, and your coding agent uses all of it to make your product better.
Most of the 16 other tools here run the datasets, scorers and judges your team writes and keeps current. Cortad learns its tests from your code and rebuilds them when your code changes. None of the 16 runs your app on your machine and talks to its real endpoints without a harness you write, and none grades a fix on tests hidden from the coding agent. In production, Cortad shows how your agent behaves in every conversation, in real time, and it gets better with time: it studies how your real users behave, so its users, metrics and understanding sharpen with every conversation. The side-by-side pages go further: Cortad vs Braintrust, Cortad vs LangSmith and Cortad vs Langfuse.
Where do the tests come from, and how do they reach your app?
Most tools run tests you write, through a function or endpoint you connect, and your team keeps them current. Cortad learns the tests from your code and calls your real endpoints on your machine.
| Tool | Where the tests come from | How a test reaches your app |
|---|---|---|
| Arize | Datasets from traces, uploads, and synthetic examples from its in-app agent, Alyx. (source) | Your own code, or a public HTTP endpoint; internal endpoints are not supported on its cloud. (source) |
| Braintrust | Hand-built datasets, production logs, and datasets its agent Loop generates from logs. (source) | A task function you write. Remote evals run it on your machine or server. (source) |
| Cekura | Generated from your agent’s main prompt and knowledge base, and from production calls. (source) | Phone lines, voice platforms and WebSocket, or an API chat mode your code drives; ngrok for a local app. (source) |
| Confident AI | Goldens you write, a synthesizer that works from documents, and a conversation simulator. (source) | DeepEval runs in your Python and calls a function you write. The platform calls a public HTTPS endpoint; a tunnel to internal ones is Enterprise only. (source) |
| Cortad | Learned from your code and rebuilt when it changes: the rules your prompts state, your tools, data stores and the journeys your product exists for. | Runs your app on your machine with your own start command, and its simulated users talk to your real endpoints, signed in the way you are. No harness to write. |
| Coval | Hand-built, drafted from a description, or made from a production failure. (source) | Your public HTTPS chat endpoint (“Cannot use local/private IP addresses”), phone or WebSocket. You create the QA users. (source) |
| Galileo (Splunk Agent Observability) | Uploads, rows you add, and synthetic data from examples. (source) | Your own function, or a prompt template it calls. (source) |
| Langfuse | Datasets built by hand or from traces. (source) | A task function you write. (source) |
| LangSmith | Datasets from traces or by hand, and a simulated user inside your own test code. (source) | A function you write. (source) |
| LangWatch | Scenario tests in your repository written by your coding agent, and scenarios on the platform. (source) | An SDK function that connects out, with no public URL, or a tunnel. (source) |
| Latitude | Evaluations generated from failures it finds in production. (source) | Its SDK in production, and a replay runner you write. (source) |
| Maxim AI | Datasets, and synthetic scenarios and personas. (source) | A public HTTP endpoint, or a function you write for a local one. (source) |
| Okareo | Generated from seed scenarios or failed runs, or written by your coding agent through MCP. (source) | An endpoint mapping its cloud calls, or an SDK target in your own process. (source) |
| Openlayer | Your datasets, and tests picked from its catalog. (source) | Runs a batch command from your repository on each commit. (source) |
| Promptfoo | YAML test files you write or generate, red-team generators, and a simulated user. (source) | Its CLI in your environment; the HTTP provider can call localhost. (source) |
| Raindrop | Signals found in production. Replay simulations are in early access. (source) | Its SDK in production, and a local tool that replays a trace against your code. (source) |
| W&B Weave | Datasets from code or from recorded calls. (source) | Your own function, in your code. (source) |
What can your coding agent change?
Most of these tools let your coding agent write tests, datasets or scorers, so the agent that writes the fix can also change what grades it. None of the 16 documents a set of tests hidden from the agent. Cortad’s agent tools run tests and read results; none of them changes a test.
| Tool | What the agent can write through MCP or skills |
|---|---|
| Arize | Nothing through the AX MCP server, which has 46 tools and no create, update or delete tools. Phoenix’s MCP server can write. (source) |
| Braintrust | 42 tools, including edit_dataset_rows and create_evaluator. (source) |
| Cekura | 66 or more tools, including scenarios_create and metrics_create. (source) |
| Confident AI | 75 tools, including update_golden and update_metric. (source) |
| Cortad | Nothing. Ten tools start runs and replays, read results and file disputes; none writes a test, a check or a score. |
| Coval | 27 tools, including create_test_case and update_test_case. Metrics are read-only through MCP. (source) |
| Galileo (Splunk Agent Observability) | Datasets and prompt templates, through an MCP server it calls “under construction”. (source) |
| Langfuse | 86 tools, covering datasets, evaluators, prompts and scores; read and write on by default. (source) |
| LangSmith | Nothing through the remote MCP server. Its CLI and skills create and delete datasets and evaluators. (source) |
| LangWatch | 97 tools in the published package, including create, update and delete for scenarios and evaluators. (source) |
| Latitude | Datasets, signals, scores and monitors. (source) |
| Maxim AI | No MCP server for coding agents found in its docs index. (source) |
| Okareo | 60 tools, including save_scenario and create_or_update_check. (source) |
| Openlayer | 56 tools, including create, update and delete for tests. (source) |
| Promptfoo | Tests are files in your repository; the local MCP server includes generate_test_cases. (source) |
| Raindrop | 55 tools for datasets, signals and experiments; none creates a test case. (source) |
| W&B Weave | Reports and logged runs only; no tool edits datasets or scorers. (source) |
Who owns what now?
Half of this market changed owner, name or direction in 2025 or 2026.
| Company | What changed | When |
|---|---|---|
| Langfuse | Acquired by ClickHouse. (source) | 16 Jan 2026 |
| Helicone | Acquired by Mintlify; the product is in maintenance mode. (source) | 3 Mar 2026 |
| Promptfoo | Acquired by OpenAI. (source) | 9 Mar 2026 |
| Traceloop | Joining ServiceNow. (source) | Mar 2026 |
| Galileo | Acquired by Cisco, then renamed Splunk Agent Observability. (source) | 22 May and 7 Aug 2026 |
| Patronus AI | Now trains “Digital World Models” for AI labs. Its eval platform is still priced; its public docs now need a login. (source) | 25 Jun 2026 |
| Arize | Dynatrace agreed to acquire it for $915M. The deal had not closed as of 30 Sep 2026. (source) | 13 Aug 2026 |
| Guardrails AI (Snowglobe) | Acquired by Harvey. (source) | 9 Sep 2026 |
| W&B Weave | CoreWeave, its owner since May 2025, names Agent Lens “the successor to Weights & Biases Weave”. (source) | 2026 |
| Latitude | New features ship only to its V2 observability product; the V1 prompt manager runs for existing customers. (source) | 2026 |
| OpenAI Evals (hosted) | Read-only from 31 Oct 2026 and shut down on 30 Nov 2026. OpenAI’s migration guide points to Promptfoo. (source) | Notice 3 Jun 2026 |