test

Each time your file is applied (saved to Vestra by deploy, or by test after an edit), the tests in vestra.yaml become the deployment's scenarios. A new version is promoted only when they pass. vestra test --generate drafts more tests from your instructions. It drops a draft that reuses a test's name or copies its goal word for word, and it is told to avoid repeating the rest. The tool_results it writes copy the field names and formats of each tool's example, or of your existing tests. It reads the tests as your file was last applied, so after an edit, run vestra test first.

vestra test tests the vestra.yaml you just edited, so a test after a fix tests the fix. With a test key, if the file changed since the deployment's latest version, Vestra first saves it as a new version without promoting it (as vestra deploy --no-promote does), and tests that version. It tells you so: vestra.yaml changed since version 4: testing it as version 5, not promoted. If the deployment does not exist yet, this creates it. On a pass, it adds version 5 is not promoted: vestra deploy promotes it without testing it again.

Skills and knowledge are not versioned yet: every version, the live one included, reads them as they are now. So if your edit changes a skill or a knowledge document, vestra test writes nothing, names what changed and exits with code 2, so that you apply it with vestra deploy (with a live key while a deployment runs live). A live key saves nothing and tests the latest version. An unchanged file tests the latest version, and vestra test <name> or --version <n> tests a version already saved in Vestra. The test tool in vestra mcp works the same way.

A test never runs your code. Instead, tool_results gives what each of your tools returns in that test, such as tool_results: { get_booking: { cabin_kg: 7 } }. It can only name tools your file declares. Any other name is refused with tool_not_declared. A tool with no entry returns an error for every call. The transcript records each call's arguments, and notes that tool_results answered it.

A test never runs an MCP server tool that the server does not mark read-only. Its tool_results entry answers instead, and without an entry the tool does not run. The entry's key is <slug>.<tool>, using the server's slug, never its name.

A server your file registers gets its slug from its URL's host: lowercase letters, digits and dashes, starting with a letter, at most 40 characters. For example, https://mcp.tripnest.example/mcp becomes mcp-tripnest-example. If another server already has that slug, Vestra adds -2, -3 and so on. A server you register with POST /v1/mcp-servers keeps the slug you gave it. GET /v1/mcp-servers lists each server's slug. When no entry answered a call, the transcript says which key would have.

By default, the pretend user says no to every approval. So a test of an action checks that Bash asks first, and if the test fails after a no, its result says so.

A test with approves: true says yes to each approval for your MCP server's tools and your functions, as your user would in your app. That lets it check what Bash does after the yes. The tool still returns its tool_results entry, or does not run. Every other approval, such as a browser action on an account or connecting an app, is declined in every test, and a failure after such a no says so.

vestra test --generate writes approves: true on a test whose user agrees to such a tool's action. It checks what the approval shows, such as The approval shows the charge of 12 and the new return time 2026-10-07 17:30., never what Bash says before it. When one of your functions needs approval, the starter tests include one where the user says yes and one where the user says no. The no test checks that Bash says the action did not happen.

A test run survives a restart of Vestra's servers. If a restart cuts a run off, it resumes where it stopped. It keeps the verdicts it already has and plays its unfinished tests again, and each verdict is billed once. The result says so, as in passed, interrupted by a restart and resumed. A resumed run that passes promotes the version as usual. A run cut off a third time (interrupted 3 times by restarts), or lost for longer than 15 minutes, ends in error with interrupted and promotes nothing. Start it again.

An AI judge grades each conversation against what to expect and what must never happen. A failure goes to a second, stronger judge. It comes from a different model maker than the models that played the test, so no model grades its own maker's work. If the second judge passes the test, the test passes, and the reason quotes both judges and names the second. If it fails the test too, the failure stands. A pass is judged only once. If no second judge could check a failure, the failure stands, and its reason says why.

Each scenario bills its judge's call along with the work it does. A failing scenario also bills the second judge's call. The run's cost_usd is the total.

Your webhooks receive test_run.completed with how many tests passed, and an error_code that is null unless the run ended in error. The CLI and the SDKs say in plain words how the run ended, never with a bare error. Each result includes its conversation and the reason for its verdict.