# test

> Run your test scenarios against a version before real users see it.

Run your test scenarios against a version of your agent before real users see it. A scenario is a pretend user with a goal, a description of success, and things that must never happen. A test run plays every scenario, and an AI judge grades each conversation. You see what failed, and why, before a real user does.

## TypeScript

```ts
// Run the tests. Each test in vestra.yaml is a scenario. This runs them all and waits for the judge.
const { passed, failed, outcome } = await vestra.test('vestra.yaml');
// outcome is passed or failed, or says more about how it went, such as: passed, interrupted by a restart and resumed.
console.log(`${passed} passed, ${failed} failed: ${outcome}`);

// Draft more tests. Bash writes new ones for you to add to vestra.yaml, each with what your tools return in it
// (tool_results).
for (const drafted of await vestra.test.generate('vestra.yaml', { count: 3 })) console.log(drafted.name, drafted.tool_results);
```

## Python

```python
# Run the tests. Each test in vestra.yaml is a scenario. This runs them all and waits for the judge.
result = vestra.test("vestra.yaml")
# outcome is passed or failed, or says more about how it went, such as: passed, interrupted by a restart and resumed.
print(f"{result.passed} passed, {result.failed} failed: {result.outcome}")

# Draft more tests. Bash writes new ones for you to add to vestra.yaml, each with what your tools return in it
# (tool_results).
for drafted in vestra.test.generate("vestra.yaml", count=3):
    print(drafted.name, drafted.get("tool_results"))
```

## CLI

```bash
# Run the tests. The command waits for the judge, and a failed test makes it fail, so your CI fails too.
# It prints how the run went: passed, failed, or more, such as passed, interrupted by a restart and resumed.
vestra test

# Print the run as JSON. Use this in a script.
vestra test --json

# Draft more tests. Bash writes them into vestra.yaml, each with what your tools return in it.
vestra test --generate
```

## HTTP

```bash
# Draft more tests. The tests in your file are the deployment's scenarios, and Bash can draft more from your
# instructions.
curl "$VESTRA_BASE_URL/v1/deployments/dep_4kq/scenarios/generate" \
  -H "Authorization: Bearer $VESTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "count": 3 }'

# Run the tests. This plays every scenario against the latest version, by the deployment's name. Then read
# each verdict. If a restart cuts a run off, it resumes on its own: resumes says how often, and error_code says
# why a run ended in error.
curl -X POST "$VESTRA_BASE_URL/v1/deployments/trip-rescue/test-runs" \
  -H "Authorization: Bearer $VESTRA_API_KEY"

curl "$VESTRA_BASE_URL/v1/test-runs/trun_2k1" \
  -H "Authorization: Bearer $VESTRA_API_KEY"
```

## MCP

```json
{
  "method": "tools/call",
  "params": {
    "name": "test",
    "arguments": { "deployment": "trip-rescue" }
  }
}
```

## Good to know

### Where tests come from

Each time your file is applied (saved to Vestra by `deploy`, or by `test` after an edit), the `tests` in `vestra.yaml` become the deployment's scenarios. A new version is promoted only when they pass. `vestra test --generate` drafts more tests from your instructions. It drops a draft that reuses a test's name or copies its goal word for word, and it is told to avoid repeating the rest. The `tool_results` it writes copy the field names and formats of each tool's example, or of your existing tests. It reads the tests as your file was last applied, so after an edit, run `vestra test` first.

### Testing the file you just edited

`vestra test` tests the `vestra.yaml` you just edited, so a test after a fix tests the fix. With a test key, if the file changed since the deployment's latest version, Vestra first saves it as a new version without promoting it (as `vestra deploy --no-promote` does), and tests that version. It tells you so: `vestra.yaml changed since version 4: testing it as version 5, not promoted.` If the deployment does not exist yet, this creates it. On a pass, it adds `version 5 is not promoted: vestra deploy promotes it without testing it again.`

Skills and knowledge are not versioned yet: every version, the live one included, reads them as they are now. So if your edit changes a skill or a knowledge document, `vestra test` writes nothing, names what changed and exits with code 2, so that you apply it with `vestra deploy` (with a live key while a deployment runs live). A live key saves nothing and tests the latest version. An unchanged file tests the latest version, and `vestra test <name>` or `--version <n>` tests a version already saved in Vestra. The `test` tool in `vestra mcp` works the same way.

### Your tools in a test

A test never runs your code. Instead, `tool_results` gives what each of your tools returns in that test, such as `tool_results: { get_booking: { cabin_kg: 7 } }`. It can only name tools your file declares. Any other name is refused with `tool_not_declared`. A tool with no entry returns an error for every call. The transcript records each call's arguments, and notes that `tool_results` answered it.

### Your MCP server's tools in a test

A test never runs an MCP server tool that the server does not mark read-only. Its `tool_results` entry answers instead, and without an entry the tool does not run. The entry's key is `<slug>.<tool>`, using the server's slug, never its `name`.

A server your file registers gets its slug from its URL's host: lowercase letters, digits and dashes, starting with a letter, at most 40 characters. For example, `https://mcp.tripnest.example/mcp` becomes `mcp-tripnest-example`. If another server already has that slug, Vestra adds `-2`, `-3` and so on. A server you register with `POST /v1/mcp-servers` keeps the slug you gave it. `GET /v1/mcp-servers` lists each server's `slug`. When no entry answered a call, the transcript says which key would have.

### Approvals in a test

By default, the pretend user says no to every approval. So a test of an action checks that Bash asks first, and if the test fails after a no, its result says so.

A test with `approves: true` says yes to each approval for your MCP server's tools and your functions, as your user would in your app. That lets it check what Bash does after the yes. The tool still returns its `tool_results` entry, or does not run. Every other approval, such as a browser action on an account or connecting an app, is declined in every test, and a failure after such a no says so.

`vestra test --generate` writes `approves: true` on a test whose user agrees to such a tool's action. It checks what the approval shows, such as `The approval shows the charge of 12 and the new return time 2026-10-07 17:30.`, never what Bash says before it. When one of your functions needs approval, the starter tests include one where the user says yes and one where the user says no. The no test checks that Bash says the action did not happen.

### If a run is interrupted

A test run survives a restart of Vestra's servers. If a restart cuts a run off, it resumes where it stopped. It keeps the verdicts it already has and plays its unfinished tests again, and each verdict is billed once. The result says so, as in `passed, interrupted by a restart and resumed`. A resumed run that passes promotes the version as usual. A run cut off a third time (`interrupted 3 times by restarts`), or lost for longer than 15 minutes, ends in `error` with `interrupted` and promotes nothing. Start it again.

### How tests are judged

An AI judge grades each conversation against what to `expect` and what must `never` happen. A failure goes to a second, stronger judge. It comes from a different model maker than the models that played the test, so no model grades its own maker's work. If the second judge passes the test, the test passes, and the reason quotes both judges and names the second. If it fails the test too, the failure stands. A pass is judged only once. If no second judge could check a failure, the failure stands, and its reason says why.

### What a run costs

Each scenario bills its judge's call along with the work it does. A failing scenario also bills the second judge's call. The run's `cost_usd` is the total.

### When a run finishes

Your webhooks receive `test_run.completed` with how many tests passed, and an `error_code` that is null unless the run ended in `error`. The CLI and the SDKs say in plain words how the run ended, never with a bare error. Each result includes its conversation and the reason for its verdict.

## The routes behind test

- `GET` [/v1/deployments/{deployment_id}/scenarios](https://vestra.ai/developers/reference/api/tests/get-v1-deployments-deployment-id-scenarios.md): The deployment's test scenarios. Key scope: `setup` (or `all`).
- `POST` [/v1/deployments/{deployment_id}/scenarios](https://vestra.ai/developers/reference/api/tests/post-v1-deployments-deployment-id-scenarios.md): Add one: a pretend user, their goal, what success looks like, what must never happen. Key scope: `setup` (or `all`).
- `POST` [/v1/deployments/{deployment_id}/scenarios/generate](https://vestra.ai/developers/reference/api/tests/post-v1-deployments-deployment-id-scenarios-generate.md): Have Bash draft scenarios from the deployment's instructions. Key scope: `setup` (or `all`).
- `PATCH` [/v1/deployments/{deployment_id}/scenarios/{scenario_id}](https://vestra.ai/developers/reference/api/tests/patch-v1-deployments-deployment-id-scenarios-scenario-id.md): Change one. Key scope: `setup` (or `all`).
- `DELETE` [/v1/deployments/{deployment_id}/scenarios/{scenario_id}](https://vestra.ai/developers/reference/api/tests/delete-v1-deployments-deployment-id-scenarios-scenario-id.md): Remove one. Key scope: `setup` (or `all`).
- `POST` [/v1/deployments/{deployment_id}/test-runs](https://vestra.ai/developers/reference/api/tests/post-v1-deployments-deployment-id-test-runs.md): Run every scenario against a version: version, or the latest when left out. The deployment is named by its id or its name. user_secrets sets the test user's own secrets for your MCP servers, replacing any an earlier run set. Key scope: `setup` (or `all`).
- `GET` [/v1/deployments/{deployment_id}/test-runs](https://vestra.ai/developers/reference/api/tests/get-v1-deployments-deployment-id-test-runs.md): Past test runs. Key scope: `setup` (or `all`).
- `GET` [/v1/test-runs/{test_run_id}](https://vestra.ai/developers/reference/api/tests/get-v1-test-runs-test-run-id.md): Pass or fail per scenario, the conversation and the judge's reason. Key scope: `setup` (or `all`).

Next: [`ask`](https://vestra.ai/developers/ask.md)