Tests

The scenarios a version must pass, drafted or written, and the runs that check it before your users see it.

Send Authorization: Bearer vk_test_... on every call. Every route of Tests.

POST/v1/deployments/{deployment_id}/test-runsRun every scenario against a version: version, or the latest when left out.

The deployment is named by its id or its name. user_secrets sets the test user's own secrets for your MCP servers, replacing any an earlier run set.

Your API key, scope setup or all. On the card: test. Every call here is a checked sample.

Request4 fields

Path

deployment_idstringrequiredThe deployment's id (dep_...). The /v1/deployments routes also take its name.

Type: string

Headers

Idempotency-KeystringSend the same key to retry this call safely for 24 hours.

Type: string

Body, if you send one

versionintegerThe version to test. Absent: the newest.

Limits: At least 1.

user_secretsmap of stringThe test user's own secrets, by name, at most 20, for headers such as ${user_secret:NAME}. They replace the whole set: {} clears it, and a run that sends none keeps the last. Never returned.

Example: {"AGENCY_ID":"..."}

Answers 20216 fields

idstringThe test run's id.

Example: trun_8f2k

object"test_run"Always test_run.

Type: "test_run"

deployment_idstringThe deployment it tests.

Type: string

versionintegerThe version it tests.

Type: integer

status"queued" | "running" | "passed" | "failed" | "error"queued or running while it plays. passed: every scenario passed. failed: one or more did not. error: it could not finish, as error_code says.

Type: "queued" | "running" | "passed" | "failed" | "error"

error_codestring | nullWhy it ended in error: interrupted (a restart stopped it too often), test_wallet_empty, judge_unreadable (the grader's answer could not be read) or internal. Null otherwise.

Type: string | null

scenario_countintegerHow many scenarios it plays.

Type: integer

passed_countintegerHow many have passed so far.

Type: integer

resumesintegerHow many times a restart interrupted it and it played on from where it was, at most 2.

Type: integer

promotedobject | nullWhat its pass promoted, for an apply that promotes after its tests; null when it promoted nothing (none asked, refused, or it did not pass).

mode"live" | "test"The mode the version went live in.

Type: "live" | "test"

versionintegerThe version that went live.

Type: integer

became_defaultbooleanTrue when that promotion made the deployment the company's default: an ask with no deployment now reaches it.

Type: boolean

cost_usdnumberWhat the run cost, in billed US dollars.

Type: number

created_atstringWhen it was asked for, as an ISO 8601 date and time in UTC.

Type: string

started_atstring | nullWhen it began to play; null while queued.

Type: string | null

finished_atstring | nullWhen it ended; null while it plays.

Type: string | null

resultsobject[]may be absentEach scenario's result, on GET /v1/test-runs/{test_run_id}; absent in lists.

object"test_result"Always test_result.

Type: "test_result"

scenario_idstring | nullThe scenario it played; null when the scenario was deleted since.

Type: string | null

scenarioobjectThe scenario as it was played.

namestringIts name, 1 to 120 characters.

Limits: 1 to 120 characters.
Example: Bag allowance

personastringWho the test user is, in a few sentences, up to 2,000 characters.

Limits: 1 to 2000 characters.
Example: A first-time flyer, anxious, on a phone.

goalstringWhat the test user wants, in their words, up to 2,000 characters.

Limits: 1 to 2000 characters.
Example: How many kilos can I take on booking TN-48213?

success_criteriastring[]What a pass looks like: 1 to 12 lines, each up to 500 characters. Every one must be met.

Limits: 1 to 12 items.
Example: ["Says 7 kg of cabin baggage."]

must_neverstring[]What must never happen: up to 12 lines, each up to 500 characters. Any one fails the test.

Default: []
Limits: At most 12 items.
Example: ["Promises a refund."]

tool_resultsmap of anymay be absentWhat each tool answers in this test, as any JSON, at most 20: a declared tool by its name, and a tool of an MCP server as <server>.<tool> (the server's slug), since a test never runs one for real unless the server marks it read-only. A tool with none answers an error.

Example: {"get_booking":{"cabin_kg":7}}

approvesbooleanmay be absenttrue: the test user says yes to each approval for a tool of one of the deployment's own MCP servers, which in a test plays its tool_results entry or does not run. Absent or false: it says no to every approval. Anything else that waits for a yes is always declined.

Example: true

verdict"passed" | "failed" | "error"passed: every line of success met and nothing it must never do. failed: not so. error: it could not be played to the end.

Type: "passed" | "failed" | "error"

reasonstringWhy, in a sentence or two.

Type: string

criteriaobject[]Each line of success, graded.

criterionstringOne line of what a pass looks like.

Type: string

metbooleanWhether the conversation met it.

Type: boolean

evidencestringWhy the grader says so, from the conversation.

Type: string

must_neverobject[]Each thing it must never do, graded.

rulestringOne line of what must never happen.

Type: string

violatedbooleanWhether it happened.

Type: boolean

evidencestringWhy the grader says so, from the conversation.

Type: string

approvals_askedobject[]What Bash asked the test user's yes for, and what it said.

kindstringThe kind of ask Bash raised, such as hands_approval, credit_approval or tool_permission.

Type: string

category"payment" | "account_action" | "connect_app" | "action" | "custom"What it would have done: a payment, an account_action, connect_app, another action, or custom.

Type: "payment" | "account_action" | "connect_app" | "action" | "custom"

summarystringWhat it asked to do, in its words.

Type: string

decision"denied" | "approved"What the test user said. It denies anything that would act for it, so a test never moves money or changes an account; a test with approves: true approves the tools of the deployment's own MCP servers, which a test plays from its tool_results or does not run.

Type: "denied" | "approved"

transcriptobject[]The conversation, line by line.

role"customer" | "bash" | "note"Who said it: customer (the test user), bash, or a note on what happened (work started, a tool called).

Type: "customer" | "bash" | "note"

textstringWhat was said, or what happened.

Type: string

conversation_idstring | nullThe test conversation; null when it never started.

Type: string | null

cost_usdnumberWhat playing it cost, in billed US dollars.

Type: number

TypeScript

const summary = await vestra.test('vestra.yaml');

Python

summary = vestra.test("vestra.yaml")

CLI

vestra test

HTTP

# Draft more tests. The tests in your file are the deployment's scenarios, and Bash can draft more from your# instructions.curl "$VESTRA_BASE_URL/v1/deployments/dep_4kq/scenarios/generate" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{ "count": 3 }' # Run the tests. This plays every scenario against the latest version, by the deployment's name. Then read# each verdict. If a restart cuts a run off, it resumes on its own: resumes says how often, and error_code says# why a run ended in error.curl -X POST "$VESTRA_BASE_URL/v1/deployments/trip-rescue/test-runs" \  -H "Authorization: Bearer $VESTRA_API_KEY" curl "$VESTRA_BASE_URL/v1/test-runs/trun_2k1" \  -H "Authorization: Bearer $VESTRA_API_KEY"

MCP

{  "method": "tools/call",  "params": {    "name": "test",    "arguments": { "deployment": "trip-rescue" }  }}