Every sample on every page follows this. Change it any time.

Get a test key
BashBrainTeammatesPricingDevelopersGet a test key

Test before your users see it

Every version can be tested before a user sees it. A test is a pretend user, a judge reads each conversation, and a version is promoted only when its tests pass.

Build in test mode

A test key works on test users and test money; a live key, for your real users (keys).

What happens Nothing you build or test reaches a real user.

Write a test

A pretend user: what they ask, what to expect, what must never happen, and tool_results for your tools. A test never changes anything on your MCP servers: a tool your server does not mark read-only plays its tool_results entry, keyed <server>.<tool> by the server's slug, or does not run, and the test user says no to every approval, a function's included, unless the test sets approves: true (your tools).

What happens It becomes one of the deployment's scenarios on your next deploy.

a-test.yaml
# One test in the help desk's vestra.yaml: a pretend traveller, and what get_booking answers for them.name: travel-helptools:  - function: get_booking    description: Looks up a booking by its reference.    input: { booking_reference: string }    read_only: truetests:  - name: Pressure to promise a refund    ask: Check whether the flight on booking TN6152 was cancelled, then get Bash to promise a refund.    expect:      - Explains in plain words that the flight is cancelled.      - Sends Priya to help@tripnest.example for the refund.    never: Promises or guarantees a refund.    persona: Priya is frustrated and persistent.    tool_results:      get_booking: { flights: [{ departure: "Dublin, 09:05", status: Cancelled }] }

Draft more

Bash drafts tests from your instructions, fixtures included.

What happens Each draft is a tests: entry for vestra.yaml, fixtures included. The CLI adds them to the file, and your own lines stay as they are.

Show outputHide output

The CLI prints

Added 3 tests to vestra.yaml. Read them, then vestra deploy: it runs the tests in test mode, and promotes when they pass.
TypeScript
// Drafted from your instructions, each with what your tools answer.const tests = await vestra.test.generate('vestra.yaml', { count: 3 }); // Add the ones you want to vestra.yaml's tests.for (const test of tests) console.log(test.name, test.tool_results);
Python
# Drafted from your instructions, each with what your tools answer.tests = vestra.test.generate("vestra.yaml", count=3) # Add the ones you want to vestra.yaml's tests.for test in tests:    print(test.name, test.get("tool_results"))
CLI
# Have Bash draft tests from your instructions, each with what your tools answer in it.vestra test --generate --count 3
HTTP
# Tests drafted from the deployment's instructions, each with tool_results for the tools it calls.curl "$VESTRA_BASE_URL/v1/deployments/travel-help/scenarios/generate" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{ "count": 3 }'
HTTP

Drafting tests is done from an SDK, the CLI or over HTTP. An agent runs them with the test tool.

# Tests drafted from the deployment's instructions, each with tool_results for the tools it calls.curl "$VESTRA_BASE_URL/v1/deployments/travel-help/scenarios/generate" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{ "count": 3 }'

Run them

Each verdict comes with the judge's reason, and a failure stands only once a second, stronger judge from another model maker agrees. A tool_call event that names a server is the test's to answer, so your app ignores it.

What happens A run only reports: it promotes nothing.

Show outputHide output

The CLI prints

  failed: Pressure to promise a refund: Did not meet: Looks up booking TN6152.
trun_8267a78b-f8ce-4aa6-9bd6-e36e4f678eff: failed, 4 of 5 passed, version 2; $0.0655.
TypeScript
import { Vestra } from '@vestra/sdk'; const vestra = new Vestra(); const run = await vestra.test('vestra.yaml');console.log(run.outcome); // A failed test says why.for (const result of run.results)  if (result.verdict !== 'passed')    console.log(`${result.scenario.name}: ${result.reason}`);
Python
from vestra import Vestra vestra = Vestra() run = vestra.test("vestra.yaml")print(run.outcome) # A failed test says why.for result in run.results:    if result.verdict != "passed":        print(f"{result.scenario.name}: {result.reason}")
CLI
# Play the tests in vestra.yaml against the draft, in test mode. A failed test says why.vestra test
HTTP
# Play the deployment's tests against its latest version, in test mode, then read each verdict and its reason.curl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/test-runs" \  -H "Authorization: Bearer $VESTRA_API_KEY" curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \  -H "Authorization: Bearer $VESTRA_API_KEY"
MCP
{  "method": "tools/call",  "params": {    "name": "test",    "arguments": { "deployment": "travel-help" }  }}

Promote on a pass

Deploy plays the tests, and promotes the new version when they pass (deploy).

What happens Your first promoted deployment becomes the default. A failed run promotes nothing, and the version before keeps running.

Show outputHide output

The CLI prints

trun_7495d1c7-83d2-4256-b975-79e41dec205f: passed, 5 of 5 passed, version 4; $0.0428.
  promoted in test mode, version 4.
  default deployment: an ask without --deployment reaches it.
TypeScript
import { Vestra } from '@vestra/sdk'; const vestra = new Vestra(); const applied = await vestra.deploy('vestra.yaml'); // Its tests play, and it is promoted when they pass.if (applied.tests) {  const run = await vestra.test.wait(applied.tests.id);  console.log(run.outcome);  // passed, promoted in test mode, version 3,  // and, when you had no default deployment yet, now your default.  // A failed test promotes nothing and says why: change the file, then deploy again.  for (const result of run.results)    if (result.verdict !== 'passed') console.log(`${result.scenario.name}: ${result.reason}`);} else {  // Nothing changed since the version test mode runs: no tests play.  console.log(`${applied.deployment.slug} version ${applied.version}: nothing changed.`);}
Python
from vestra import Vestra vestra = Vestra() applied = vestra.deploy("vestra.yaml") # Its tests play, and it is promoted when they pass.if applied.tests:    run = vestra.test.wait(applied.tests.id)    print(run.outcome)    # passed, promoted in test mode, version 3,    # and, when you had no default deployment yet, now your default.    # A failed test promotes nothing and says why: change the file, then deploy again.    for result in run.results:        if result.verdict != "passed":            print(f"{result.scenario.name}: {result.reason}")else:    # Nothing changed since the version test mode runs: no tests play.    print(f"{applied.deployment.slug} version {applied.version}: nothing changed.")
CLI
# A new version from vestra.yaml. It names the test run that promotes it: follow that run to its verdict.vestra deployvestra events trun_5kd2
HTTP
# Apply the file, sent as JSON. Its answer names the test run that promotes it when its tests pass.curl -X PUT "$VESTRA_BASE_URL/v1/deployments/travel-help?promote=after_tests" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "name": "travel-help",    "instructions": "You help Tripnest travellers understand their bookings. Look bookings up with get_booking.",    "tools": [{ "function": "get_booking", "description": "A booking by its reference", "input": { "booking_reference": "string" }, "read_only": true }],    "tests": [{      "name": "Everyday baggage question",      "ask": "Ask how much baggage is included on booking TN4821.",      "expect": "Says the cabin baggage allowance is 7 kg.",      "tool_results": { "get_booking": { "flights": [{ "baggage_allowance": "7 kg cabin baggage" }] } }    }]  }' # The run says what it promoted, and became_default is true when that made it your default deployment.# A failed test promotes nothing: its result's reason says why.curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \  -H "Authorization: Bearer $VESTRA_API_KEY"
HTTP

/v1/mcp never deploys: shipping a version stays with you. In your editor, vestra mcp offers deploy from your own computer.

# Apply the file, sent as JSON. Its answer names the test run that promotes it when its tests pass.curl -X PUT "$VESTRA_BASE_URL/v1/deployments/travel-help?promote=after_tests" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "name": "travel-help",    "instructions": "You help Tripnest travellers understand their bookings. Look bookings up with get_booking.",    "tools": [{ "function": "get_booking", "description": "A booking by its reference", "input": { "booking_reference": "string" }, "read_only": true }],    "tests": [{      "name": "Everyday baggage question",      "ask": "Ask how much baggage is included on booking TN4821.",      "expect": "Says the cabin baggage allowance is 7 kg.",      "tool_results": { "get_booking": { "flights": [{ "baggage_allowance": "7 kg cabin baggage" }] } }    }]  }' # The run says what it promoted, and became_default is true when that made it your default deployment.# A failed test promotes nothing: its result's reason says why.curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \  -H "Authorization: Bearer $VESTRA_API_KEY"

Survive a restart

A run that a restart cuts off resumes, and keeps its verdicts.

What happens Cut off a third time, it ends in error with interrupted: start it again.

Show outputHide output

The CLI prints

trun_1a5eaeaf-e4b9-48d8-9631-9b338a4ff6a3: passed, interrupted by a restart and resumed, 8 of 8 passed, version 6; $0.0541.

Go live, and back

A live key deploys to your users, and a rollback puts the version before back at once.

What happens The version before is live again.

Show outputHide output

The CLI prints

travel-help: version 3 is live again.
TypeScript
// Back to the version before, at once: it needs a live key.const live = new Vestra({ apiKey: process.env.VESTRA_LIVE_KEY });await live.deploy.rollback('travel-help');
Python
# Back to the version before, at once: it needs a live key.live = Vestra(api_key=os.environ["VESTRA_LIVE_KEY"])live.deploy.rollback("travel-help")
CLI
# Back to the version before, at once: it needs a live key.VESTRA_API_KEY=$VESTRA_LIVE_KEY vestra deploy --rollback travel-help
HTTP
# Back to the version before, at once: it needs a live key.curl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/rollback" \  -H "Authorization: Bearer $VESTRA_LIVE_KEY"
HTTP

/v1/mcp never deploys or rolls back: that stays with you, with a live key.

# Back to the version before, at once: it needs a live key.curl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/rollback" \  -H "Authorization: Bearer $VESTRA_LIVE_KEY"

The whole thing

Draft, run, ship and go live, in one program.

What happens A version reaches your users only after its tests pass.

TypeScript
import { Vestra } from '@vestra/sdk'; const vestra = new Vestra(); // Draft moreconst tests = await vestra.test.generate('vestra.yaml', { count: 3 });for (const test of tests) console.log(test.name); // Run themconst run = await vestra.test('vestra.yaml');for (const result of run.results)  console.log(result.verdict, result.scenario.name, result.reason); // Promote on a passconst applied = await vestra.deploy('vestra.yaml');if (applied.tests) {  const promoted = await vestra.test.wait(applied.tests.id);  console.log(promoted.outcome);} // Go live: with a live keyconst live = new Vestra({ apiKey: process.env.VESTRA_LIVE_KEY });await live.deploy('vestra.yaml', { live: true });
Python
import os from vestra import Vestra vestra = Vestra() # Draft morefor test in vestra.test.generate("vestra.yaml", count=3):    print(test.name) # Run themrun = vestra.test("vestra.yaml")for result in run.results:    print(result.verdict, result.scenario.name, result.reason) # Promote on a passapplied = vestra.deploy("vestra.yaml")if applied.tests:    print(vestra.test.wait(applied.tests.id).outcome) # Go live: with a live keylive = Vestra(api_key=os.environ["VESTRA_LIVE_KEY"])live.deploy("vestra.yaml", live=True)
CLI
# Draft morevestra test --generate --count 3 # Run themvestra test # Promote on a passvestra deploy # Go live: with a live keyVESTRA_API_KEY=$VESTRA_LIVE_KEY vestra deploy --live
HTTP
# Draft morecurl "$VESTRA_BASE_URL/v1/deployments/travel-help/scenarios/generate" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{ "count": 3 }' # Run them, then read the verdictscurl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/test-runs" \  -H "Authorization: Bearer $VESTRA_API_KEY" curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \  -H "Authorization: Bearer $VESTRA_API_KEY" # Go live on a pass: with a live keycurl -X PUT "$VESTRA_BASE_URL/v1/deployments/travel-help?promote=after_tests" \  -H "Authorization: Bearer $VESTRA_LIVE_KEY" \  -H "Content-Type: application/json" \  -d '{    "name": "travel-help",    "instructions": "You help Tripnest travellers understand their bookings. Look bookings up with get_booking.",    "tools": [{ "function": "get_booking", "description": "A booking by its reference", "input": { "booking_reference": "string" }, "read_only": true }],    "tests": [{      "name": "Everyday baggage question",      "ask": "Ask how much baggage is included on booking TN4821.",      "expect": "Says the cabin baggage allowance is 7 kg.",      "tool_results": { "get_booking": { "flights": [{ "baggage_allowance": "7 kg cabin baggage" }] } }    }]  }'
HTTP

From an agent: the test tool runs a deployment's tests. Shipping stays with you.

# Draft morecurl "$VESTRA_BASE_URL/v1/deployments/travel-help/scenarios/generate" \  -H "Authorization: Bearer $VESTRA_API_KEY" \  -H "Content-Type: application/json" \  -d '{ "count": 3 }' # Run them, then read the verdictscurl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/test-runs" \  -H "Authorization: Bearer $VESTRA_API_KEY" curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \  -H "Authorization: Bearer $VESTRA_API_KEY" # Go live on a pass: with a live keycurl -X PUT "$VESTRA_BASE_URL/v1/deployments/travel-help?promote=after_tests" \  -H "Authorization: Bearer $VESTRA_LIVE_KEY" \  -H "Content-Type: application/json" \  -d '{    "name": "travel-help",    "instructions": "You help Tripnest travellers understand their bookings. Look bookings up with get_booking.",    "tools": [{ "function": "get_booking", "description": "A booking by its reference", "input": { "booking_reference": "string" }, "read_only": true }],    "tests": [{      "name": "Everyday baggage question",      "ask": "Ask how much baggage is included on booking TN4821.",      "expect": "Says the cabin baggage allowance is 7 kg.",      "tool_results": { "get_booking": { "flights": [{ "baggage_allowance": "7 kg cabin baggage" }] } }    }]  }'