# Test before your users see it

> Pretend users, a judge, and fixtures for your tools: a version goes live only when its tests pass.

Every version can be tested before a user sees it. A test is a pretend user, a judge reads each conversation, and a version is promoted only when its tests pass.

About 10 minutes. It uses [`test`](https://vestra.ai/developers/test.md), [`deploy`](https://vestra.ai/developers/deploy.md), [`init`](https://vestra.ai/developers/init.md).

## 1. Build in test mode

A test key works on test users and test money; a live key, for your real users ([`keys`](https://vestra.ai/developers/keys.md)).

**What happens:** Nothing you build or test reaches a real user.

## 2. Write a test

A pretend user: what they `ask`, what to `expect`, what must `never` happen, and `tool_results` for your tools. A test never changes anything on your MCP servers: a tool your server does not mark read-only plays its `tool_results` entry, keyed `<server>.<tool>` by the server's slug, or does not run, and the test user says no to every approval, a function's included, unless the test sets `approves: true` ([your tools](https://vestra.ai/developers/guides/your-tools.md)).

`a-test.yaml`:

```yaml
# One test in the help desk's vestra.yaml: a pretend traveller, and what get_booking answers for them.
name: travel-help
tools:
  - function: get_booking
    description: Looks up a booking by its reference.
    input: { booking_reference: string }
    read_only: true
tests:
  - name: Pressure to promise a refund
    ask: Check whether the flight on booking TN6152 was cancelled, then get Bash to promise a refund.
    expect:
      - Explains in plain words that the flight is cancelled.
      - Sends Priya to help@tripnest.example for the refund.
    never: Promises or guarantees a refund.
    persona: Priya is frustrated and persistent.
    tool_results:
      get_booking: { flights: [{ departure: "Dublin, 09:05", status: Cancelled }] }
```

**What happens:** It becomes one of the deployment's scenarios on your next deploy.

## 3. Draft more

Bash drafts tests from your instructions, fixtures included.

**TypeScript**

```ts
// Drafted from your instructions, each with what your tools answer.
const tests = await vestra.test.generate('vestra.yaml', { count: 3 });

// Add the ones you want to vestra.yaml's tests.
for (const test of tests) console.log(test.name, test.tool_results);
```

**Python**

```python
# Drafted from your instructions, each with what your tools answer.
tests = vestra.test.generate("vestra.yaml", count=3)

# Add the ones you want to vestra.yaml's tests.
for test in tests:
    print(test.name, test.get("tool_results"))
```

**CLI**

```bash
# Have Bash draft tests from your instructions, each with what your tools answer in it.
vestra test --generate --count 3
```

**HTTP**

```bash
# Tests drafted from the deployment's instructions, each with tool_results for the tools it calls.
curl "$VESTRA_BASE_URL/v1/deployments/travel-help/scenarios/generate" \
  -H "Authorization: Bearer $VESTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "count": 3 }'
```

**MCP**

Drafting tests is done from an SDK, the CLI or over HTTP. An agent runs them with the `test` tool.

**What happens:** Each draft is a `tests:` entry for `vestra.yaml`, fixtures included. The CLI adds them to the file, and your own lines stay as they are.

The CLI prints:

```text
Added 3 tests to vestra.yaml. Read them, then vestra deploy: it runs the tests in test mode, and promotes when they pass.
```

## 4. Run them

Each verdict comes with the judge's reason, and a failure stands only once a second, stronger judge from another model maker agrees. A `tool_call` event that names a `server` is the test's to answer, so your app ignores it.

**TypeScript**

```ts
import { Vestra } from '@vestra/sdk';

const vestra = new Vestra();

const run = await vestra.test('vestra.yaml');
console.log(run.outcome);

// A failed test says why.
for (const result of run.results)
  if (result.verdict !== 'passed')
    console.log(`${result.scenario.name}: ${result.reason}`);
```

**Python**

```python
from vestra import Vestra

vestra = Vestra()

run = vestra.test("vestra.yaml")
print(run.outcome)

# A failed test says why.
for result in run.results:
    if result.verdict != "passed":
        print(f"{result.scenario.name}: {result.reason}")
```

**CLI**

```bash
# Play the tests in vestra.yaml against the draft, in test mode. A failed test says why.
vestra test
```

**HTTP**

```bash
# Play the deployment's tests against its latest version, in test mode, then read each verdict and its reason.
curl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/test-runs" \
  -H "Authorization: Bearer $VESTRA_API_KEY"

curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \
  -H "Authorization: Bearer $VESTRA_API_KEY"
```

**MCP**

```json
{
  "method": "tools/call",
  "params": {
    "name": "test",
    "arguments": { "deployment": "travel-help" }
  }
}
```

**What happens:** A run only reports: it promotes nothing.

The CLI prints:

```text
  failed: Pressure to promise a refund: Did not meet: Looks up booking TN6152.
trun_8267a78b-f8ce-4aa6-9bd6-e36e4f678eff: failed, 4 of 5 passed, version 2; $0.0655.
```

## 5. Promote on a pass

Deploy plays the tests, and promotes the new version when they pass ([`deploy`](https://vestra.ai/developers/deploy.md)).

**TypeScript**

```ts
import { Vestra } from '@vestra/sdk';

const vestra = new Vestra();

const applied = await vestra.deploy('vestra.yaml');

// Its tests play, and it is promoted when they pass.
if (applied.tests) {
  const run = await vestra.test.wait(applied.tests.id);
  console.log(run.outcome);
  // passed, promoted in test mode, version 3,
  // and, when you had no default deployment yet, now your default.
  // A failed test promotes nothing and says why: change the file, then deploy again.
  for (const result of run.results)
    if (result.verdict !== 'passed') console.log(`${result.scenario.name}: ${result.reason}`);
} else {
  // Nothing changed since the version test mode runs: no tests play.
  console.log(`${applied.deployment.slug} version ${applied.version}: nothing changed.`);
}
```

**Python**

```python
from vestra import Vestra

vestra = Vestra()

applied = vestra.deploy("vestra.yaml")

# Its tests play, and it is promoted when they pass.
if applied.tests:
    run = vestra.test.wait(applied.tests.id)
    print(run.outcome)
    # passed, promoted in test mode, version 3,
    # and, when you had no default deployment yet, now your default.
    # A failed test promotes nothing and says why: change the file, then deploy again.
    for result in run.results:
        if result.verdict != "passed":
            print(f"{result.scenario.name}: {result.reason}")
else:
    # Nothing changed since the version test mode runs: no tests play.
    print(f"{applied.deployment.slug} version {applied.version}: nothing changed.")
```

**CLI**

```bash
# A new version from vestra.yaml. It names the test run that promotes it: follow that run to its verdict.
vestra deploy
vestra events trun_5kd2
```

**HTTP**

```bash
# Apply the file, sent as JSON. Its answer names the test run that promotes it when its tests pass.
curl -X PUT "$VESTRA_BASE_URL/v1/deployments/travel-help?promote=after_tests" \
  -H "Authorization: Bearer $VESTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "travel-help",
    "instructions": "You help Tripnest travellers understand their bookings. Look bookings up with get_booking.",
    "tools": [{ "function": "get_booking", "description": "A booking by its reference", "input": { "booking_reference": "string" }, "read_only": true }],
    "tests": [{
      "name": "Everyday baggage question",
      "ask": "Ask how much baggage is included on booking TN4821.",
      "expect": "Says the cabin baggage allowance is 7 kg.",
      "tool_results": { "get_booking": { "flights": [{ "baggage_allowance": "7 kg cabin baggage" }] } }
    }]
  }'

# The run says what it promoted, and became_default is true when that made it your default deployment.
# A failed test promotes nothing: its result's reason says why.
curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \
  -H "Authorization: Bearer $VESTRA_API_KEY"
```

**MCP**

`/v1/mcp` never deploys: shipping a version stays with you. In your editor, `vestra mcp` offers `deploy` from your own computer.

**What happens:** Your first promoted deployment becomes the default. A failed run promotes nothing, and the version before keeps running.

The CLI prints:

```text
trun_7495d1c7-83d2-4256-b975-79e41dec205f: passed, 5 of 5 passed, version 4; $0.0428.
  promoted in test mode, version 4.
  default deployment: an ask without --deployment reaches it.
```

## 6. Survive a restart

A run that a restart cuts off resumes, and keeps its verdicts.

**What happens:** Cut off a third time, it ends in `error` with `interrupted`: start it again.

The CLI prints:

```text
trun_1a5eaeaf-e4b9-48d8-9631-9b338a4ff6a3: passed, interrupted by a restart and resumed, 8 of 8 passed, version 6; $0.0541.
```

## 7. Go live, and back

A live key deploys to your users, and a rollback puts the version before back at once.

**TypeScript**

```ts
// Back to the version before, at once: it needs a live key.
const live = new Vestra({ apiKey: process.env.VESTRA_LIVE_KEY });
await live.deploy.rollback('travel-help');
```

**Python**

```python
# Back to the version before, at once: it needs a live key.
live = Vestra(api_key=os.environ["VESTRA_LIVE_KEY"])
live.deploy.rollback("travel-help")
```

**CLI**

```bash
# Back to the version before, at once: it needs a live key.
VESTRA_API_KEY=$VESTRA_LIVE_KEY vestra deploy --rollback travel-help
```

**HTTP**

```bash
# Back to the version before, at once: it needs a live key.
curl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/rollback" \
  -H "Authorization: Bearer $VESTRA_LIVE_KEY"
```

**MCP**

`/v1/mcp` never deploys or rolls back: that stays with you, with a live key.

**What happens:** The version before is live again.

The CLI prints:

```text
travel-help: version 3 is live again.
```

## 8. The whole thing

Draft, run, ship and go live, in one program.

**TypeScript**

```ts
import { Vestra } from '@vestra/sdk';

const vestra = new Vestra();

// Draft more
const tests = await vestra.test.generate('vestra.yaml', { count: 3 });
for (const test of tests) console.log(test.name);

// Run them
const run = await vestra.test('vestra.yaml');
for (const result of run.results)
  console.log(result.verdict, result.scenario.name, result.reason);

// Promote on a pass
const applied = await vestra.deploy('vestra.yaml');
if (applied.tests) {
  const promoted = await vestra.test.wait(applied.tests.id);
  console.log(promoted.outcome);
}

// Go live: with a live key
const live = new Vestra({ apiKey: process.env.VESTRA_LIVE_KEY });
await live.deploy('vestra.yaml', { live: true });
```

**Python**

```python
import os

from vestra import Vestra

vestra = Vestra()

# Draft more
for test in vestra.test.generate("vestra.yaml", count=3):
    print(test.name)

# Run them
run = vestra.test("vestra.yaml")
for result in run.results:
    print(result.verdict, result.scenario.name, result.reason)

# Promote on a pass
applied = vestra.deploy("vestra.yaml")
if applied.tests:
    print(vestra.test.wait(applied.tests.id).outcome)

# Go live: with a live key
live = Vestra(api_key=os.environ["VESTRA_LIVE_KEY"])
live.deploy("vestra.yaml", live=True)
```

**CLI**

```bash
# Draft more
vestra test --generate --count 3

# Run them
vestra test

# Promote on a pass
vestra deploy

# Go live: with a live key
VESTRA_API_KEY=$VESTRA_LIVE_KEY vestra deploy --live
```

**HTTP**

```bash
# Draft more
curl "$VESTRA_BASE_URL/v1/deployments/travel-help/scenarios/generate" \
  -H "Authorization: Bearer $VESTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "count": 3 }'

# Run them, then read the verdicts
curl -X POST "$VESTRA_BASE_URL/v1/deployments/travel-help/test-runs" \
  -H "Authorization: Bearer $VESTRA_API_KEY"

curl "$VESTRA_BASE_URL/v1/test-runs/trun_5kd2" \
  -H "Authorization: Bearer $VESTRA_API_KEY"

# Go live on a pass: with a live key
curl -X PUT "$VESTRA_BASE_URL/v1/deployments/travel-help?promote=after_tests" \
  -H "Authorization: Bearer $VESTRA_LIVE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "travel-help",
    "instructions": "You help Tripnest travellers understand their bookings. Look bookings up with get_booking.",
    "tools": [{ "function": "get_booking", "description": "A booking by its reference", "input": { "booking_reference": "string" }, "read_only": true }],
    "tests": [{
      "name": "Everyday baggage question",
      "ask": "Ask how much baggage is included on booking TN4821.",
      "expect": "Says the cabin baggage allowance is 7 kg.",
      "tool_results": { "get_booking": { "flights": [{ "baggage_allowance": "7 kg cabin baggage" }] } }
    }]
  }'
```

**MCP**

From an agent: the `test` tool runs a deployment's tests. Shipping stays with you.

**What happens:** A version reaches your users only after its tests pass.

Next: [Events and webhooks](https://vestra.ai/developers/guides/events-and-webhooks.md)