Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@
* [Creating a preview](verify/how-to-guides/creating-a-preview.md)
* [Managing previews](verify/how-to-guides/managing-previews.md)
* [Seed data for previews](verify/how-to-guides/seed-data-for-previews.md)
* [Registering an external preview](verify/how-to-guides/registering-an-external-preview.md)
* [Running with remote agents](verify/how-to-guides/running-with-remote-agents.md)
* [Configuring branch protection](verify/how-to-guides/configuring-branch-protection.md)
* [Fixing verification failures](verify/how-to-guides/fixing-verification-failures.md)
Expand Down
30 changes: 30 additions & 0 deletions verify/concepts/previews.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ A **preview** is an ephemeral environment Verify builds on demand and runs scena

A preview is short-lived: built for the branch's current commit, used by the scenario runner (and optionally a human reviewer), and torn down when the branch moves to a new commit. A re-run on the same commit reuses the running preview — including its state. See [Lifecycle](#lifecycle) for the exact contract.

Aviator can build the environment itself — the **sandbox** method, described through most of this page — or use one your own CI deploys, the **endpoint** method. See [Externally hosted previews](#externally-hosted-previews).

### Optional, not required

Previews are optional. Verify works on day one with code-scan alone — without a preview, every criterion is routed to code-scan (static analysis of the diff) and you get verdicts on structural criteria from the first PR.
Expand Down Expand Up @@ -43,6 +45,33 @@ A preview is composed of inputs from three places: a preview image, your secret

Aviator stitches these together into a single ephemeral container. Your Verify settings are the contract — see [Preview YAML reference](../reference/preview-yaml.md) for every field.

### Externally hosted previews

Many teams already deploy an ephemeral environment for every pull request from their own CI. Rather than have Aviator build a second one, a preview can be declared as **externally hosted** — `method: endpoint` in the config. The environment stays yours; Aviator only borrows it.

The split of responsibility:

| Your CI does | Aviator does |
| ---------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Builds and deploys the environment for the pull request | Correlates the registration to the verification session for that PR or branch |
| Registers its URL (and the commit it's running) with Aviator | Waits for a registration when a run needs one |
| Destroys the environment on its own schedule | Drives the app at the registered URL and records the evidence |
| Optionally deregisters early when it tears the environment down | Stops using the registration at its expiry |

Aviator never builds or tears down an externally hosted preview, and never holds infrastructure credentials for it — the environment is already running by the time Aviator sees it.

**Waiting, then degrading.** Verification and the deploy pipeline run on their own clocks, so a run that needs the preview waits for one to be registered, up to a configured timeout. If the registration arrives in time, scenarios run against it. If it doesn't, verification still completes on code analysis alone — a degraded result rather than a failure — and a registration that lands afterwards starts a follow-up verification automatically. A registration also has to match the commit being verified: one built from a stale commit never drives the run.

**Choosing between the two methods:**

| Choose sandbox when… | Choose endpoint when… |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| You don't already build per-PR environments and don't want to | You already run per-PR ephemeral environments in CI |
| The service boots from an image plus a setup script | The environment needs infrastructure Aviator can't host — managed data stores, private networks, a multi-service topology |
| You want previews with no CI work at all | You want verification to run against the same environment your team already reviews |

See [Registering an external preview](../how-to-guides/registering-an-external-preview.md) for the registration call and its semantics.

### Multiple previews per repo

A repo can have more than one preview configured. Common patterns:
Expand Down Expand Up @@ -86,3 +115,4 @@ Previews look like CI environments but they're not the same thing:
* [Creating a preview](../how-to-guides/creating-a-preview.md) — walkthrough
* [Managing previews](../how-to-guides/managing-previews.md) — bake vs. setup, refresh, cleanup
* [Seed data for previews](../how-to-guides/seed-data-for-previews.md) — fixtures and deterministic state
* [Registering an external preview](../how-to-guides/registering-an-external-preview.md) — using an environment your CI deploys
2 changes: 2 additions & 0 deletions verify/how-to-guides/creating-a-preview.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ This guide walks through adding a preview to a repo end-to-end: registering an i

By the end, you'll have a working `default` preview that scenarios can run against.

> This guide covers previews Aviator builds and runs. If your CI already deploys an ephemeral environment for every pull request, you can point Verify at that instead — see [Registering an external preview](registering-an-external-preview.md).

> Previews are optional. Verify works without one — every criterion routes to code-scan in that case. Add a preview when behavioral verdicts (endpoint contracts, UI flows, error shapes) start mattering for your team. See [Concepts: Previews](../concepts/previews.md).

**Time:** ~15 minutes
Expand Down
160 changes: 160 additions & 0 deletions verify/how-to-guides/registering-an-external-preview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Registering an external preview

Some teams already deploy an ephemeral environment for every pull request from their own CI. An **endpoint preview** lets Verify use that environment instead of building one: your pipeline deploys the app as it always has, then registers the resulting URL with Aviator, and verification runs against it.

Aviator never builds or tears down an endpoint preview. It correlates the registration to the verification session, waits for it when a run needs it, drives the app at that URL, and stops using it when the registration expires.

For the alternative — Aviator booting a container from a preview image — see [Creating a preview](creating-a-preview.md). For the concept, see [Concepts: Previews](../concepts/previews.md).

**Time:** ~20 minutes

**Prerequisites:**

* The repo connected to Aviator — see [Connect a repository](connect-a-repository.md)
* A CI pipeline that already deploys a reachable per-PR environment
* An Aviator API token — see [API authentication](../../api/reference/authentication.md)

### Step 1: Declare an endpoint preview

Endpoint previews are configured in your **Aviator Verify settings** (**Verify → Settings → Verify**, with the repo selected). Add a `preview` entry with `method: endpoint`:

```yaml
verify:
preview:
- name: ci
method: endpoint
wait_timeout_sec: 1800 # how long a run waits for a registration
expires_default_sec: 7200 # registration lifetime when a registration omits expires_at
```

Endpoint entries take none of the sandbox fields (`image`, `port`, `setup`, `teardown`, `secrets`) — the environment isn't Aviator's to build. See [Preview YAML reference](../reference/preview-yaml.md) for the full schema.

Registrations are rejected until this entry exists: a repo with no `method: endpoint` preview configured returns `400`.

### Step 2: Register the preview from CI

Make the registration the **last step of the deploy job**, after the environment is deployed and reachable. Registering earlier means Verify may start driving a URL that isn't serving yet.

`POST https://api.aviator.co/api/v1/verify/preview-environment`

Authenticate with `Authorization: Bearer <api token>`, the same as the rest of the [REST API](../../api/reference/json-api.md).

#### Request body

| Field | Type | Required | Description |
| --------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `repository` | object | yes | `{"org": "...", "name": "..."}` — the repository the preview was built from. |
| `branch` | string | yes | The branch the preview was built from. Used to correlate the registration when no PR number is supplied. |
| `pr_number` | integer | no | The pull request number. Preferred: when present it takes precedence over `branch` for correlation. |
| `commit_sha` | string | yes | The commit the environment is running. A registration only drives verification while this matches the head of the branch being verified. |
| `preview_url` | string | yes | Absolute `http(s)` base URL of the live environment. Must be reachable from Aviator. |
| `secrets` | object | no | Credentials for driving the app as a signed-in test user. See [Credentials](#credentials-for-signing-in). |
| `expires_at` | string | no | ISO 8601 timestamp. When omitted, the config's `expires_default_sec` applies. |

`branch` is required even when you send `pr_number`, because every CI job knows its branch — including jobs that run before a pull request exists.

#### Responses

| Status | Meaning |
| ------ | -------------------------------------------------------------------------------------------------------------- |
| `202` | Registration accepted. The body carries `registration_id`, `status`, `url`, `message`, and `expires_at`. |
| `400` | Invalid payload — including a repo whose Verify config declares no `method: endpoint` preview. |
| `401` | Missing or invalid API token. |
| `404` | Repository not found for this account. |

A `202` body looks like:

```json
{
"registration_id": 412,
"status": "attached",
"url": "https://app.aviator.co/r/1042",
"message": "Preview registered and attached to this Verify session.",
"expires_at": "2026-09-05T18:00:00+00:00"
}
```

`status` tells you what the registration found:

* **`attached`** — a verification session already existed for this pull request or branch, and the registration is now driving it. `url` links to that session.
* **`buffered`** — the registration arrived before the session existed. It's held and attaches automatically once the session is created, so CI outrunning session creation is expected and safe.

Registrations are append-only: re-POSTing records a new registration and the newest one wins, so blind CI retries are safe.

### Step 3: Add the CI step

A GitHub Actions example — adapt the shell and variables to your provider:

```yaml
- name: Register preview with Aviator
run: |
curl -sS -X POST https://api.aviator.co/api/v1/verify/preview-environment \
-H "Authorization: Bearer $AVIATOR_API_TOKEN" \
-H "Content-Type: application/json" \
-d @- <<EOF
{
"repository": {"org": "acme-corp", "name": "webapp"},
"branch": "${GITHUB_HEAD_REF}",
"pr_number": ${{ github.event.pull_request.number }},
"commit_sha": "${{ github.event.pull_request.head.sha }}",
"preview_url": "https://pr-${{ github.event.pull_request.number }}.preview.acme.dev",
"secrets": {"TEST_USER_PASSWORD": "..."}
}
EOF
```

Store the token as a CI secret (`AVIATOR_API_TOKEN` above) — never inline it in the workflow file.

### Credentials for signing in

The optional `secrets` object carries the credentials Verify needs to **drive your app as a signed-in user** — a session cookie value, a test user's password. These are for the user agent operating the app, not infrastructure secrets; an endpoint preview is already deployed by the time Aviator sees it, so it never needs your database password or third-party keys.

Your [Verify skill](writing-a-skill-md.md) references them with the same placeholder syntax as account secrets:

```markdown
3. Fill the password field with `{{ secrets.TEST_USER_PASSWORD }}`.
```

The contract:

* **Resolution.** Registered values resolve alongside your account secrets. On a name collision, the registration's value wins for that run.
* **Lifetime.** Values are scoped to the registration — they're deleted when it expires or is deregistered.
* **Handling.** Encrypted at rest, never shown to the verification agent: values are substituted outside the model conversation and scrubbed from captured evidence.

### What a verification run does

When a run reaches work that needs the preview and no current registration exists, it **waits** — the session shows that it's waiting for CI to register a preview — for up to `wait_timeout_sec`.

* **The registration arrives in time.** Verification proceeds against the registered URL.
* **The wait times out.** Verification still completes, using code analysis only; runtime scenarios aren't exercised. This is a degraded result, not a failure — and a registration that lands afterwards starts a follow-up verification automatically.
* **The registration doesn't match the branch head.** A registration whose `commit_sha` isn't the head of the branch being verified never drives the run. The session shows which commit is registered against which commit it's verifying — usually a sign the deploy job is lagging behind a newer push.

Tune `wait_timeout_sec` to your deploy time, with headroom for a queued runner. Too short and runs degrade on a slow build; too long and a broken deploy job holds the run open.

### Re-registering

Registering again is the normal way to move a preview forward:

* **A new commit.** Registering a different `commit_sha` re-verifies it. If that commit is the branch head and was previously verified without a preview, a follow-up verification starts automatically.
* **The same commit at a new URL.** A redeploy — for example, a rebuilt environment on a fresh hostname — makes the new URL current without triggering a re-run.

### Expiry and deregistration

A registration stops driving runs at `expires_at` (or after `expires_default_sec` when the registration didn't set one). At that point its credentials are removed and no new run uses it.

To release it earlier — from a CI teardown job, or when the environment is destroyed on PR close — send a `DELETE` to the same path with the same authentication:

```bash
curl -sS -X DELETE https://api.aviator.co/api/v1/verify/preview-environment \
-H "Authorization: Bearer $AVIATOR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"repository": {"org": "acme-corp", "name": "webapp"}, "branch": "feature/checkout", "pr_number": 412}'
```

Deregistration is idempotent: an unknown or already-deleted registration returns `200` with `{"deleted": 0}`, so teardown jobs can run unconditionally. Registered credentials are deleted immediately.

### Next steps

* [Preview YAML reference](../reference/preview-yaml.md) — every preview field
* [Writing a Verify skill](writing-a-skill-md.md) — teach the runner how to drive your app
* [Concepts: Previews](../concepts/previews.md) — how previews fit into verification
Loading