Skip to content

Commit 582dd23

Browse files
docs(tests): document test plans, the runner and the conformance suite
Describe how a plan runs on CI and locally in TESTING.md and a new tests/plans/README.md, add a README for the sandbox suite, point the testkit README at the runner, and add rows for tests/suites, tests/tmachine, tests/runner, tests/plans and tests/environments to the project-structure tables in AGENTS.md and CONTRIBUTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent edb45fb commit 582dd23

6 files changed

Lines changed: 225 additions & 5 deletions

File tree

‎AGENTS.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,11 @@ Do not rely on this file for a full inventory. The detailed public and contribut
6262
| `python/openshell/` | Python SDK | Python bindings and CLI packaging |
6363
| `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |
6464
| `proto/` | Protobuf definitions | gRPC service contracts |
65-
| `tests/testkit/` | E2E testkit | Support library and `#[testkit::test]` attribute for SDK-driven E2E suites |
65+
| `tests/testkit/` | E2E testkit | Support library, `#[testkit::test]` attribute, and `testkit-spec` parser for SDK-driven E2E suites |
66+
| `tests/suites/` | E2E suites | One Cargo workspace per test family; `conformance/sandbox` is the first testkit suite |
67+
| `tests/tmachine/` | Test machines | QEMU machines provisioned with Ansible, and the `Session` API that drives them |
68+
| `tests/runner/` | Test runner | `openshell-test`: expands plans into jobs, runs them in tmachine environments, checks JUnit results, emits the CI matrix |
69+
| `tests/plans/`, `tests/environments/` | Test plans | Plans pair suite families with environments; environments say what each provides |
6670
| `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests |
6771
| `docs/` | Published docs | MDX pages, navigation, and content assets |
6872
| `fern/` | Docs site config | Fern site config, components, and theme assets |

‎CONTRIBUTING.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,10 @@ These are the primary `mise` tasks for day-to-day development:
345345
| `proto/` | Protocol buffer definitions and [public API conventions](proto/README.md) |
346346
| `tasks/` | `mise` task definitions and build scripts |
347347
| `tests/testkit/` | Support library for SDK-driven end-to-end suites |
348+
| `tests/suites/` | End-to-end suites, one Cargo workspace per family (`conformance/sandbox` runs through a plan) |
349+
| `tests/tmachine/` | QEMU test machines provisioned with Ansible, and the `Session` API that drives them |
350+
| `tests/runner/` | `openshell-test`: expands test plans into jobs, runs them in tmachine environments, and checks the results |
351+
| `tests/plans/`, `tests/environments/` | Test plans, and the environments they run in with the capabilities each provides |
348352
| `deploy/` | Dockerfiles, Helm chart, Kubernetes manifests |
349353
| `docs/` | Published Fern docs source, navigation, and content assets |
350354
| `fern/` | Fern site config, components, and theme assets |

‎TESTING.md‎

Lines changed: 88 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@ python/openshell/ # Python unit tests (*_test.py suffix)
1717
e2e/python/ # Python E2E tests (test_*.py prefix)
1818
e2e/rust/ # Rust CLI E2E tests
1919
tests/testkit/ # Support library for SDK-driven E2E suites
20+
tests/suites/conformance/ # Testkit suites (the `sandbox` crate) beside the older CLI conformance crate
21+
tests/tmachine/ # QEMU test machines, Ansible playbooks, and the Session API
22+
tests/runner/ # openshell-test: runs test plans in environments
23+
tests/plans/ # Test plans: which suites run in which environments
24+
tests/environments/ # Environment definitions and the capabilities they provide
25+
tests/waivers.yaml # Known bugs that skip a test in some environments
2026
```
2127

2228
## Rust Tests
@@ -140,10 +146,88 @@ by `OPENSHELL_BIN`. See
140146
[tests/testkit/README.md](tests/testkit/README.md) for how to write and run
141147
tests.
142148

143-
The `testkit-e2e` job in Branch E2E (`test:e2e` label) runs the suites under
144-
`tests/suites/sandbox` against a Docker-backed gateway with
145-
`mise run e2e:testkit`. The same task works locally. It isn't part of the Core
146-
E2E result yet.
149+
Suites live under `tests/suites/<family>/`. Each family is a Cargo workspace
150+
with its own lockfile, and a family name is what a plan's `type` selects.
151+
`tests/suites/conformance/sandbox` is the first testkit suite; it has one smoke
152+
test. Run it against a local Docker gateway with:
153+
154+
```shell
155+
mise run e2e:testkit
156+
```
157+
158+
That task runs `e2e/with-docker-gateway.sh`, which starts a gateway and exports
159+
`OPENSHELL_GATEWAY` and `OPENSHELL_BIN`, so the suite runs the same way the
160+
runner runs it in a guest. The `testkit-e2e` job in Branch E2E runs the same
161+
task on a Docker-backed gateway without a VM.
162+
163+
Workspaces that build root crates through path dependencies must lock shared
164+
crates at the versions in the root `Cargo.lock`; otherwise CI can build against
165+
dependencies the Nix toolchain doesn't support. `mise run rust:lockfiles:check`
166+
verifies this, and `uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix`
167+
re-locks a workspace from the root lockfile. Add new testkit workspaces to
168+
`ALIGNED_WORKSPACES` in that script.
169+
170+
### Test plans and the runner (`tests/runner/`)
171+
172+
`openshell-test` runs testkit suites in provisioned environments. It is the
173+
runner and plan layer from NVIDIA/OpenShell#4629 by Evan Lezar, on top of the
174+
testkit and tmachine here. Tests say what they need, environments say what they
175+
provide, and a plan selects which suites run where. The plan is the only list of
176+
what runs where: CI reads it, so no workflow or playbook repeats it.
177+
178+
- **Plans** (`tests/plans/*.yaml`) pair a family (a workspace under
179+
`tests/suites/`, such as `conformance`) and optionally some of its suites with
180+
environment IDs. See [tests/plans/README.md](tests/plans/README.md).
181+
- **Environments** (`tests/environments/*.yaml`) name the tmachine environment and
182+
installer that provision them, the gateway name the installer registers, and
183+
their capabilities: `compute_driver`, `rootless`, `gpu`,
184+
`persistent_workspace` and `service_exposure`. The runner passes them to the
185+
tests in `OPENSHELL_TEST_CAPABILITIES`, and points the tests at the registered
186+
gateway and the installed CLI with `OPENSHELL_GATEWAY` and `OPENSHELL_BIN`.
187+
- **Capabilities** are optional on a test:
188+
`#[testkit::test(requires = [persistent_workspace])]`. The runner leaves the
189+
test out of environments without it. `id = "t-0123abcd"` is optional too and
190+
gives a test an identity that survives renames. See
191+
[tests/testkit/README.md](tests/testkit/README.md#capabilities).
192+
- **Waivers** (`tests/waivers.yaml`) skip a test in some environments because of
193+
a known bug. Each names the test (by ID, or by function name when unique), an
194+
issue link and a reason, and the runner lists what it skipped.
195+
- **Results** are checked, not trusted. A job passes only if nextest succeeds and
196+
its JUnit report has a passing case for exactly the tests the runner selected.
197+
A test missing from the report, or skipped, fails the job. An `#[ignore]` test
198+
is not selected, so nextest's skipped case for it is accepted.
199+
- **Strict mode** (`--strict`, or `OPENSHELL_TEST_STRICT=1`) is for CI. It fails
200+
instead of skipping an environment the host can't provision, an environment
201+
where no test is left to run, or a plan that selects nothing. Without it the
202+
runner reports those and exits successfully, so a laptop can run the part of a
203+
plan it can boot.
204+
205+
```shell
206+
nix run .#test-runner -- test --plan tests/plans/cpu-tests.yaml --dry-run
207+
nix run .#test-runner -- test --plan tests/plans/cpu-tests.yaml --strict
208+
nix run .#test-runner -- matrix --host # every test against every environment
209+
```
210+
211+
`nix run .#test-runner` cross-builds the suites for the guest, boots each
212+
environment through tmachine (this needs KVM), runs nextest in the guest over SSH, and
213+
writes JUnit reports, logs and gateway diagnostics to
214+
`artifacts/test-results/<plan>/`. Every SSH and nextest step has a timeout, and
215+
the runner kills its VMs if it receives SIGTERM. Two runs on one host need
216+
different `TMACHINE_SSH_PORT` values (default 2222). The installer inputs, such
217+
as the Debian package and the images, come from `artifacts/`, built by
218+
`nix run .#build-artifacts` or downloaded by CI.
219+
220+
**CI runs the plan.** `.github/workflows/e2e-suites.yml` takes a plan name. It
221+
runs `openshell-test test --plan tests/plans/<name>.yaml --emit-github-matrix`,
222+
which prints the `{"include": [...]}` a workflow reads with `fromJSON`: one job
223+
per environment with the runner label that can boot it. Each job then runs
224+
`nix run .#test-runner -- test --plan ... --env <id> --strict` and uploads its
225+
JUnit reports and logs. Branch E2E calls the workflow for `cpu-tests` when the
226+
`test:e2e` label is set. To change what CI runs, edit the plan.
227+
228+
The older conformance, feature and driver lanes in `.github/workflows/` are still
229+
written by hand and run through the tmachine testsuites in
230+
`tests/tmachine/config.nix`. They move to plans as their suites move to testkit.
147231

148232
### Python E2E (`e2e/python/`)
149233

‎tests/plans/README.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Test plans
2+
3+
A plan lists which test suites run in which environments. CI and local runs read
4+
the same file, and nothing else repeats it: a workflow does not carry a copy of
5+
the lane list, and a suite does not carry a playbook that says where it runs.
6+
7+
```yaml
8+
matrix:
9+
- type: conformance # a family: a workspace under tests/suites/
10+
suites: [smoke] # optional; omit to run every suite in the family
11+
environments:
12+
- tmachine.ubuntu2404-amd64.docker-rootful.deb # IDs from tests/environments/
13+
```
14+
15+
A family is a Cargo workspace directly under `tests/suites/` with at least one
16+
testkit suite. A suite is a test file (`<crate>/tests/<suite>.rs`) containing
17+
`#[testkit::test]` functions. A test that `requires` a capability the
18+
environment does not provide is left out of that environment.
19+
20+
| Plan | Runs |
21+
|---|---|
22+
| `cpu-tests.yaml` | The `conformance` family on Ubuntu 24.04 amd64 with rootful Docker and the Debian package |
23+
24+
## How a plan runs on CI
25+
26+
Branch E2E calls `.github/workflows/e2e-suites.yml` with the plan name when the
27+
`test:e2e` label is set.
28+
29+
1. A planning job runs
30+
`openshell-test test --plan tests/plans/<plan>.yaml --emit-github-matrix`. It
31+
prints one job per environment, with the GitHub runner label that can boot it.
32+
2. Each job downloads the integration inputs the tmachine lanes use (the Debian
33+
and RPM packages and the images), then runs
34+
`nix run .#test-runner -- test --plan tests/plans/<plan>.yaml --env <id> --strict`.
35+
3. The runner builds the family's tests for the guest, boots the environment
36+
with tmachine, and runs nextest in it with `OPENSHELL_GATEWAY` set to the
37+
gateway the installer registered, `OPENSHELL_BIN` set to the installed CLI and
38+
`OPENSHELL_TEST_CAPABILITIES` set from the environment. It checks the JUnit
39+
report against the tests it selected: a missing or skipped test fails the job.
40+
4. The job uploads the JUnit reports and logs, and an aggregator job reports the
41+
result.
42+
43+
Adding an environment or a suite to a plan changes what CI runs with no workflow
44+
change. Arm64 environments are defined in `tests/environments/` but are not in a
45+
plan until a runner that can boot them with KVM exists.
46+
47+
## Running a plan locally
48+
49+
Print what a plan would run, without building or booting anything:
50+
51+
```shell
52+
nix run .#test-runner -- test --plan tests/plans/cpu-tests.yaml --dry-run
53+
```
54+
55+
Running it for real boots QEMU guests, so it needs KVM and the installer inputs in
56+
`artifacts/` (`nix run .#build-artifacts`). The runner skips an environment the
57+
host cannot boot unless you pass `--strict`, which is what CI does.
58+
59+
To run a suite against a gateway on your machine, without a guest, use the
60+
Docker gateway wrapper. It exports `OPENSHELL_GATEWAY` and `OPENSHELL_BIN`, which
61+
is how the runner points tests at the system under test:
62+
63+
```shell
64+
mise run e2e:testkit
65+
```
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Sandbox conformance suite
2+
3+
The `sandbox` crate holds testkit suites for sandbox behavior that every compute
4+
driver must provide. It is a member of the `conformance` workspace, which a plan
5+
selects with `type: conformance`. Each file in `tests/` is one suite; suite names
6+
are the file names.
7+
8+
| Suite | Covers |
9+
|---|---|
10+
| `smoke` | Create a sandbox, run `echo` in it, delete it |
11+
12+
The suite is small on purpose. It proves that a plan can run a real test through
13+
the runner on CI. The older suites move to testkit by adding suites here (or a
14+
new family under `tests/suites/`) and listing them in a plan.
15+
16+
## Running it
17+
18+
Against a local Docker gateway:
19+
20+
```shell
21+
mise run e2e:testkit
22+
```
23+
24+
Against any gateway the CLI can reach, with the testkit environment variables
25+
described in [tests/testkit/README.md](../../../testkit/README.md):
26+
27+
```shell
28+
OPENSHELL_GATEWAY=<name> cargo nextest run --locked \
29+
--manifest-path tests/suites/conformance/Cargo.toml -p sandbox --test-threads 1
30+
```
31+
32+
On CI, `tests/plans/cpu-tests.yaml` runs it through
33+
[the plan runner](../../../plans/README.md).
34+
35+
## Adding a test
36+
37+
Add a `#[testkit::test]` function to a suite, or a new file in `tests/` for a new
38+
area. No playbook, test archive or workflow entry is needed: the runner discovers
39+
tests from source and the plan selects them.
40+
41+
## What a test may assert
42+
43+
These tests run on every compute driver, so they check the public contract and
44+
leave driver behavior alone.
45+
46+
- Assert only what the public contract promises: the proto comments and the
47+
published docs.
48+
- Assert `phase` and `exit_code`. Don't assert condition reasons.
49+
- Check that something did not happen by reading state, not by sleeping.
50+
- If a behavior is left to the driver, declare it with
51+
`#[testkit::test(requires = [...])]`.
52+
- Use the CLI for upload, download, connect and forward. Rely on its exit code
53+
and `--output json`.
54+
- Add a testkit helper instead of calling raw gRPC from a test.
55+
56+
Guidance adapted from NVIDIA/OpenShell#4629 by Evan Lezar.

‎tests/testkit/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,13 @@ time or the CLI does not report it connected.
139139
ephemeral gateway and export `OPENSHELL_GATEWAY` and `OPENSHELL_BIN`, so a
140140
suite runs under them unchanged.
141141

142+
`mise run e2e:testkit` does this for the suites in `tests/suites/conformance`.
143+
On CI the test runner (`tests/runner/`) does the same for a guest: it sets
144+
`OPENSHELL_GATEWAY` to the gateway the installer registered, `OPENSHELL_BIN` to
145+
the installed CLI and `OPENSHELL_TEST_CAPABILITIES` from the environment, then
146+
runs the plan's suites with nextest. See [TESTING.md](../../TESTING.md) and
147+
[tests/plans/README.md](../plans/README.md).
148+
142149
## Checks
143150

144151
```shell

0 commit comments

Comments
 (0)