Skip to content

Commit cc0642f

Browse files
test(e2e): run the testkit conformance suite in the host lanes (#5078)
* test(e2e): run the testkit conformance suite in the host lanes The Docker, Podman, Kubernetes and VM host lanes, including external-driver mode and e2e:rust:push, now run the testkit `sandbox` package with nextest instead of the old CLI conformance package. The tests find the lane's gateway the same way the CLI does, through the OPENSHELL_GATEWAY_ENDPOINT and OPENSHELL_GATEWAY values the gateway wrappers export, and `--no-tests fail` makes a lane that selects no tests fail. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(conformance): delete the old CLI conformance suite Nothing runs tests/suites/conformance/cli now that the host lanes run the testkit suite. Remove it from the conformance workspace and its lockfile, drop the e2e:testkit exclusion for it, and point the TESTING.md manual run at the `sandbox` package. e2e:rust:push also loses its dependency on e2e:conformance:build, which no longer exists, and the unused OPENSHELL_CONFORMANCE_BIN it set. crates/openshell-conformance stays for the features and drivers suites. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(e2e): drop the package filter from the host-lane conformance run The conformance workspace has one package now that the old CLI suite is gone, so run the whole workspace. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(conformance): make the conformance family one crate Conformance is one crate per family, so the family directory is the crate: tests/suites/conformance/sandbox becomes the `conformance` package at tests/suites/conformance, with its suites still in tests/. The workspace manifest path, its lockfile and the scripts that run it by path are unchanged. Nextest's binaries for these tests are now `conformance::<suite>`; test names are unchanged. The tmachine runner rows, the fmt and clippy lines and the docs name the new package. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 4782868 commit cc0642f

24 files changed

Lines changed: 80 additions & 264 deletions

File tree

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ descriptions, events, and current/previous container logs. A CNI rollout timeout
131131
occurs before the listener tests run; use these diagnostics to distinguish image
132132
pulls, init failures, and readiness failures.
133133

134-
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-sandbox-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
134+
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-conformance-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
135135

136136
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL (exit 1) after the end of the nextest log (which names the failed tests), or ERROR (exit 2) with the step that broke. FAIL and ERROR also appear as annotations on the run's summary page. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
137137

‎.github/workflows/branch-checks.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,7 @@ jobs:
233233
cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
234234
cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check
235235
cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check
236-
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox -- --check
236+
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance -- --check
237237
cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check
238238
cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check
239239
@@ -297,7 +297,7 @@ jobs:
297297
cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
298298
cargo clippy --locked --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings
299299
cargo clippy --locked --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings
300-
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --all-targets -- -D warnings
300+
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml -p conformance --all-targets -- -D warnings
301301
cargo clippy --locked --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings
302302
cargo clippy --locked --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings
303303

‎.github/workflows/e2e-suites.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,11 @@ jobs:
6464
# installer registers with the CLI: `openshell` for the packages,
6565
# `tmachine` for the k3s installers.
6666
rows='[
67-
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
68-
{"environment":"fedora-podman-rootful","installer":"rpm","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
69-
{"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
70-
{"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"sandbox","gateway":"tmachine","runner":"ubuntu-24.04"},
71-
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"sandbox","gateway":"tmachine","runner":"ubuntu-24.04"},
67+
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
68+
{"environment":"fedora-podman-rootful","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
69+
{"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
70+
{"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
71+
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
7272
{"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}
7373
]'
7474
matrix=$(jq -c --arg family "${FAMILY}" \

‎CI.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ JUnit report, nextest log and, after a failure, the guest's diagnostics as an
204204

205205
### Run a subset of the conformance tests
206206

207-
The conformance lanes run the `sandbox` package of `tests/suites/conformance`
207+
The conformance lanes run the `conformance` crate in `tests/suites/conformance`
208208
through `.github/workflows/e2e-suites.yml`, which keeps the list of
209209
environments. To run one suite in one environment from a checkout, build the
210210
candidate inputs (see [TESTING.md](TESTING.md)) and narrow the tests with a
@@ -213,8 +213,8 @@ nextest filterset:
213213
```shell
214214
nix run .#openshell-test-runner -- \
215215
--environment ubuntu-docker-rootful --installer deb \
216-
--family conformance --package sandbox \
217-
--filter 'binary_id(=sandbox::policy_advisor)'
216+
--family conformance --package conformance \
217+
--filter 'binary_id(=conformance::policy_advisor)'
218218
```
219219

220220
A local run does not replace the required PR E2E gate.

‎TESTING.md‎

Lines changed: 21 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ OpenShell that the VM's installer set up.
223223
```shell
224224
nix run .#openshell-test-runner -- \
225225
--environment ubuntu-docker-rootful --installer deb \
226-
--family conformance --package sandbox
226+
--family conformance --package conformance
227227
```
228228

229229
It needs KVM, and tmachine only boots guests of the host's architecture. The
@@ -412,23 +412,23 @@ HTTP `Host`, TLS SNI, and mTLS handling.
412412
Suites:
413413

414414
- Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
415-
- CLI conformance (`tests/suites/conformance`) - portable Cargo tests for
416-
lifecycle, mechanistic drafts, file transfer, and the sandbox-local API,
417-
including agent-authored permission requests. Driver E2E runs the complete
418-
Cargo test package. The installed-artifact conformance lanes run the `sandbox`
419-
package through `openshell-test-runner` (see below).
415+
- Conformance (`tests/suites/conformance`) - the testkit `conformance` crate:
416+
lifecycle, the CLI, persistent workspaces, file transfer, and the policy
417+
advisor. Driver E2E runs it with nextest against the lane's gateway. The
418+
installed-artifact conformance lanes run the same crate through
419+
`openshell-test-runner` (see above).
420420
- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
421-
`e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
421+
`e2e-vm`) - conformance plus the common and driver-specific coverage for
422422
the selected deployment.
423423
- Docker suite (`--features e2e-docker`) - includes Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start.
424424
- Docker GPU suite (`--features e2e-docker-gpu`) - Docker suite plus GPU sandbox smoke coverage.
425-
- Configuration push suite (`--features e2e-config-push`) - CLI conformance and the policy tests against a Docker gateway with `config_delivery_mode = "push"`. Run it with `mise run e2e:rust:push`.
425+
- Configuration push suite (`--features e2e-config-push`) - conformance and the policy tests against a Docker gateway with `config_delivery_mode = "push"`. Run it with `mise run e2e:rust:push`.
426426
- VM suite (`--features e2e-vm`) - runs e2e tests on a VM.
427427
- Kubernetes credential-driver suite (`--features e2e-kubernetes-credential-drivers`) - targeted Kubernetes Secrets and Vault provider credential storage coverage.
428428

429-
The Docker, Podman, Kubernetes, and VM runners execute CLI
430-
conformance before their Rust tests. Set the driver's feature variable
431-
(`OPENSHELL_E2E_DOCKER_FEATURES`, `OPENSHELL_E2E_PODMAN_FEATURES`,
429+
The Docker, Podman, Kubernetes, and VM runners execute conformance before
430+
their Rust tests; a run with no conformance tests fails. Set the driver's
431+
feature variable (`OPENSHELL_E2E_DOCKER_FEATURES`, `OPENSHELL_E2E_PODMAN_FEATURES`,
432432
`OPENSHELL_E2E_KUBERNETES_FEATURES`, or `OPENSHELL_E2E_VM_FEATURES`) to
433433
`e2e-conformance-only` to skip the Rust tests; the runner logs a notice when
434434
it skips them. Unset or empty values select the driver's default Rust
@@ -454,52 +454,27 @@ Run the Docker-backed Rust CLI e2e suite:
454454
mise run e2e:docker
455455
```
456456

457-
Run the portable CLI conformance suite against the gateway selected in your
458-
OpenShell CLI configuration:
457+
Run the conformance suite against the gateway selected in your OpenShell CLI
458+
configuration:
459459

460460
```shell
461461
cargo build --package openshell-cli
462462
OPENSHELL_BIN="$PWD/target/debug/openshell" \
463-
cargo test \
463+
cargo nextest run \
464464
--locked \
465465
--manifest-path tests/suites/conformance/Cargo.toml \
466-
--package openshell-test-conformance-cli \
467-
--no-fail-fast \
468-
-- \
469-
--test-threads=1 \
470-
--nocapture
466+
--test-threads 1 \
467+
--no-fail-fast
471468
```
472469

473470
The gateway must already be installed, reachable, and selected before the tests
474471
start. The test suite does not provision a gateway or select a compute driver.
475472
Set `OPENSHELL_BIN` to another executable to test a different prebuilt CLI.
473+
[tests/testkit/README.md](tests/testkit/README.md) covers gateway selection and
474+
cleanup.
476475

477-
The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without
478-
depending on how the gateway was installed or which driver is configured. It
479-
requires machine-readable gRPC status, creates a uniquely named detached
480-
sandbox with the configured default image, verifies the sandbox is `Ready` by finding its
481-
unique name in paginated JSON list output, executes `echo` with a run-specific
482-
marker, deletes the sandbox, and verifies that its name no longer appears.
483-
Driver suites enable the same profile
484-
instead of maintaining a separate smoke implementation. Sandbox lifecycle,
485-
label matrices, VM overlay, and TLS-key permission assertions remain regular
486-
E2E coverage.
487-
488-
Each invocation prints a ten-character run ID before creating resources.
489-
Conformance sandboxes use names such as `ct-<run-id>-01`. The runner tracks the
490-
exact name and uses it for cleanup; phase 1 does not add ownership labels.
491-
492-
The runner deletes owned resources after both success and failure. If the test
493-
process is interrupted before cleanup, locate leftovers without touching
494-
unrelated gateway state:
495-
496-
```shell
497-
openshell sandbox list --output json
498-
openshell sandbox delete <sandbox-name>
499-
```
500-
501-
Gateway-backed Rust E2E tasks run the CLI conformance Cargo tests against the
502-
configured gateway, then run any lane-specific Rust tests that still apply.
476+
Gateway-backed Rust E2E tasks run the conformance suite against the configured
477+
gateway, then run any lane-specific Rust tests that still apply.
503478
Run the Podman-backed Rust CLI e2e suite:
504479

505480
```shell
@@ -528,7 +503,7 @@ rebuilds only the RPMs. Run conformance against the installed RPMs:
528503
nix run .#build-artifacts
529504
nix run .#openshell-test-runner -- \
530505
--environment fedora-podman-rootful --installer rpm \
531-
--family conformance --package sandbox
506+
--family conformance --package conformance
532507
```
533508

534509
Run the portable subset in a disposable rootless Podman guest:

‎e2e/policy-advisor/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,14 @@ requests through `policy.local`. See [TESTING.md](../../TESTING.md#rust-cli-e2e)
6262
for the supported source and installed-artifact test paths.
6363

6464
To run only the policy-related conformance tests against an installed
65-
candidate, run the `policy-advisor` suite of the `sandbox` package through the
65+
candidate, run the `policy-advisor` suite of the `conformance` crate through the
6666
test runner:
6767

6868
```bash
6969
nix run .#openshell-test-runner -- \
7070
--environment ubuntu-docker-rootful --installer deb \
71-
--family conformance --package sandbox \
72-
--filter 'binary_id(=sandbox::policy_advisor)'
71+
--family conformance --package conformance \
72+
--filter 'binary_id(=conformance::policy_advisor)'
7373
```
7474

7575
The GitHub write test above and the regressions below still exercise distinct

‎e2e/support/conformance.sh‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,15 +30,16 @@ e2e_run_openshell_conformance() {
3030

3131
e2e_require_openshell_bin || return
3232

33+
# The tests find the gateway as the CLI does: OPENSHELL_GATEWAY_ENDPOINT,
34+
# else OPENSHELL_GATEWAY, else the active gateway. The gateway wrappers set
35+
# these for the gateway they started.
3336
echo "==> Running conformance suites against the ${gateway_label} gateway"
34-
cargo test \
37+
cargo nextest run \
3538
--locked \
3639
--manifest-path "${root}/tests/suites/conformance/Cargo.toml" \
37-
--package openshell-test-conformance-cli \
40+
--test-threads 1 \
3841
--no-fail-fast \
39-
-- \
40-
--test-threads=1 \
41-
--nocapture
42+
--no-tests fail
4243
}
4344

4445
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then

‎tasks/rust.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ run = [
3939
"cargo clippy --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings",
4040
"cargo clippy --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings",
4141
"cargo clippy --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings",
42-
"cargo clippy --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --all-targets -- -D warnings",
42+
"cargo clippy --manifest-path tests/suites/conformance/Cargo.toml -p conformance --all-targets -- -D warnings",
4343
"cargo clippy --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings",
4444
"cargo clippy --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings",
4545
]
@@ -57,7 +57,7 @@ run = [
5757
"cargo fmt --manifest-path tests/testkit/Cargo.toml --all",
5858
"cargo fmt --manifest-path tests/tmachine/Cargo.toml --all",
5959
"cargo fmt --manifest-path tests/runner/Cargo.toml --all",
60-
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox",
60+
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance",
6161
"cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh",
6262
"cargo fmt --manifest-path tests/support/Cargo.toml --all",
6363
]
@@ -74,7 +74,7 @@ run = [
7474
"cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check",
7575
"cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check",
7676
"cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check",
77-
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox -- --check",
77+
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p conformance -- --check",
7878
"cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check",
7979
"cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check",
8080
]

‎tasks/test.toml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -169,13 +169,12 @@ run = "OPENSHELL_TEST_CONTAINER_ENGINE=podman e2e/with-podman-gateway.sh cargo t
169169

170170
["e2e:testkit"]
171171
description = "Run the testkit suites in tests/suites/conformance against a Docker-backed gateway"
172-
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --exclude openshell-test-conformance-cli --test-threads 1 --no-fail-fast"
172+
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --test-threads 1 --no-fail-fast"
173173

174174
["e2e:rust:push"]
175-
description = "Run CLI conformance and policy e2e tests against a Docker gateway that pushes configuration"
176-
depends = ["e2e:conformance:build"]
175+
description = "Run conformance and policy e2e tests against a Docker gateway that pushes configuration"
177176
env = { OPENSHELL_CONFIG_DELIVERY_MODE = "push", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-config-push", OPENSHELL_E2E_DOCKER_TEST = "config_push live_policy_update policy_activation" }
178-
run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh"
177+
run = "e2e/rust/e2e-docker.sh"
179178

180179
["e2e:workload:build"]
181180
description = "Build the Docker E2E workload fixture"

‎tests/suites/conformance/Cargo.lock‎

Lines changed: 10 additions & 29 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)