You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cc0642f
Browse filesBrowse the repository at this point in the historyBrowse files
test(e2e): run the testkit conformance suite in the host lanes (#5078)
* test(e2e): run the testkit conformance suite in the host lanes
The Docker, Podman, Kubernetes and VM host lanes, including external-driver
mode and e2e:rust:push, now run the testkit `sandbox` package with nextest
instead of the old CLI conformance package. The tests find the lane's
gateway the same way the CLI does, through the OPENSHELL_GATEWAY_ENDPOINT
and OPENSHELL_GATEWAY values the gateway wrappers export, and `--no-tests
fail` makes a lane that selects no tests fail.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(conformance): delete the old CLI conformance suite
Nothing runs tests/suites/conformance/cli now that the host lanes run the
testkit suite. Remove it from the conformance workspace and its lockfile,
drop the e2e:testkit exclusion for it, and point the TESTING.md manual run
at the `sandbox` package.
e2e:rust:push also loses its dependency on e2e:conformance:build, which no
longer exists, and the unused OPENSHELL_CONFORMANCE_BIN it set.
crates/openshell-conformance stays for the features and drivers suites.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(e2e): drop the package filter from the host-lane conformance run
The conformance workspace has one package now that the old CLI suite is gone, so run the whole workspace.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(conformance): make the conformance family one crate
Conformance is one crate per family, so the family directory is the crate: tests/suites/conformance/sandbox becomes the `conformance` package at tests/suites/conformance, with its suites still in tests/. The workspace manifest path, its lockfile and the scripts that run it by path are unchanged.
Nextest's binaries for these tests are now `conformance::<suite>`; test names are unchanged. The tmachine runner rows, the fmt and clippy lines and the docs name the new package.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Copy file name to clipboardExpand all lines: .agents/skills/watch-github-actions/SKILL.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,7 +131,7 @@ descriptions, events, and current/previous container logs. A CNI rollout timeout
131
131
occurs before the listener tests run; use these diagnostics to distinguish image
132
132
pulls, init failures, and readiness failures.
133
133
134
-
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-sandbox-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
134
+
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-conformance-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
135
135
136
136
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL (exit 1) after the end of the nextest log (which names the failed tests), or ERROR (exit 2) with the step that broke. FAIL and ERROR also appear as annotations on the run's summary page. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
Copy file name to clipboardExpand all lines: TESTING.md
+21-46Lines changed: 21 additions & 46 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -223,7 +223,7 @@ OpenShell that the VM's installer set up.
223
223
```shell
224
224
nix run .#openshell-test-runner -- \
225
225
--environment ubuntu-docker-rootful --installer deb \
226
-
--family conformance --package sandbox
226
+
--family conformance --package conformance
227
227
```
228
228
229
229
It needs KVM, and tmachine only boots guests of the host's architecture. The
@@ -412,23 +412,23 @@ HTTP `Host`, TLS SNI, and mTLS handling.
412
412
Suites:
413
413
414
414
- Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
415
-
-CLI conformance (`tests/suites/conformance`) - portable Cargo tests for
416
-
lifecycle, mechanistic drafts, file transfer, and the sandbox-local API,
417
-
including agent-authored permission requests. Driver E2E runs the complete
418
-
Cargo test package. The installed-artifact conformance lanes run the `sandbox`
419
-
package through `openshell-test-runner` (see below).
415
+
-Conformance (`tests/suites/conformance`) - the testkit `conformance` crate:
416
+
lifecycle, the CLI, persistent workspaces, file transfer, and the policy
417
+
advisor. Driver E2E runs it with nextest against the lane's gateway. The
418
+
installed-artifact conformance lanes run the same crate through
419
+
`openshell-test-runner` (see above).
420
420
- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
421
-
`e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
421
+
`e2e-vm`) - conformance plus the common and driver-specific coverage for
422
422
the selected deployment.
423
423
- Docker suite (`--features e2e-docker`) - includes Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start.
424
424
- Docker GPU suite (`--features e2e-docker-gpu`) - Docker suite plus GPU sandbox smoke coverage.
425
-
- Configuration push suite (`--features e2e-config-push`) - CLI conformance and the policy tests against a Docker gateway with `config_delivery_mode = "push"`. Run it with `mise run e2e:rust:push`.
425
+
- Configuration push suite (`--features e2e-config-push`) - conformance and the policy tests against a Docker gateway with `config_delivery_mode = "push"`. Run it with `mise run e2e:rust:push`.
426
426
- VM suite (`--features e2e-vm`) - runs e2e tests on a VM.
427
427
- Kubernetes credential-driver suite (`--features e2e-kubernetes-credential-drivers`) - targeted Kubernetes Secrets and Vault provider credential storage coverage.
428
428
429
-
The Docker, Podman, Kubernetes, and VM runners execute CLI
430
-
conformance before their Rust tests. Set the driver's feature variable
0 commit comments