Repository navigation
Description
User Story
I maintain OpenShell's end-to-end tests. When a family of tests moves from e2e/rust onto testkit, the tmachine runner has to give it the timeouts and concurrency the old lanes gave it, so a ported test doesn't pass in the old lane and time out in the runner.
Description
The runner runs every family with one fixed nextest profile: 30 s slow-timeout × 4 and one test thread. The old e2e/rust lanes get more from the repository's .config/nextest.toml: per-test slow-timeout overrides and test groups such as kubernetes-ha and kubernetes-shared-supervisor-pool. The runner passes its own --config-file, so that file doesn't apply to runner rows.
Add the runner side of this when the first batch that needs it is ready:
- Pass the runner's
openshellprofile as a nextest tool config (--tool-config-file openshell-test-runner:<path>) instead of--config-file, so a family's own config is read. - Have the bundle carry
tests/suites/<family>/.config/nextest.tomlinto the remapped workspace, so a family can set per-test timeouts and test groups, or override the profile. - Add a
--test-threads Nrunner flag. The default of 1 keeps today's serial--no-capturerun. Parallel runs lose streamed output, so document that. - Add a
test_threadscolumn to thee2e-suites.ymlrows so CI can set it.
An implementation was written and reviewed in #5277 (commit 8441388e5, reverted before merge to keep that PR small). Start from it.
Context
Likely first users:
- The Kubernetes batch: shared-supervisor, HA and rebalancing tests need the existing test groups and longer timeouts.
credential_gatingafter test(e2e): split credential_gating into independent parallel tests #5264 splits it: independent tests with unique names can run in parallel.- Any common batch that wants to run in parallel so lanes don't get slower as tests move in.
settings_management::settings_global_override_round_trip, which PR1 (test(conformance): port sandbox lifecycle and API e2e tests to testkit #5431) left ine2e/rust. It changes a gateway-wide setting, so it needs a serial test group before it moves to conformancesettings.sandbox_setting_round_tripalso clears and restores the gateway-wideocsf_json_enabled, so it belongs in the same group.
Not needed by the first conformance, Podman or Docker batches if they run serially under the default profile.
Part of #5281.
Definition of Done
- A family can set per-test timeouts and test groups through
tests/suites/<family>/.config/nextest.toml, and they apply under the runner. -
--test-threadsis configurable from the runner and from thee2e-suites.ymlrows; the default is unchanged. - Runner unit tests cover the nextest arguments and the family config copy.
-
TESTING.mddocuments the behavior.
Activity
matthewgrossman commentedon Nov 9, 2026
Heads-up from #5332: conformance rows now run the prebuilt
openshell-conformanceprogram (--program), so they don't readnextest.tomlat all. Whatever this issue adds for per-family timeouts and thread counts needs a program equivalent too (run --timeout/--grace; the program already runs tests serially), or conformance rows will silently skip it. Tracked with the other #5332 follow-ups in #5416.test(e2e): keep the on-failure restart and global settings tests in e…
elezar commentedon Nov 12, 2026
Tracking note from the review of #5431.
settings::global_setting_overrides_sandbox_scopechanges a gateway-wide setting, so nothing else may run beside it. #5431 leaves it out of conformance and keeps the legacye2e/rust/tests/settings_management.rsuntil the runner can isolate it.What this issue needs to provide for it to move: a serial test group (for example a nextest
test-groupwithmax-threads = 1) that the global-setting test can opt into, applied on every conformance row.For context, the legacy lanes did not isolate this test either. On Docker it ran alone only because
settings_management.rsis a single-test binary run bycargo test. On Kubernetes, nextest runs each test in its own process in parallel, and thee2e-kubernetesprofile only defines test groups for the HA and shared-supervisor-pool tests.settings::sandbox_setting_round_tripstays in conformance. It also clears and restores the global value for its key, so it relies on nothing else writing that key in the same run.