Skip to content

test(runner): apply per-family nextest config and a thread count to runner rows #5295

Open

Description

@matthewgrossman

User Story

I maintain OpenShell's end-to-end tests. When a family of tests moves from e2e/rust onto testkit, the tmachine runner has to give it the timeouts and concurrency the old lanes gave it, so a ported test doesn't pass in the old lane and time out in the runner.

Description

The runner runs every family with one fixed nextest profile: 30 s slow-timeout × 4 and one test thread. The old e2e/rust lanes get more from the repository's .config/nextest.toml: per-test slow-timeout overrides and test groups such as kubernetes-ha and kubernetes-shared-supervisor-pool. The runner passes its own --config-file, so that file doesn't apply to runner rows.

Add the runner side of this when the first batch that needs it is ready:

  • Pass the runner's openshell profile as a nextest tool config (--tool-config-file openshell-test-runner:<path>) instead of --config-file, so a family's own config is read.
  • Have the bundle carry tests/suites/<family>/.config/nextest.toml into the remapped workspace, so a family can set per-test timeouts and test groups, or override the profile.
  • Add a --test-threads N runner flag. The default of 1 keeps today's serial --no-capture run. Parallel runs lose streamed output, so document that.
  • Add a test_threads column to the e2e-suites.yml rows so CI can set it.

An implementation was written and reviewed in #5277 (commit 8441388e5, reverted before merge to keep that PR small). Start from it.

Context

Likely first users:

  • The Kubernetes batch: shared-supervisor, HA and rebalancing tests need the existing test groups and longer timeouts.
  • credential_gating after test(e2e): split credential_gating into independent parallel tests #5264 splits it: independent tests with unique names can run in parallel.
  • Any common batch that wants to run in parallel so lanes don't get slower as tests move in.
  • settings_management::settings_global_override_round_trip, which PR1 (test(conformance): port sandbox lifecycle and API e2e tests to testkit #5431) left in e2e/rust. It changes a gateway-wide setting, so it needs a serial test group before it moves to conformance settings. sandbox_setting_round_trip also clears and restores the gateway-wide ocsf_json_enabled, so it belongs in the same group.

Not needed by the first conformance, Podman or Docker batches if they run serially under the default profile.

Part of #5281.

Definition of Done

  • A family can set per-test timeouts and test groups through tests/suites/<family>/.config/nextest.toml, and they apply under the runner.
  • --test-threads is configurable from the runner and from the e2e-suites.yml rows; the default is unchanged.
  • Runner unit tests cover the nextest arguments and the family config copy.
  • TESTING.md documents the behavior.

Activity

  1. matthewgrossman commented on Nov 9, 2026

    @matthewgrossman
    MemberAuthor

    Heads-up from #5332: conformance rows now run the prebuilt openshell-conformance program (--program), so they don't read nextest.toml at all. Whatever this issue adds for per-family timeouts and thread counts needs a program equivalent too (run --timeout/--grace; the program already runs tests serially), or conformance rows will silently skip it. Tracked with the other #5332 follow-ups in #5416.

  2. elezar commented on Nov 12, 2026

    @elezar
    Member

    Tracking note from the review of #5431.

    settings::global_setting_overrides_sandbox_scope changes a gateway-wide setting, so nothing else may run beside it. #5431 leaves it out of conformance and keeps the legacy e2e/rust/tests/settings_management.rs until the runner can isolate it.

    What this issue needs to provide for it to move: a serial test group (for example a nextest test-group with max-threads = 1) that the global-setting test can opt into, applied on every conformance row.

    For context, the legacy lanes did not isolate this test either. On Docker it ran alone only because settings_management.rs is a single-test binary run by cargo test. On Kubernetes, nextest runs each test in its own process in parallel, and the e2e-kubernetes profile only defines test groups for the HA and shared-supervisor-pool tests.

    settings::sandbox_setting_round_trip stays in conformance. It also clears and restores the global value for its key, so it relies on nothing else writing that key in the same run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions