Skip to content

ci(e2e): cut tmachine E2E suite row setup time #5429

Closed

Description

@matthewgrossman

User Story

As a contributor migrating E2E coverage to testkit suites (#5281), I want each tmachine E2E suite row to spend most of its time running tests, so that adding a features package doesn't add ~6 minutes of setup to every PR run.

Description

A features row spends about 11 seconds running tests and about 5.5 minutes on setup. Most of that setup repeats work that could be cached or done once per run.

Breakdown for features/host-gateway on fedora-podman-rootful (run 42707934133, job 129484694793, "Run the suites" step 342s):

Phase Time Notes
Job, Nix, cache setup and input download ~35s
Fetch openshell-test-runner Nix closure ~65s 581 paths, 2.7 GiB, every job
cargo build of testkit-ready (musl) 85s No cargo cache
cargo build of the features tests (musl) 82s Separate workspace and target dir, so the SDK, tonic and aws-lc compile a second time
VM install layer ~94s Wait for SSH 18s, RPM install, image copy and load ~30s, gateway start. Misses the cache on every commit because the hash covers the RPMs and images
Tests 11s

Other features rows in the same run: oci-image podman-rootful 7m (step 385s), host-gateway docker 5m (step 248s), host-gateway k3s 11m.

Conformance rows avoid the two cargo builds because they use the prebuilt openshell-conformance bundle from #5332 (podman-rootful conformance takes 4.7m total). Features rows still build in each job.

Causes

  • Per-row cargo builds. tests/runner builds testkit-ready and then the suite bundle in each job, with no cargo cache. The two builds use different workspaces and target dirs.
  • Large runner closure. In tests/tmachine/config.nix, the testRunner runtimeEnv folds in every toolchain (aarch64 and x86_64, gnu and musl cross gcc). qemuPkgs.qemu.override { hostCpuOnly = true; } still pulls in GUI dependencies (SDL2, ALSA, at-spi).
  • One VM per row. Each features package gets its own job and VM, so the install layer is paid once per package.
  • Install layer cache misses. The layer hash includes the per-commit RPMs and images, so the cached layer is never reused across commits.

Proposed fixes, roughly in order of payoff

  1. Build the features bundle once per workflow run and download it in each row, following the ci(e2e): run the tmachine conformance rows with the prebuilt openshell-conformance #5332 conformance pattern. At minimum, build testkit-ready once, or share a target dir and add a cargo cache.
  2. Slim the runner closure: host-arch toolchains only, and a headless QEMU build. Moved to ci(e2e): shrink the Nix download of tmachine E2E suite rows #5433.
  3. Run all features packages for an environment in a single VM to amortize the install. Moved to ci(e2e): cut the VM install time of tmachine E2E suite rows #5438.
  4. Trim the install layer: split the cacheable base (packages, boot) from the per-commit artifacts, and speed up the image load. Moved to ci(e2e): cut the VM install time of tmachine E2E suite rows #5438.

Context

  • .github/workflows/e2e-suites.yml: rows matrix, "Run the suites" step, prebuilt conformance download, tmachine cache key
  • .github/workflows/branch-e2e.yml: build-conformance job (ci(e2e): run the tmachine conformance rows with the prebuilt openshell-conformance #5332 pattern)
  • tests/runner/src/main.rs, tests/runner/src/bundle.rs: build_ready, bundle::build, --program prebuilt path
  • tests/tmachine/config.nix: testRunner inputs and runtimeEnv, qemu override
  • tests/tmachine/src/qemu/install.rs, tests/tmachine/src/qemu/session.rs: install layer hash and overlay boot
  • flake.nix: toolchains and openshell-test-runner

Definition of Done

  • Features rows don't run cargo build in the row job; they consume a bundle built once per run.
  • testkit-ready is built at most once per workflow run.
  • The Nix closure fetched per row is handled in ci(e2e): shrink the Nix download of tmachine E2E suite rows #5433.
  • A Podman features row like host-gateway completes in under ~2.5 minutes.
  • Before and after timings are recorded in the PR.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions