Skip to content

[CI] Record per-test timings in run-cpu-test.sh containers - #67337

Draft
Thangnguyenvn98 wants to merge 2 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/cpu-test-otel-timings
Draft

Thangnguyenvn98 wants to merge 2 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/cpu-test-otel-timings

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Trunk change: needs line-by-line review (a shared .buildkite script that every hardware_tests/cpu.yaml job runs).

The x86 CPU jobs in hardware_tests/cpu.yaml record no per-test timings on main. On main builds the pipeline generator traces each job's command on the host and sets a trace ID, but run-cpu-test.sh starts its own container and passes none of that in, so pytest inside it runs untraced. This forwards the trace context into the container and loads the existing CI timing helpers there, so these jobs record per-test timings like the other test jobs. It is a first step toward automatic runtime sharding for the CPU jobs.

Claims

  • On main builds, pytest inside run-cpu-test.sh's container records one timing span per test, parented to the job's command span, and uploads them with the job's own Buildkite identity.
  • PR builds and local runs are unchanged: with no trace ID in the environment, the script adds no docker run flags and no setup.
  • A test failure still fails the job with the same exit status, and the offline-to-online Hugging Face retry is untouched.
  • Tracing can't fail a job: if the timing helpers are missing or don't parse, the container prints tracing setup skipped and runs the tests as before.

Validation

  • Local end-to-end run of the real script, with docker build stubbed to a small pytest image and host-only flags (--cpuset-*, --privileged) stripped:
    • with a trace ID: both demo tests spooled as pytest.test spans with their node IDs, parented to the host command span;
    • without a trace ID: no extra flags, no spans, tests run as before;
    • a failing pytest exits 1 in both modes.
    • with the helper script removed from the image, or with a syntax error in it: tracing setup skipped, tests pass, exit 0. On the first commit of this PR both cases failed the job, which the second commit fixes.
  • shellcheck with the repo's pre-commit config: clean.
  • Real CI on the current head, cpu-quantization-model-tests only, with main-build tracing turned on for the PR ref (build 102568, https://buildkite.com/vllm/ci/builds/102568): the job passed with the same pass and skip counts per command as main build 102556 (1; 11 + 1 skipped; 1; 2 + 4 skipped). In the container the helpers loaded with no "tracing skipped" message, pytest recorded spans, and the exit handler attempted the upload. The upload got 401, as expected: the timing ingest accepts only main and its treatment branch, and this build ran on refs/pull/67337/head. The first main build after merge is what shows the timings landing for this job. Details are in the validation comment on this PR.

Details

  • Explicit list of forwarded variables rather than every BUILDKITE_*/CI_INFRA_* variable: the host's CI_INFRA_OTEL_* directory variables point at host paths that don't exist in the container. The list is what .buildkite/scripts/ci-otel/ reads.
  • Setup path is fixed to /vllm-workspace/.buildkite/scripts/ci-otel, which the CPU vllm-test image already contains; the image has no .git, so the helpers' own lookup can't find it.
  • Gated on CI_INFRA_TRACE_ID, which the generator's tracing sets only on main builds. An ordinary PR build has no trace ID, so nothing changes there.
  • Known limit: when the offline attempt misses the cache and the script retries online, the tests that ran before the miss are timed twice. The timings service leaves a build with a repeated test out of that step's baseline, so such a build is skipped rather than skewing the numbers.
  • Setup is loaded inside an if, the same form the pipeline generator uses for its own setup, so set -e in the container can't turn a tracing problem into a job failure.
  • Not in this PR: run-cpu-test-arm.sh (the same change, in its own PR for the Arm owners), and enrolling the CPU jobs in automatic sharding (they still use no_plugin and pytest lines inside a bash string).

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

Thangnguyenvn98 and others added 2 commits November 13, 2026 13:44
On main builds the pipeline generator traces each CPU test command on
the host, but the container this script starts gets none of that
context, so pytest inside it never records per-test timings. Forward
the trace context and the Buildkite job variables the CI timing helpers
need, and load .buildkite/scripts/ci-otel/ci_otel.sh in the container
when a trace ID is present. PR builds and local runs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Load the ci-otel helpers inside an `if`, as the pipeline generator's own
setup does, so a missing or broken helper skips tracing instead of
failing the job under `set -e`. Write the forwarded variables as one
array literal, which also keeps them to one line of `set -x` output.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/cpu-test-otel-timings branch from e39546d to aeb2382 Compare November 13, 2026 18:44
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

Validation build for the current head (aeb2382043)

Build: https://buildkite.com/vllm/ci/builds/102568

It runs cpu-quantization-model-tests only, on this PR's ref, with main-build tracing turned on for that ref. A normal PR build can't exercise this change, because tracing is off there and the script then adds nothing.

Check Result
Job passed, no retries
Tests per command, against main build 102556 identical: 1 passed; 11 passed, 1 skipped; 1 passed; 2 passed, 4 skipped
Trace context reaches the container yes, the docker run line carries the forwarded variables
Helpers load in the container yes, no tracing setup skipped
pytest records per-test spans yes, the container's exit handler had spans to upload
Upload refused with 401, as expected on a PR ref

The baseline is the newest main run of this job before the PR's base commit; the job's test files did not change in between.

Things a reviewer may stop on:

  • Two CI timing upload skipped: HTTP Error 401 lines. One is the container's upload and one is the host's own, which was already there before this PR. The timing service accepts uploads only from main and the tracing treatment branch, and this build ran on the PR ref. It goes away on main, and an upload failure never fails the job.
  • Timings arriving are not shown here. No build we can start on these agents is accepted by the timing service. The same helpers do upload and land from a CPU test container on another queue (the staging build of [pipeline-generator] Find the tracing helpers in the CPU test image ci-infra#908, https://buildkite.com/vllm/ci/builds/102535). The first main build after merge is the check for this job.
  • PR builds are unchanged. With no trace ID the script adds no docker run flags and no setup; the local runs in the description cover that case, and the case where the helper is missing or broken.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant