Repository navigation
[CI] Record per-test timings in run-cpu-test.sh containers - #67337
Draft
Thangnguyenvn98 wants to merge 2 commits into
Draft
Thangnguyenvn98 wants to merge 2 commits into
Thangnguyenvn98 wants to merge 2 commits into
Conversation
6 tasks done
On main builds the pipeline generator traces each CPU test command on the host, but the container this script starts gets none of that context, so pytest inside it never records per-test timings. Forward the trace context and the Buildkite job variables the CI timing helpers need, and load .buildkite/scripts/ci-otel/ci_otel.sh in the container when a trace ID is present. PR builds and local runs are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Load the ci-otel helpers inside an `if`, as the pipeline generator's own setup does, so a missing or broken helper skips tracing instead of failing the job under `set -e`. Write the forwarded variables as one array literal, which also keeps them to one line of `set -x` output. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Thangnguyenvn98
force-pushed
the
agent/cpu-test-otel-timings
branch
from
November 13, 2026 18:44
e39546d to
aeb2382
Compare
Contributor
Author
Validation build for the current head (
|
| Check | Result |
|---|---|
| Job | passed, no retries |
| Tests per command, against main build 102556 | identical: 1 passed; 11 passed, 1 skipped; 1 passed; 2 passed, 4 skipped |
| Trace context reaches the container | yes, the docker run line carries the forwarded variables |
| Helpers load in the container | yes, no tracing setup skipped |
| pytest records per-test spans | yes, the container's exit handler had spans to upload |
| Upload | refused with 401, as expected on a PR ref |
The baseline is the newest main run of this job before the PR's base commit; the job's test files did not change in between.
Things a reviewer may stop on:
- Two
CI timing upload skipped: HTTP Error 401lines. One is the container's upload and one is the host's own, which was already there before this PR. The timing service accepts uploads only frommainand the tracing treatment branch, and this build ran on the PR ref. It goes away on main, and an upload failure never fails the job. - Timings arriving are not shown here. No build we can start on these agents is accepted by the timing service. The same helpers do upload and land from a CPU test container on another queue (the staging build of [pipeline-generator] Find the tracing helpers in the CPU test image ci-infra#908, https://buildkite.com/vllm/ci/builds/102535). The first main build after merge is the check for this job.
- PR builds are unchanged. With no trace ID the script adds no
docker runflags and no setup; the local runs in the description cover that case, and the case where the helper is missing or broken.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Trunk change: needs line-by-line review (a shared
.buildkitescript that everyhardware_tests/cpu.yamljob runs).The x86 CPU jobs in
hardware_tests/cpu.yamlrecord no per-test timings on main. On main builds the pipeline generator traces each job's command on the host and sets a trace ID, butrun-cpu-test.shstarts its own container and passes none of that in, so pytest inside it runs untraced. This forwards the trace context into the container and loads the existing CI timing helpers there, so these jobs record per-test timings like the other test jobs. It is a first step toward automatic runtime sharding for the CPU jobs.Claims
run-cpu-test.sh's container records one timing span per test, parented to the job's command span, and uploads them with the job's own Buildkite identity.docker runflags and no setup.tracing setup skippedand runs the tests as before.Validation
docker buildstubbed to a small pytest image and host-only flags (--cpuset-*,--privileged) stripped:pytest.testspans with their node IDs, parented to the host command span;tracing setup skipped, tests pass, exit 0. On the first commit of this PR both cases failed the job, which the second commit fixes.shellcheckwith the repo's pre-commit config: clean.cpu-quantization-model-testsonly, with main-build tracing turned on for the PR ref (build 102568, https://buildkite.com/vllm/ci/builds/102568): the job passed with the same pass and skip counts per command as main build 102556 (1; 11 + 1 skipped; 1; 2 + 4 skipped). In the container the helpers loaded with no "tracing skipped" message, pytest recorded spans, and the exit handler attempted the upload. The upload got401, as expected: the timing ingest accepts onlymainand its treatment branch, and this build ran onrefs/pull/67337/head. The first main build after merge is what shows the timings landing for this job. Details are in the validation comment on this PR.Details
BUILDKITE_*/CI_INFRA_*variable: the host'sCI_INFRA_OTEL_*directory variables point at host paths that don't exist in the container. The list is what.buildkite/scripts/ci-otel/reads./vllm-workspace/.buildkite/scripts/ci-otel, which the CPUvllm-testimage already contains; the image has no.git, so the helpers' own lookup can't find it.CI_INFRA_TRACE_ID, which the generator's tracing sets only on main builds. An ordinary PR build has no trace ID, so nothing changes there.if, the same form the pipeline generator uses for its own setup, soset -ein the container can't turn a tracing problem into a job failure.run-cpu-test-arm.sh(the same change, in its own PR for the Arm owners), and enrolling the CPU jobs in automatic sharding (they still useno_pluginand pytest lines inside a bash string).Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
🤖 Generated with Claude Code
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)