Skip to content

[pipeline-generator] Find the tracing helpers in the CPU test image - #908

Merged
khluu merged 1 commit into
mainfrom
agent/otel-cpu-image-fallback
Nov 13, 2026
Merged

khluu merged 1 commit into
mainfrom
agent/otel-cpu-image-fallback

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Trunk change: needs line-by-line review (the pipeline generator's tracing setup, which runs in every traced job on main).

Summary

The CPU steps that the generator launches through the docker plugin record no per-test timings on main. Their logs say vLLM CI OTel: tracing setup skipped. The setup finds the tracing helpers through the git checkout, then through BUILDKITE_BUILD_CHECKOUT_PATH; the CPU test image has no .git, and the docker plugin passes no checkout path into the container, so both lookups fail. This has been the case since #480 moved the helpers into the vLLM checkout.

After this change the setup falls back to the copy the image already carries under /vllm-workspace. Without timings, these steps can't be enrolled in automatic sharding: the planner has nothing to plan from.

Decisions to check

  • A third lookup, not a new mechanism: the fallback is the default value of the existing BUILDKITE_BUILD_CHECKOUT_PATH expansion. Passing the checkout path into the container was the alternative; it names a host path the container can't see.
  • Fixed in the generator, not in the image: the image already has the helpers; the lookup just never checked there. Setting CI_INFRA_OTEL_DIR in the CPU Dockerfile would also work, but it needs an image rebuild, only helps commits built after it, and puts a CI path into each image that lacks .git. Copying .git into the image would add hundreds of MB to every CPU job's pull.
  • Order is unchanged: git checkout first, then the checkout path, then the image copy. Jobs where either earlier lookup works behave as before.
  • One length cap in an existing test goes from 500 to 520 characters. The setup line grew by 15 characters; the cap guards against inlining the helpers again, which this doesn't do.

What changed

  • On main, CPU docker-plugin steps start loading the tracing helpers, so pytest records per-test timings and the job uploads them at exit, like the GPU steps already do.
  • PR builds: no change. The rendered pipeline is byte-identical.

Not done

  • Only one CPU step was staged (cpu-tool-parsers). The other CPU docker-plugin steps use the same image and setup line, and get their first timings from main after merge.
  • Enrolling the CPU steps in automatic sharding is a follow-up in vllm, once main has timings for them.

Testing

  • python -m pytest buildkite/tests: 614 passed on the rebased commit. New test: with no git checkout and no checkout path, the setup loads the helper from the image workspace.
  • Mutation: the tool generates no mutants for these string lines, so three by hand (fallback removed, wrong path, image copy taking precedence over the checkout path); the tests fail on each.
  • Generator diff against main with the same inputs: PR scenario identical; main scenario, the fallback is the only difference in each traced step and no step is added or removed.
  • Staging build on the tracing treatment branch with this branch's generator, scoped to cpu-tool-parsers (https://buildkite.com/vllm/ci/builds/102535): passed with no retries. The job no longer logs tracing setup skipped; its tests match main (1501 passed, 1 skipped, 30 xfailed); and the timings service now answers for the step with 43 files whose times sum to the pytest run time, under the file names the planner uses. It had no timings for this step before.
  • Not checked: the agent slot after the build. The tracing helpers write only to the container's temp directory, not the checkout.

AI assistance

Written with Claude Code. Not yet reviewed line by line by a human.

🤖 Generated with Claude Code

The tracing setup looks for the ci-otel helpers through the git
checkout, then through BUILDKITE_BUILD_CHECKOUT_PATH. The CPU test image
has no .git, and the docker plugin passes no checkout path into the
container, so on CPU docker-plugin steps both lookups fail, the job logs
"tracing setup skipped", and main records no per-test timings for them.

Fall back to the image's own copy under /vllm-workspace. Steps where
either earlier lookup works are unchanged, and PR builds render the same
pipeline as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98

Copy link
Copy Markdown
Collaborator Author

Validation builds: tracing and timings on the CPU steps with this branch's generator

Two builds on the tracing treatment branch, each with VLLM_CI_BRANCH=agent/otel-cpu-image-fallback (head 32b7dcd4) and scoped to CPU docker-plugin steps. The treatment branch is used because the timing service accepts uploads only from main and that branch, so these builds show the whole path, not just the setup.

All four jobs passed with no retries.

Step tracing setup skipped on main With this branch Tests per command vs main Per-test spans stored
CPU Tool Parsers printed (main 102522) not printed identical (1 command) 1532
CPU Multimodal + Config printed (main 102522) not printed identical (6 commands) 947
CPU Reasoning + Renderers printed (main 102525) not printed identical (2 commands) 1268
CPU V1 Others printed (main 102525) not printed identical (9 commands) 2978
  • Tests per command compares each pytest summary line (passed, skipped, xfailed, deselected) in the staged job with the same job in the main build named, the newest main run of it before the staging build. Turning tracing on changed nothing about what ran.
  • Spans stored is the number of pytest.test spans the timing service holds for the job. It equals the tests pytest ran in each job. For V1 Others it is 4 below the summary total, because 4 of its skips are module-level and never start a test.
  • The timing service now answers for these steps. /api/timings/latest returns per-file timings from these builds for Tool Parsers (43 files), Multimodal + Config (30) and Reasoning + Renderers (54), with file names in the tests/... form the planner uses. It had nothing for them before. V1 Others has its spans stored too, but the endpoint keeps answering from older main builds for that step until main records new ones, which this change makes possible.

Not covered: the other CPU docker-plugin steps (same image and setup line), and the agent slots after the builds. The helpers write only to the container's temp directory.

@khluu
khluu merged commit bfb48fa into main Nov 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants