Skip to content

[CI] Turn automatic sharding on by default - #64796

Open
Thangnguyenvn98 wants to merge 1 commit into
vllm-project:mainfrom
Thangnguyenvn98:agent/shard-default-on
Open

Thangnguyenvn98 wants to merge 1 commit into
vllm-project:mainfrom
Thangnguyenvn98:agent/shard-default-on

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Trunk change: needs line-by-line review (one line in .buildkite/ci_config.yaml that changes how every step's shards are decided).

Sets automatic_shard_default: true, so the pipeline generator plans shards for every step whose test commands are plain pytest, without a per-step automatic_shard: true. The generator support is vllm-project/ci-infra#885. A step opts out with automatic_shard: false. A step that can't be split (a script, cargo, a fixed hand split) keeps running exactly as it does today.

Main carries 464 automatic_shard: true flags today, so for almost every step this changes nothing. After it merges, a new step shards without anyone adding the flag.

This replaces the earlier content of this PR, which predated the generator rework and marked exclusions step by step; those are no longer needed.

Claims

  • No step is added or removed: 382 in a PR build that runs everything, 549 in a nightly.
  • Every step that already has the flag is planned as before.
  • Six steps that are not flagged today start being planned. Each is already in an open enrollment PR. Since main started recording timings for the CPU steps, V1 Others and Reasoning + Renderers also shard under the switch (2 shards each); they are covered by the build below:
Step Jobs today → with this Runs in Enrollment PR
CPU Multimodal + Config 1 → 2 PR and nightly #67534
Rust Frontend Serve/Admin Coverage 1 → 2 PR and nightly #64128
its AMD mirror 1 → 2 PR and nightly #64128
LM Eval Spec Decode 1 → 3 nightly #64128, #61473
its AMD mirror 1 → 2 nightly #64128
Elastic EP Scaling 1 → 2 nightly #61476
  • Every build gets one collapsed annotation, "N jobs ran as one job without a shard plan", listing those jobs and why. Nothing else changes in what people see.

Validation

  • Rendered main's pipeline with the merged generator, switch off and on, twice each, as a PR build that runs everything and as a nightly. The six steps above are the only ones whose job count the switch changes. Two already-enrolled steps flipped between renders of the same side, because each render reads live timings; they are not counted.
  • The switch itself was staged before the generator change merged ([CI][Do not merge] Staging: automatic sharding on by default #66965, builds 101689 and 101703): sharded jobs ran the same node IDs as main, and jobs that can't be split ran as one job under the annotation.
  • Scoped build on this head (https://buildkite.com/vllm/ci/builds/102675) for the two CPU steps the switch shards that had never run sharded, V1 Others and Reasoning + Renderers: both split into 2 shards with no flag, and their node IDs equal main's (2978 and 1267; 0 missing, added or changed); machine time +4% and -1%. Multimodal + Config was checked the same way on [CI] Enroll 3 CPU steps in automatic sharding #67534 (build 102575). Details in the build note on this PR.
  • Not run on this head: the bot's full build, which is what posts the merge status.

Details

  • Only the switch, the flags stay. Deleting the 464 now-redundant flags is a follow-up with no behavior change; doing it here would conflict with every open enrollment PR.
  • No automatic_shard: false is added. Whether any of the six steps should be held back is a per-step call. If their enrollment PRs merge first, this PR changes no step at all.
  • A new step shards without a validation build. A test that depends on running after another test will then fail on the PR that adds it. The per-build missed-test check (runtime-shard: check on every build that no test dropped out of its shards ci-infra#884) is not merged yet.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

@mergify mergify Bot added the cpu Related to CPU backends label Oct 30, 2026
@mergify

mergify Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Thangnguyenvn98.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 31, 2026
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI][Do not merge yet] Turn automatic sharding on by default [CI] Turn automatic sharding on by default Nov 13, 2026
@mergify mergify Bot removed the needs-rebase label Nov 13, 2026
@Thangnguyenvn98
Thangnguyenvn98 marked this pull request as ready for review November 13, 2026 21:58
@Thangnguyenvn98
Thangnguyenvn98 requested a review from a team November 13, 2026 21:58

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Set `automatic_shard_default: true` in the CI config, so the pipeline
generator plans shards for every step whose test commands are plain
pytest, without a per-step `automatic_shard: true`. A step opts out with
`automatic_shard: false`; a step that can't be split keeps running as
one job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

Build note: #102675 (scoped, head 7430061934)

Scoped to the two CPU steps that the switch would shard and that have never run sharded: v1-others-cpu and cpu-reasoning-renderers. Neither step has automatic_shard: true, so the shards below come from the default alone. All jobs passed, no retries. Node IDs are compared against main #102673, which ran on this PR's base commit.

Job Shards Slowest shard Node IDs vs main Machine time vs main
CPU V1 Others 2 15.5 min (main: one job, 27 to 31 min) 2978 = 2978; 0 missing, 0 added, 0 outcome changed, 0 duplicates +4%
CPU Reasoning + Renderers 2 10.0 min (main: one job, 19 to 21 min) 1267 = 1267; 0 missing, 0 added, 0 outcome changed -1%

Main's single-job times are from its five newest runs of each, so the baseline is typical.

Things a reviewer may stop on:

  • Reasoning + Renderers shows one "duplicate" node ID in the comparison. It is one parametrized test whose ID contains a multi-line template, so two of its cases print the same first line in the log. Main shows the same thing.
  • Reasoning + Renderers' test total sits right at one shard's budget, so its plan can switch between one and two shards from build to build as timings move. Both are covered: one shard is how it runs today, and this build shows two.
  • CPU Multimodal + Config, the third CPU step the switch changes in PR builds, was checked the same way on [CI] Enroll 3 CPU steps in automatic sharding #67534 (scoped build #102575: 947 node IDs equal to main, +6%).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends nvidia rust

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant