Skip to content

[pipeline-generator] AMD steps run on their own definition changes, not on any edit to their yaml - #904

Open
ahsan-ca wants to merge 1 commit into
vllm-project:mainfrom
ahsan-ca:amd-steps-yaml-definition-trigger
Open

ahsan-ca wants to merge 1 commit into
vllm-project:mainfrom
ahsan-ca:amd-steps-yaml-definition-trigger

Conversation

@ahsan-ca

@ahsan-ca ahsan-ca commented Nov 10, 2026 •

Copy link
Copy Markdown

Problem: in a step yaml with a group-level depends_on, the generator adds the file to every step's source_file_dependencies, so any edit to it runs every step it defines.

Change: AMD steps (native and amd- mirrors) read an edit to their own yaml the way #838 does for NVIDIA:

  1. step.py: the loader records each step's source_file, and whether the step lists that file itself (source_file_declared). Neither field is emitted.
  2. step_definitions.py (new): changed_step_keys diffs the base and head versions of a step yaml by key.
    • A changed entry redefines the step, and its amd- mirror if it has one.
    • A change only to mirror.amd redefines just the mirror.
    • The whole file counts, as before, in these cases: anything outside steps changed, a new or unparsable file, or an entry without a unique key.
  3. buildkite_step._step_should_run, AMD steps only:
  4. The base is the commit list_file_diff is taken against: the merge base, or HEAD~1 on main. The bootstrap log prints one line per edited yaml with the keys it redefines.
  5. ci-selector: source_file and source_file_declared are added to KNOWN_STEP_FIELDS. The contract test gets stubs for the two new helpers.

NVIDIA steps are unaffected.

Results: I replayed those 78 builds with each build's base and head yaml.

  • Skipped: 535 of the 551 runs, 100.5 MI355 agent-hours.
    • By queue: amd_mi355_dpx 54.6, amd_mi355_1 12.9, amd_mi355_4 12.9, amd_mi355_8 11.1, amd_mi355_2 9.1.
  • Kept:
    • 7 runs whose own entry the PR changed (0.6 h).
    • 9 runs of steps that list their own yaml (0.6 h).
  • Added: 18 runs of optional AMD steps whose own entry the PR changed (4.0 h).
  • No edit fell back to the whole file.

Step definitions come from vLLM main as of Nov 10. Agent-hours use each step's median MI355 job time over Nov 2–9.

Not changed: ci-selector's offline crosscheck replica (validate/generator_replica.py) still models the whole-file rule for AMD. It already leaves out the generator's other AMD-only rules.

Tests: tests/pipeline_generator/test_step_definitions.py (new) covers:

  • changed, reordered, added, removed and mirror-only entries, entries without an AMD mirror, and a removed mirror;
  • edits outside steps, missing or duplicate keys, unparsable and new files;
  • AMD steps skipping another step's edit, and running on their own edit, on an edit that can't be pinned, and on their other dependencies;
  • optional AMD steps, self-declared yaml, an end-to-end git run, and the log line.

python -m pytest buildkite/tests: 563 passed on Python 3.9 and 3.12. ci_selector suite at VLLM_PIN: 1,393 passed, 4 skipped.

AI assistance (Claude, in Cursor) was used to write this change and its description.

@ahsan-ca
ahsan-ca force-pushed the amd-steps-yaml-definition-trigger branch from 3dd8398 to bacca04 Compare November 10, 2026 21:30
@ahsan-ca

ahsan-ca commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

Tested on vllm/ci with
VLLM_CI_BRANCH=bacca0454108f45bb3b0ae2d55a1a52451516694
VLLM_CI_PLATFORM=amd
on a vLLM branch that edits models_basic.yaml:

  • #101996: edit to the NVIDIA-only step qwen3-5-unit-tests-b200. Log: AMD steps count the edit only if redefined: ['qwen3-5-unit-tests-b200']. No AMD test steps scheduled; all 16 offered under "Run more jobs".
  • #101998 (control, ci-infra main, same commit): 16 AMD test steps scheduled (29 jobs, 4 on amd_mi355_8).
  • #102005: the same edit plus one to kimi-k3-specialized-decode-amd. Only that step scheduled (2 shards on amd_mi355_8); the other 15 offered under "Run more jobs".

…ot on any edit to their yaml

Every step depends on the yaml file that defines it, so an edit to one
step runs every step in the file. ci-infra#838 narrowed such an edit to
the steps whose own definition changed, and runs those even when they are
optional, but it did so in ci-selector and its gate, which AMD steps
bypass. So editing models_basic.yaml still schedules every AMD step in
it, 8-GPU MI355 ones included, and an edit to an optional AMD step still
leaves it untested until the nightly.

The generator now applies the same reading to AMD steps (native and amd-
mirrors):
- An edit to a step's own yaml counts only when it changes the step's own
  entry (its key's fields), or its mirror.amd block for the mirror.
- An optional AMD step whose own entry the edit changes or adds runs.
- An edit outside `steps`, a new or unparsable file, or a step without a
  unique key counts for every non-optional step in the file, as before,
  and runs no optional one.
- A step that lists its own yaml in source_file_dependencies, itself or
  in its mirror.amd block, keeps counting every edit to it.
- The bootstrap log names, per edited yaml, the steps it redefines.

ci-selector models the loader's two new fields, and its contract test
supplies the two new helpers.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Ahsan Saghir <ahsaghir@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@ahsan-ca
ahsan-ca force-pushed the amd-steps-yaml-definition-trigger branch from bacca04 to b60eee4 Compare November 13, 2026 17:28

@AndreasKaratzas AndreasKaratzas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please self review this first. I believe that code can be reduced and written a bit better. For example this changed step keys has all these try catch routines, and why are we not raising any error if there is a yaml error? What is the logic behind the lru cache decorator in the changed step keys in file function? Generally please parse these changes and let me know.

AMD_NATIVE_RUNTIME_SOURCE_DEPENDENCIES
),
"is_amd_gpu_device": lambda d: family_of_device(d) == "amd",
"is_amd_device": lambda d: family_of_device(d) == "amd",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is there this definition here ? it's the exact same as the line above.

)


def _amd_dependencies_match(step: Step, list_file_diff: List[str]) -> bool:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if there are amd functions please move them to amd.py.


A step runs when its `source_file_dependencies` match the change (or on
nightly / `RUN_ALL` builds); optional steps never run on their own. Every
nightly / `RUN_ALL` builds); optional steps never run on their own. In a file

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if all this is needed.

AMD_MIRROR = "amd"


def changed_step_keys(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is amd specific i would prefer that this lives inside amd py instead of a new file.


YAML = ".buildkite/test_areas/models.yaml"

HEADER = """\

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we shorten the new tests (and potentially move them elsewhere since the suggestion is for the new funcs to be moved to amd py)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants