Repository navigation
[pipeline-generator] AMD steps run on their own definition changes, not on any edit to their yaml - #904
[pipeline-generator] AMD steps run on their own definition changes, not on any edit to their yaml#904ahsan-ca wants to merge 1 commit into
Conversation
3dd8398 to
bacca04
Compare
|
Tested on vllm/ci with
|
…ot on any edit to their yaml Every step depends on the yaml file that defines it, so an edit to one step runs every step in the file. ci-infra#838 narrowed such an edit to the steps whose own definition changed, and runs those even when they are optional, but it did so in ci-selector and its gate, which AMD steps bypass. So editing models_basic.yaml still schedules every AMD step in it, 8-GPU MI355 ones included, and an edit to an optional AMD step still leaves it untested until the nightly. The generator now applies the same reading to AMD steps (native and amd- mirrors): - An edit to a step's own yaml counts only when it changes the step's own entry (its key's fields), or its mirror.amd block for the mirror. - An optional AMD step whose own entry the edit changes or adds runs. - An edit outside `steps`, a new or unparsable file, or a step without a unique key counts for every non-optional step in the file, as before, and runs no optional one. - A step that lists its own yaml in source_file_dependencies, itself or in its mirror.amd block, keeps counting every edit to it. - The bootstrap log names, per edited yaml, the steps it redefines. ci-selector models the loader's two new fields, and its contract test supplies the two new helpers. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Ahsan Saghir <ahsaghir@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
bacca04 to
b60eee4
Compare
AndreasKaratzas
left a comment
There was a problem hiding this comment.
Please self review this first. I believe that code can be reduced and written a bit better. For example this changed step keys has all these try catch routines, and why are we not raising any error if there is a yaml error? What is the logic behind the lru cache decorator in the changed step keys in file function? Generally please parse these changes and let me know.
| AMD_NATIVE_RUNTIME_SOURCE_DEPENDENCIES | ||
| ), | ||
| "is_amd_gpu_device": lambda d: family_of_device(d) == "amd", | ||
| "is_amd_device": lambda d: family_of_device(d) == "amd", |
There was a problem hiding this comment.
why is there this definition here ? it's the exact same as the line above.
| ) | ||
|
|
||
|
|
||
| def _amd_dependencies_match(step: Step, list_file_diff: List[str]) -> bool: |
There was a problem hiding this comment.
if there are amd functions please move them to amd.py.
|
|
||
| A step runs when its `source_file_dependencies` match the change (or on | ||
| nightly / `RUN_ALL` builds); optional steps never run on their own. Every | ||
| nightly / `RUN_ALL` builds); optional steps never run on their own. In a file |
There was a problem hiding this comment.
not sure if all this is needed.
| AMD_MIRROR = "amd" | ||
|
|
||
|
|
||
| def changed_step_keys( |
There was a problem hiding this comment.
if this is amd specific i would prefer that this lives inside amd py instead of a new file.
|
|
||
| YAML = ".buildkite/test_areas/models.yaml" | ||
|
|
||
| HEADER = """\ |
There was a problem hiding this comment.
can we shorten the new tests (and potentially move them elsewhere since the suggestion is for the new funcs to be moved to amd py)
Problem: in a step yaml with a group-level
depends_on, the generator adds the file to every step'ssource_file_dependencies, so any edit to it runs every step it defines.models_basic.yaml.qwen4-exp-hc-kernels-b200as changed.amd_mi355_8is the scarcest MI355 queue).test_areasyaml. 551 AMD step runs (~102 MI355 agent-hours) had no trigger other than that edit.models_basic.yaml240,kernels.yaml144,misc.yaml75,distributed.yaml40,lm_eval.yaml34.Change: AMD steps (native and
amd-mirrors) read an edit to their own yaml the way #838 does for NVIDIA:step.py: the loader records each step'ssource_file, and whether the step lists that file itself (source_file_declared). Neither field is emitted.step_definitions.py(new):changed_step_keysdiffs the base and head versions of a step yaml bykey.amd-mirror if it has one.mirror.amdredefines just the mirror.stepschanged, a new or unparsable file, or an entry without a uniquekey.buildkite_step._step_should_run, AMD steps only:source_file_dependenciesstill counts every edit to it.list_file_diffis taken against: the merge base, orHEAD~1on main. The bootstrap log prints one line per edited yaml with the keys it redefines.source_fileandsource_file_declaredare added toKNOWN_STEP_FIELDS. The contract test gets stubs for the two new helpers.NVIDIA steps are unaffected.
Results: I replayed those 78 builds with each build's base and head yaml.
amd_mi355_dpx54.6,amd_mi355_112.9,amd_mi355_412.9,amd_mi355_811.1,amd_mi355_29.1.Step definitions come from vLLM main as of Nov 10. Agent-hours use each step's median MI355 job time over Nov 2–9.
Not changed: ci-selector's offline crosscheck replica (
validate/generator_replica.py) still models the whole-file rule for AMD. It already leaves out the generator's other AMD-only rules.Tests:
tests/pipeline_generator/test_step_definitions.py(new) covers:steps, missing or duplicate keys, unparsable and new files;python -m pytest buildkite/tests: 563 passed on Python 3.9 and 3.12. ci_selector suite atVLLM_PIN: 1,393 passed, 4 skipped.AI assistance (Claude, in Cursor) was used to write this change and its description.