[AMD] Enable AITER MoE for MiniMax-M3 FP4 MI355X vLLM MTP (fix EP startup hang)

[Fangzhou-Ai]

Problem

The MiniMax-M3 FP4 MI355X vLLM MTP sweep (minimaxm3-fp4-mi355x-vllm-mtp) hangs for ~8 h on every expert-parallel config (ep=8, ep=4, dp-attn). See the failed run 28443025904 (PR #1958): all ep>1 spec-mtp jobs ran 8h20m until the CI job timeout; all ep=1 jobs passed.

Root cause (debugged from the failed job log + vLLM source)

With EP enabled and moe_backend=auto, vLLM's MXFP4 MoE selection skips the AITER backend — EP turns on the BatchedExperts activation format, which AITER_MXFP4_MXFP4 doesn't support — and falls back to Mxfp4MoeBackend.EMULATION. The emulation path builds the Quark hw-emulation C++ kernel (quark/torch/kernel/hw_emulation/extensions.py → torch.utils.cpp_extension.load, kernel_ext, Building for 9 ROCm arches gfx90a;gfx942;gfx950;...;gfx1151) at warmup. All EP workers build into the shared /root/.cache/torch_extensions/.../kernel_ext dir simultaneously and deadlock on the build lock, so engine-core never finishes startup:

QUARK-INFO: C++ kernel compilation check start ... First-time compilation may take a few minutes... Building for architectures PYTORCH_ROCM_ARCH='gfx90a;gfx942;gfx950;...'
... INFO shm_broadcast: No available shared memory broadcast block found in 60 seconds ...  (repeats for 8h)

The ep=1 jobs pass because they run --moe-backend aiter (the AITER path, no emulation/build). FP8 MTP EP jobs (28427946510) pass for the same reason — MXFP8 auto-selects the native AITER_MXFP8 backend.

Fix

Mirror the STP recipe minimaxm3_fp4_mi355x_vllm.sh (PR #1954) into the MTP script — export the AITER env knobs and pass --moe-backend aiter unconditionally (including EP). Forcing the AITER backend selects AITER_MXFP4_MXFP4, so the emulation C++ build never happens. Also pin the MTP image to nightly-4559c43a9 (the verified tip), matching the STP recipe.

Difference vs #1958 / #1955

Those PRs disabled AITER MoE under EP (VLLM_ROCM_USE_AITER_MOE=0, no --moe-backend aiter) believing AITER MoE is EP-incompatible. That is exactly what forces the emulation fallback and the build hang. AITER MoE works under EP here — this PR enables it instead of disabling it.

Verification (local, pinned tip 4559c43a9 + aiter tip)

Ran the exact MTP recipe with TP4 + EP4 (--enable-expert-parallel --moe-backend aiter) + EAGLE3 spec decoding:

• MoE backend selected: AITER_MXFP4_MXFP4 (no EMULATION, no kernel_ext build).
• Engine starts and serves; correct output (17×23=391).
• Spec-decode healthy: mean acceptance length ≈ 3.5 (per-position 0.91/0.91/0.73).
• conc=16 burst: 32/32 requests successful, mean TPOT 10.66 ms.
• For reference, tp4 ep1 (no EP) MTP: GSM8K(100)=0.93, perf conc 1/32/128 = 6.10/20.04/87.28 ms.

Caveat: verified against the local vLLM @ 4559c43a9 + aiter tip; the CI re-sweep on the pinned docker image will confirm end-to-end.

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

---

感谢你的贡献！对于 vLLM 与 SGLang，请确保你的 recipe 与官方 vLLM recipes 和/或 SGLang cookbook 保持一致

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

如果不一致，请先创建一个 PR，之后我们才能将你的单节点 PR 合并到 master 分支。让我们确保文档保持一流水准，使整个 ML 社区都能从你的辛勤工作中受益！谢谢

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。如果选择重新运行失败的任务，PR 作者有责任确保其最终通过。参见 GitHub 关于重新运行失败任务的文档：https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

一般而言，PR 作者应先向相应公司的 CODEOWNERS 请求审阅并获得 PR 批准，然后再请求核心维护者审阅。

如需更多帮助，PR 作者可通过 Slack 联系核心维护者。

[claude]

🔴 This PR bumps the minimaxm3-fp4-mi355x-vllm-mtp image tag (nightly-3f5a1e173... → nightly-4559c43a9...) and adds VLLM_ROCM_USE_AITER=1, VLLM_ROCM_USE_AITER_MOE=1, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, and --moe-backend aiter, but does not append the required entry to perf-changelog.yaml. Without that entry the run-sweep workflow has no trigger record for the new config on this PR, so the EP-hang fix that this PR exists to deliver will not be re-benchmarked end-to-end by CI before merge. Mirror the sister STP entry at perf-changelog.yaml lines 4320–4325 (from #1954): a 4–6 line block under config-keys: [minimaxm3-fp4-mi355x-vllm-mtp] describing the image pin and AITER MoE enablement, with pr-link pointing at this PR.

Extended reasoning...

What's missing

AGENTS.md §"Updating Docker images" (line 126) explicitly mandates: "Update the image tag in the relevant .github/configs/*-master.yaml and/or benchmarks/*.sh, update any related env vars / config params, and append a perf-changelog.yaml entry (required - triggers benchmarks)". AGENTS.md line 58 reinforces this: "Changes to perf-changelog.yaml trigger benchmark runs."

This PR makes both of the changelog-triggering changes AGENTS.md calls out — (1) bumps the image tag at .github/configs/amd-master.yaml:2669 from nightly-3f5a1e1733200760169ff31ebe60a271072b199e to nightly-4559c43a9526597c00cbcc4f59979496500268d1, and (2) adds three VLLM_ROCM_USE_AITER* env vars plus --moe-backend aiter to benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm_mtp.sh — but the PR diff modifies only those two files. perf-changelog.yaml is not touched.

Why this matters here (not generic policy concern)

The sister STP PR #1954 — which makes byte-for-byte equivalent changes (same image pin, same three AITER env vars, same --moe-backend aiter) for the non-MTP variant minimaxm3-fp4-mi355x-vllm — did append a proper entry at perf-changelog.yaml lines 4320–4325 with the matching config-keys: [minimaxm3-fp4-mi355x-vllm] block and a pr-link to #1954. The MTP twin (this PR) omits the mirror entry. The most recent existing entry for minimaxm3-fp4-mi355x-vllm-mtp in perf-changelog.yaml (lines 4292–4298, from PR #1939) still pins the old hanging image nightly-3f5a1e173... and describes "automatic MoE backend selection" — exactly the configuration this PR is trying to replace.

CI consequence (step-by-step)

1. .github/workflows/run-sweep.yml (push to main and pull_request triggers) gates on paths: perf-changelog.yaml — only changes to that file cause the full sweep to fire for the affected configs.
2. This PR does not modify perf-changelog.yaml, so on push/PR events the run-sweep workflow has no config-keys delta naming minimaxm3-fp4-mi355x-vllm-mtp to trigger on.
3. Therefore the new AITER+aiter configuration for the MTP recipe will not be re-benchmarked end-to-end on this PR.
4. The PR description itself promises this verification: "verified against the local vLLM @ 4559c43a9 + aiter tip; the CI re-sweep on the pinned docker image will confirm end-to-end." That CI re-sweep cannot happen without the changelog entry.
5. Once merged, future artifact-reuse runs will keep treating the recipe as unchanged from the [codex] add MiniMax M3 FP4 MI355X vLLM MTP benchmark #1939 baseline (hanging image, auto MoE backend) until someone notices the omission.

This is precisely the case run-sweep exists to catch: the entire reason for this PR is to fix an ~8-hour CI hang on EP configs of minimaxm3-fp4-mi355x-vllm-mtp. Merging without the changelog entry leaves the remediation unverified by the gate it was built to satisfy.

Fix

Append an entry to the end of perf-changelog.yaml mirroring the #1954 entry at lines 4320–4325:

- config-keys: [minimaxm3-fp4-mi355x-vllm-mtp]
  description: >-
    Enable AITER MoE on MiniMax-M3 MXFP4 MI355X single-node vLLM MTP:
    export VLLM_ROCM_USE_AITER=1, VLLM_ROCM_USE_AITER_MOE=1, and
    VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1; pass --moe-backend aiter.
    Pin image to nightly-4559c43a9526597c00cbcc4f59979496500268d1 to
    match the STP recipe and fix the EP startup hang.
  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1964

This is the same mechanical 4–6 line append the sister STP PR #1954 already used; copying its shape is the cleanest fix.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28483398517
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28483398517

[Fangzhou-Ai]

Hi @functionstackx @andyluo7 this is our latest minimax m3 fp4 MTP config can we merge this PR?

[functionstackx]

@adibarra

missing pr checklist

[functionstackx]

/reuse-sweep-run 28483398517

[chunfangamd]

@haic0 could you please check whether we've updated the corresponding vllm-project/recipes