[CICD] Unify dependency-coupled CI across accelerator platforms by AlexMa616 · Pull Request #1285 · flagos-ai/FlagScale

AlexMa616 · 2026-09-08T04:54:20Z

Summary

Unify Megatron-LM-FL and TransformerEngine-FL dependency management and test execution across CUDA, Ascend, Enflame, Hygon, Kunlunxin, MetaX, and MUSA platforms.

Changes

Add platform-specific runtime configuration and environment setup.
Add reusable installation and validation scripts for:
- Megatron-LM-FL
- TransformerEngine-FL
- Training dependencies
Add a dedicated dependency preparation workflow with cache and artifact handling.
Isolate runtime, tokenizer, and prepared dependency paths between CI jobs.
Prevent stale runtime wheels and installation artifacts from being reused.
Improve import-path compatibility between FlagScale and Megatron-LM-FL.
Add platform-aware CUDA checks and device handling.
Fix platform-specific training and heterogeneous-training compatibility issues.
Add benchmark configurations and golden baselines for all supported platforms.
Update unit, functional, heterogeneous-training, and benchmark test configurations.
Exclude CUDA-only KERV tests on unsupported accelerator platforms.
Add KERV runtime operators and related tests.
Make benchmark log parsing tolerant of non-UTF-8 vendor runtime output.
Update unit-test process counts and timeout configuration.

Validation

Updated platform-specific benchmark baselines using completed hardware runs.
Verified benchmark logs can be parsed for vendor runtimes that emit non-UTF-8 bytes.
Verified CI configuration formatting and whitespace checks locally.

Notes

This change keeps platform-specific behavior in platform configuration, environment setup, and dedicated workflow layers while preserving the common test runner interface.

- Workaround for Python < 3.12: temporarily patch pyproject.toml - Fix verification logic to properly check megatron/__init__.py - Add detailed logging for debugging

…ogging

…lows - Add cache restore and runtime installation steps to unit_tests_common.yml - Pass dependency cache keys and configs from all_tests_common.yml - Implement safe_get_world_size() locally for Megatron API compatibility - Reorganize all_tests.yml to group prepare+test jobs per platform This fixes the issue where prepared dependencies were not being used, causing tests to import old system-installed Megatron packages instead of the cached versions.

The prepare_dependencies workflow saves cache to relative paths (megatron-lm-fl-install, te-fl-wheel) but unit_tests was trying to restore from absolute paths (/opt/flagscale/deps/*), causing cache misses even when cache exists. Changed unit_tests_common.yml to: - Restore cache to the same relative paths as prepare workflow - Use GITHUB_WORKSPACE-based paths for runtime installation - Add existence checks before running install scripts

install_te_fl_runtime.sh expects TE_FL_WHEEL_DIR environment variable, not TE_FL_INSTALL_DIR.

- kunlunxin: use typing_extensions.override for Python 3.10 compatibility - musa: add onnxscript to TE-FL runtime dependencies - enflame: add onnxscript + patch flash_attn detection for torch_gcu

…t version

…nt conflicts - **Enflame**: Remove image pre-installed TE-FL before tests to prevent flash_attn import during pytest collection - **MUSA**: Remove image pre-installed TE-FL to prevent flash_attn import during Megatron-LM-FL installation verification - **MetaX**: Remove both megatron-core and TE-FL from image, plus PYTHONPATH cleanup for /opt/flagscale/deps and /workspace paths Root cause: All platforms have pre-installed old versions of dependencies in their images that take precedence over CI-built latest versions, causing import errors at different stages (build/test collection/runtime). Solution: Explicitly uninstall image pre-installed packages in setup scripts to ensure only CI-built versions are used.

…upport Add megatron-energon[av_decode]~=7.0 to runtime pip packages for all platforms: - Kunlunxin P800 - MUSA - Enflame (燧原) - MetaX Required by: - flagscale/train/megatron/train_qwen35.py - tests/unit_tests/train/megatron/test_qwen35_tokenizer.py - Future multimodal training functionality This resolves the ModuleNotFoundError for megatron.energon module.

Enflame image contains both old megatron-core and transformer-engine that interfere with CI-built versions, causing: - ImportError: cannot import name 'core' from 'megatron' - ModuleNotFoundError: No module named 'flash_attn_2_cuda' Now uninstall both packages, matching MetaX strategy.

Apply consistent dependency management to all platforms: Config changes (add megatron-energon): - CUDA: add megatron-energon[av_decode]~=7.0 - Ascend: add megatron-energon[av_decode]~=7.0 - Hygon: add megatron-energon[av_decode]~=7.0 Setup script changes (uninstall pre-installed deps): - Kunlunxin: uninstall megatron-core + transformer-engine - CUDA: uninstall megatron-core + transformer-engine - Ascend: uninstall megatron-core + transformer-engine - Hygon: uninstall megatron-core + transformer-engine This ensures: 1. All platforms have megatron-energon for qwen35 and multimodal support 2. All platforms uninstall pre-installed deps to avoid version conflicts 3. Consistent behavior across the entire CI pipeline

…brackets YAML flow-style arrays cannot contain unescaped square brackets. The package name megatron-energon[av_decode]~=7.0 was causing parser errors. Changed from: pip_packages: [flag-gems, sqlalchemy, megatron-energon[av_decode]~=7.0] To: pip_packages: - flag-gems - sqlalchemy - "megatron-energon[av_decode]~=7.0" Fixes 3 unit test failures: - test_platform_source_refs_use_catalog[metax] - test_declared_all_images_use_one_python_environment - test_inference_images_include_serve_dependencies[metax]

Apply same YAML syntax fix as MetaX to avoid parser errors with square brackets in package names. Changed from: pip_packages: [megatron-energon[av_decode]~=7.0] To: pip_packages: - "megatron-energon[av_decode]~=7.0" Affected files: - .github/configs/cuda.yml (line 176) - .github/configs/ascend.yml (line 143) - .github/configs/hygon.yml (line 146)

- Add compatibility_patches.py to mock flash_attn_2_cuda for non-CUDA platforms - Apply patches in train_gpt.py before any heavy imports - Uninstall site-packages megatron-core before installing from PYTHONPATH - Install TE-FL with --no-deps to prevent pip from downgrading dependencies - Set PYTHONNOUSERSITE=1 to ensure PYTHONPATH priority - Add verification for cuda_graph_config module availability Fixes import errors on Enflame, MUSA, MetaX, and KunLunXin platforms. Prevents Megatron-Core downgrade from 0.18.2 to 0.17.1.

- Add noqa: F401 for flash_attn_2_cuda import check - Apply ruff format for code style consistency

- Move patch application to tests/conftest.py - Ensures flash_attn_2_cuda mock is active before any test module imports - Fixes ModuleNotFoundError in unit tests for Enflame, MUSA, MetaX, KunLunXin Previously patches were only applied in train_gpt.py, but unit tests import megatron modules directly without going through that entry point.

…ority - Place megatron-lm-fl-install at the front of PYTHONPATH - Prevents namespace conflict with flagscale/train/megatron/ - Set PYTHONNOUSERSITE=1 to block user site-packages interference Root cause: Python was finding flagscale/train/megatron/ first when importing 'megatron', which has no 'core' submodule, causing 'ImportError: cannot import name core from megatron'. Fixes: Enflame, KunLunXin ModuleNotFoundError for megatron.core.*

- Place megatron-lm-fl-install at front of PYTHONPATH - Set PYTHONNOUSERSITE=1 to block user site-packages - Mirrors the fix already applied to unit tests Without this, functional tests still have PROJECT_ROOT first, causing the same namespace conflict where flagscale/train/megatron/ shadows the real megatron-core package. Ensures ALL test types (unit, functional, benchmark, hetero_train, etc.) use consistent PYTHONPATH ordering.

CLAassistant · 2026-09-13T13:04:38Z

All committers have signed the CLA.

- Rename 18 files (6 platforms × 3 types) to include backend type - Format: {platform}0_6b_tp1_pp1-{backend}.{yaml|json} - Backend types: • reference: ascend0, enflame0, musa0 • flagos: hygon0, metax0 • vendor: kunlunxin0 - Update internal references (train/exp_name/exp_dir) to match new names This makes the TE backend preference explicit in filenames for better clarity.

Update case names in platform configs to include TE backend suffix: - reference: ascend0, enflame0, musa0 - flagos: hygon0, metax0 - vendor: kunlunxin0 This fixes the 'Config not found' error after benchmark file renaming.

- MetaX runner image does not have FlagGems installed - Using flagos backend requires FlagGems dependency - Switch to reference backend to align with other platforms (Ascend, MUSA)

Changes: - Rename metax0_6b_tp1_pp1-flagos.yaml → metax0_6b_tp1_pp1-reference.yaml - Update te_fl_prefer: flagos → reference - Update platform config reference in metax.yaml - Align with other platforms (Ascend, MUSA) using reference backend Reason: - MetaX runner image does not have FlagGems installed - flagos backend requires FlagGems dependency - reference backend works without additional dependencies

- Rename metax0_6b_tp1_pp1-flagos.json → metax0_6b_tp1_pp1-reference.json - Align with config file rename from flagos to reference backend

…tforms [Fix] Ascend get_device_arch_version() return None

AlexMa616 added 8 commits September 8, 2026 12:48

feat: add Megatron-LM-FL and TE-FL dependency coupling for all platforms

c068f26

fix(cicd): add missing test_matrix.unit.nproc_per_node for all platforms

b382b50

fix(cicd): improve Megatron-LM-FL installation verification logic

f4a1820

fix(cicd): remove extra newlines at end of config files

4941a74

fix(cicd): add Python version workaround and improve verification logic

a5a173e

- Workaround for Python < 3.12: temporarily patch pyproject.toml - Fix verification logic to properly check megatron/__init__.py - Add detailed logging for debugging

fix(cicd): improve Megatron installation verification with detailed l…

5f5e820

…ogging

fix(cicd): verify megatron.core package installation correctly

30614c4

AlexMa616 requested review from aoyulong, heavyrain-lzy and zhaoyinglia as code owners September 8, 2026 05:38

AlexMa616 added 19 commits September 8, 2026 13:46

fix: use TE_FL_WHEEL_DIR instead of TE_FL_INSTALL_DIR

a17653b

install_te_fl_runtime.sh expects TE_FL_WHEEL_DIR environment variable, not TE_FL_INSTALL_DIR.

fix(ci): resolve MUSA/Kunlunxin/Enflame platform test failures

851d695

- kunlunxin: use typing_extensions.override for Python 3.10 compatibility - musa: add onnxscript to TE-FL runtime dependencies - enflame: add onnxscript + patch flash_attn detection for torch_gcu

fix(musa): patch TE-FL flash_attn detection for torch_musa compatibility

53e9322

fix(metax): remove pre-installed Megatron paths to use CI-built lates…

a2546de

…t version

fix(musa/enflame): add NVTE env vars to disable flash_attn at build time

5ee5ba8

style: fix ruff linting errors in compatibility_patches.py

d1c13ec

- Add noqa: F401 for flash_attn_2_cuda import check - Apply ruff format for code style consistency

style: fix import order in tests/conftest.py

c395bff

fix(ci): unify prepared training runtime dependencies

b3971b0

AlexMa616 added 9 commits September 12, 2026 08:53

fix(tests): exclude cuda-only KERV tests on non-cuda platforms

eb16123

fix(kunlunxin): disable incompatible fused swiglu in test

218fb76

fix(kunlunxin): pass swiglu fusion disable flag

54d8dde

ci: extend unit test timeout

b55dab8

ci: use two processes for unit tests

9c6dd43

ci: keep eight processes for cuda unit tests

f56b467

ci: add benchmark coverage for all platforms

e949132

ci: normalize platform benchmark configs

516203f

ci: fix platform benchmark and train names

59f4936

AlexMa616 added 3 commits September 13, 2026 21:09

ci: avoid duplicate unit job prefix

0049b56

test: update platform benchmark baselines

2f63a6b

ci: tolerate non-utf8 benchmark logs

b3e2066

AlexMa616 changed the title ~~[CICD] Add Megatron-LM-FL and TE-FL dependency coupling~~ [CICD] Unify dependency-coupled CI across accelerator platforms Sep 14, 2026

AlexMa616 added 5 commits September 14, 2026 09:48

test: update MUSA benchmark baseline

63fd103

ci: clarify platform TE-FL preferences

947d0c8

ci: centralize Hugging Face module cache isolation

8fb31f6

fix: serialize Hugging Face dynamic module loading

3f902a5

fix: isolate Hugging Face module cache in CI

b136034

AlexMa616 force-pushed the feat/dependency-coupling-all-platforms branch from 722b969 to b136034 Compare September 14, 2026 04:28

AlexMa616 and others added 7 commits September 14, 2026 15:31

fix: update platform configs to match renamed benchmark cases

2c2e287

Update case names in platform configs to include TE backend suffix: - reference: ascend0, enflame0, musa0 - flagos: hygon0, metax0 - vendor: kunlunxin0 This fixes the 'Config not found' error after benchmark file renaming.

fix: change MetaX te_fl_prefer from flagos to reference

98fd10a

- MetaX runner image does not have FlagGems installed - Using flagos backend requires FlagGems dependency - Switch to reference backend to align with other platforms (Ascend, MUSA)

fix: rename MetaX gold baseline file to match config rename

92dac27

- Rename metax0_6b_tp1_pp1-flagos.json → metax0_6b_tp1_pp1-reference.json - Align with config file rename from flagos to reference backend

[Fix] Ascend get_device_arch_version() return None

a3088a5

Merge pull request #9 from Darryl233/feat/dependency-coupling-all-pla…

494e828

…tforms [Fix] Ascend get_device_arch_version() return None

zhaoyinglia approved these changes Sep 15, 2026

View reviewed changes

xmhubj approved these changes Sep 15, 2026

View reviewed changes

xmhubj merged commit 91fa035 into flagos-ai:main Sep 15, 2026
91 checks passed

Navigation Menu

[CICD] Unify dependency-coupled CI across accelerator platforms - #1285

[CICD] Unify dependency-coupled CI across accelerator platforms#1285
xmhubj merged 95 commits into
flagos-ai:mainfrom
BrianPei:feat/dependency-coupling-all-platforms

AlexMa616 commented Sep 8, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Sep 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

AlexMa616 commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Notes

Uh oh!

CLAassistant commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AlexMa616 commented Sep 8, 2026 •

edited

Loading

CLAassistant commented Sep 13, 2026 •

edited

Loading