TL/UCP: select exact allgather radix schedules - #1328
Conversation
|
/build |
|
@ci-triage-agent |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Build stage "Setup Image x86_64/ngc_pytorch/0" failed; Root cause: Implicated commit: File: Suggested fix: Replace
Then rebuild locally with
|
2d72f80 to
508e8d4
Compare
|
🤖 CI Triage Agent — TL;DR: The codestyle lint failed because commit Full analysisSummary: The "Lint (codestyle)" GitHub Actions job failed on its commit-title check — one commit title exceeds the 50-character maximum. Root cause: The commit-title validator in Implicated commit: The offending commit is the one titled File: Suggested fix: Reword that commit's title to ≤50 characters, e.g. Related: none
|
81f4268 to
afa8d2f
Compare
afa8d2f to
6e8663d
Compare
|
Maintainer note: this stacked PR inherits the intentional minimal Any clang-format/codestyle complaint about the surrounding legacy configuration |
058defb to
0603a54
Compare
|
🤖 CI Triage Agent — TL;DR: The codestyle "commit title" check failed because one commit title in PR #1328 exceeds the 50-character limit (it's 57 chars). Shorten that commit's title to ≤50 characters and force-push. Full analysisSummary: The Root cause: The commit-title check enforces Implicated commit: The commit in PR #1328 (branch File: Suggested fix: Reword the offending commit's title to ≤50 characters while keeping a valid header prefix (
Use an interactive rebase to edit that commit, then force-push: All other titles passed, so no further changes are needed. Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: The "Run set -eE" commit-title check step in the codestyle workflow exited 1 because a PR commit title is longer than the allowed 50 characters. Root cause: The linter's Implicated commit: The PR commit titled File: Suggested fix: Reword and amend the offending commit title to ≤50 characters, e.g. Related: PR #1328 ( |
|
🤖 CI Triage Agent — TL;DR: The codestyle lint job failed its commit-title-length check because one commit in PR #1328 has a 57-character title (limit is 50). Shorten that commit's title to ≤50 characters. Full analysisSummary: The Root cause: The lint script rejects non-merge commit titles longer than 50 characters. The commit titled Implicated commit: The PR commit titled File: Suggested fix: Reword the offending commit title to ≤50 characters, e.g. Related: none |
|
🤖 CI Triage Agent — TL;DR: The codestyle lint failed because one commit title in PR #1328 exceeds the 50-character limit; shorten Full analysisSummary: The Root cause: The check-title script in Implicated commit: The offending commit is the one titled File: Suggested fix: Reword that commit's subject to ≤50 characters and force-push the rebased branch. For example:
Use an interactive rebase ( Related: none
|
|
🤖 CI Triage Agent — TL;DR: The codestyle job failed its commit-title lint because one commit in PR #1328 has a 57-character title, exceeding the 50-character limit; shorten that commit's title to ≤50 chars. Full analysisSummary: The "Lint (codestyle)" GitHub Actions job failed at the commit-title check step with exit code 1. Root cause: The commit-title linter in Implicated commit: The commit with title File: Suggested fix: Reword that commit's title to 50 characters or fewer while keeping a valid
Apply via interactive rebase ( Related: none
|
|
🤖 CI Triage Agent — TL;DR: The codestyle job's commit-title linter failed because one commit title in PR #1328 exceeds the 50-character limit (57 chars); shorten that commit's subject line to ≤50 characters. Full analysisSummary: The Root cause: The check script in Implicated commit: The offending commit is the one titled File: Suggested fix: Reword that commit's title to ≤50 characters and force-push. For example, interactively rebase ( Related: none |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: GitHub Actions Root cause: The codestyle workflow's Implicated commit: The offending commit on branch File: Suggested fix: Reword the offending commit title to 50 characters or fewer while keeping a valid header prefix. For example, Related: PR #1328 (openucx/ucc); no related issues found.
|
4f2877c to
43a2ecf
Compare
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: The "Lint (codestyle)" GitHub Actions job failed at the commit-title check step with exit code 1. Root cause: The commit-title linter in Implicated commit: The offending commit is the one titled File: Suggested fix: Reword that commit title to ≤ 50 characters (keeping a valid
Related: none |
|
🤖 CI Triage Agent — TL;DR: The codestyle "Lint" job failed because one commit title in PR #1328 exceeds the 50-character limit; shorten Full analysisSummary: The Root cause: The workflow's Implicated commit: The offending commit in PR #1328 (branch File: Suggested fix: Reword the commit title to ≤50 characters and force-push. For example: Related: none |
|
🤖 CI Triage Agent — TL;DR: The codestyle job failed because one commit title in PR #1328 is 57 characters — over the 50-char limit — and the fix is to reword/amend that commit title to ≤50 characters. Full analysisSummary: The Root cause: The workflow's Implicated commit: The offending commit in PR #1328 with title File: Suggested fix: Reword the offending commit title to ≤50 characters and force-push. For example:
Since this is mid-history in a large series, rebase interactively and All other 36 titles passed, so only that one needs changing. Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The codestyle job's commit-title linter failed because one commit title in PR #1328 exceeds the 50-character limit; shorten the offending commit title to ≤50 characters. Full analysisSummary: The Root cause: The CI script in Implicated commit: The commit in PR #1328 with title File: Suggested fix: Reword that single commit title to 50 characters or fewer and force-push. For example: Related: none |
|
🤖 CI Triage Agent — TL;DR: The codestyle "commit title" check failed because one commit in PR #1328 has a 57-character title, exceeding the 50-char limit; shorten that commit's subject line to ≤50 chars and force-push. Full analysisSummary: The Root cause: The check enforces a maximum commit-title length of 50 characters ( Implicated commit: The offending commit is the one with subject File: Suggested fix: Rewrite the overlong commit title to ≤50 characters, e.g. No source-code change is required; this is purely a commit-message policy violation. Related: none found.
|
|
🤖 CI Triage Agent — TL;DR: The Lint (codestyle) job failed because one commit title in PR #1328 exceeds the 50-character limit; shorten the offending commit message and the check will pass. Full analysisSummary: The commit-title lint step ( Root cause: The codestyle check iterates over every commit in the PR range and enforces Implicated commit: The commit whose title is File: Suggested fix: Reword that commit's title to ≤50 characters, e.g. Related: none found in logs. Note: the log contains a redacted git basic-auth header ( |
31eeeeb to
2b73955
Compare
Add explicit per-phase radix schedules to TL/UCP K-nomial AllGather while preserving the fixed-radix and AllGatherV paths. Parse and validate exact schedules through the existing team configuration.
Cover schedule parsing, legacy fixed-radix behavior, and exact mixed-radix peer and segment layouts.
2b73955 to
469d39e
Compare
Add opt-in automatic exact-schedule selection. Small messages minimize phase count and total fanout, with powers of two prioritized in the generated order. Large messages retain an R2-oriented factorization, and explicit schedules remain authoritative.
Cover minimum-phase fallback, fanout and order selection, radix 9, fixed-radix normalization, the large-message boundary, odd factors, and unsupported teams.
9c1d899 to
ecf55f9
Compare
What
Add opt-in, message-size-aware exact-schedule selection for TL/UCP K-nomial
AllGather. This PR is stacked on the mixed-radix execution capability in #1327.
UCC_TL_UCP_ALLGATHER_KN_MIXED_RADICEShas three states:auto: select an exact fixed or mixed schedule;8,6: use that authoritative exact schedule.Policy
Automatic mode uses the total AllGather receive payload:
number of phases, then minimize total fanout
sum(radix - 1);Ties use a deterministic priority order: powers of two first (
8,4,2), thenthe remaining radices in descending order. Uniform selected schedules are
normalized to the existing fixed-radix path. If no supported exact
factorization exists, selection falls back to the legacy path.
Examples:
4,5,38,94,4,68,4,42,2,2,2,2,3Performance evidence
The legacy topology heuristic selects fixed R4 at 64 ranks and does not use
message size. The figures compare that measured execution path with current
auto and fixed R2/R4/R8. All values are out-of-place AllGather bus bandwidth.
Gaia: clear message-size crossover
At 64 ranks, current auto is execution-equivalent to measured explicit R8
below 1 GiB and explicit R2 from 1 GiB. It improves on legacy auto/R4 by
15.91% and 16.74% geometrically, respectively.
Doris: small-message gain and large-message convergence
A fresh 64-rank position-balanced run measured actual auto and explicit
R2/R4/R8 four times per size with 50 warmups and 100 timed iterations. Below
1 GiB, auto tracks R8 and improves on legacy auto/R4 by 5.60%
geometrically, with a 0.14% mean / 1.04% worst gap to the measured
candidate winner. At 1--8 GiB, R2/R4/R8 converge: auto is 0.10% above
legacy R4 and within 0.10% mean / 0.23% worst of the candidates. This is
non-regression evidence on Doris, not a meaningful R2 speedup.
Additional observations:
on Baby Lyris they reached about 96.4 GB/s at 8 GiB, versus 46--49 GB/s for
fixed R2/R4/R8.
the exhaustive 72-rank Baby Lyris sweep ranked every permutation of the
balanced
3,4,6factors above the more uneven three-phase factorizations.At 32 ranks on Gaia, the balanced
2,4,4family similarly beat2,2,8.no portable largest-first, largest-last, or PPN-first rule. The selector
therefore uses a topology-independent cost and a stable ordering.
best schedule remain runtime- and platform-dependent. Explicit schedules
remain available when a deployment has stronger local evidence.
Validation
R8,
8,6,8,9, balanced4,4,6and8,4,4, the large-message R2schedule, odd factors, and unsupported exact shapes.
install/config validation, 9/9 focused tests, and CUDA AllGather checks on
Gaia. Exact-head CI for the updated cost rule is pending.