Skip to content

Compute perm_c on process 0 and broadcast, instead of redundant per-process ordering - #233

Merged
xiaoyeli merged 1 commit into
maintfrom
claude/fix-perm-c-broadcast-issue-229
Oct 22, 2026
Merged

xiaoyeli merged 1 commit into
maintfrom
claude/fix-perm-c-broadcast-issue-229

Conversation

@xiaoyeli

Copy link
Copy Markdown
Owner

Summary

  • get_perm_c_dist() calls a serial ordering library (MMD, COLAMD, or METIS) independently on every process, with no broadcast or cross-check of the result afterward. Correctness silently depends on that library returning a bit-identical permutation given identical input on every call.
  • Real METIS is deterministic, so this has worked historically. It breaks when SuperLU_DIST is linked against ParMETIS built on top of Scotch's METIS-compatibility layer (libscotchmetisv5): with threads enabled, Scotch documents that its orderings are not guaranteed deterministic across calls. When two processes get different perm_c, later collectives in pdgstrf/pddistribute go mismatched and the factorization deadlocks with no error.
  • Fix: at every get_perm_c_dist() call site in the 2D/3D/ABglobal drivers (pdgssvx, pdgssvx3d, pdgssvx_ABglobal, and their single- and complex-precision counterparts, including the _d2 variants), compute the ordering only on process 0 and MPI_Bcast it to the rest of the grid — mirroring the existing MPI_Bcast(perm_r, m, MPI_INT, 0, grid->comm) pattern already used for MC64 row permutations in the same files. This makes the result independent of the ordering library's determinism, and saves every other process from redundantly repeating the same serial computation.

Fixes #229.

Test plan

  • Built the full static library (libsuperlu_dist.a) with -DXSDK_INDEX_SIZE=64 across all three precisions (single/double/complex16) — zero errors, zero warnings on any of the 11 touched files.
  • Would appreciate a maintainer re-run of the reporter's Scotch-based reproducer from Deadlock when METIS_NodeND returns different orderings on different processes (METIS_AT_PLUS_A) #229 to confirm the deadlock no longer occurs, since reproducing that specific environment (ParMETIS built against Scotch with threads enabled) wasn't available here.

Generated by Claude Code

…rocess ordering

get_perm_c_dist() calls a serial ordering library (MMD, COLAMD, or METIS)
independently on every process, with no broadcast or cross-check of the
result. This is correct only if the library is guaranteed to return an
identical permutation given identical input on every call. Real METIS is
deterministic, but Scotch's METIS-compatibility layer (libscotchmetisv5,
used when SuperLU_DIST is linked against ParMETIS built on top of Scotch)
is not: with threads enabled it can return different orderings on
different processes for the same graph. When that happens, perm_c
diverges across ranks and the processes reach mismatched collectives
later in pdgstrf/pddistribute, deadlocking with no error.

Fix: at every pdgssvx/pdgssvx3d/pdgssvx_ABglobal call site (and their
single- and complex-precision counterparts) that calls get_perm_c_dist(),
compute it only on process 0 and MPI_Bcast the result to the rest of the
grid. This makes the result independent of whether the ordering library
is deterministic, and saves every other process from redundantly
repeating the same serial computation.

Fixes #229.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011vhWoBbcvH5n4PgHLzLYev
@xiaoyeli
xiaoyeli merged commit 066ecea into maint Oct 22, 2026
29 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants