Repository navigation
Compute perm_c on process 0 and broadcast, instead of redundant per-process ordering - #233
Merged
Merged
Conversation
…rocess ordering get_perm_c_dist() calls a serial ordering library (MMD, COLAMD, or METIS) independently on every process, with no broadcast or cross-check of the result. This is correct only if the library is guaranteed to return an identical permutation given identical input on every call. Real METIS is deterministic, but Scotch's METIS-compatibility layer (libscotchmetisv5, used when SuperLU_DIST is linked against ParMETIS built on top of Scotch) is not: with threads enabled it can return different orderings on different processes for the same graph. When that happens, perm_c diverges across ranks and the processes reach mismatched collectives later in pdgstrf/pddistribute, deadlocking with no error. Fix: at every pdgssvx/pdgssvx3d/pdgssvx_ABglobal call site (and their single- and complex-precision counterparts) that calls get_perm_c_dist(), compute it only on process 0 and MPI_Bcast the result to the rest of the grid. This makes the result independent of whether the ordering library is deterministic, and saves every other process from redundantly repeating the same serial computation. Fixes #229. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011vhWoBbcvH5n4PgHLzLYev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_perm_c_dist()calls a serial ordering library (MMD, COLAMD, or METIS) independently on every process, with no broadcast or cross-check of the result afterward. Correctness silently depends on that library returning a bit-identical permutation given identical input on every call.libscotchmetisv5): with threads enabled, Scotch documents that its orderings are not guaranteed deterministic across calls. When two processes get differentperm_c, later collectives inpdgstrf/pddistributego mismatched and the factorization deadlocks with no error.get_perm_c_dist()call site in the 2D/3D/ABglobal drivers (pdgssvx,pdgssvx3d,pdgssvx_ABglobal, and their single- and complex-precision counterparts, including the_d2variants), compute the ordering only on process 0 andMPI_Bcastit to the rest of the grid — mirroring the existingMPI_Bcast(perm_r, m, MPI_INT, 0, grid->comm)pattern already used for MC64 row permutations in the same files. This makes the result independent of the ordering library's determinism, and saves every other process from redundantly repeating the same serial computation.Fixes #229.
Test plan
libsuperlu_dist.a) with-DXSDK_INDEX_SIZE=64across all three precisions (single/double/complex16) — zero errors, zero warnings on any of the 11 touched files.Generated by Claude Code