Repository navigation
Initialise snd_luind in dist_symbLU to fix hang with ParSymbFact - #231
Merged
Merged
Conversation
A process with nothing to send passed an uninitialised pointer to MPI_Alltoallv. If it equalled MPI_IN_PLACE, the collective mismatched and hung, as seen with ParSymbFact on non-power-of-two process counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2
jhale
added a commit
to FEniCS/dolfinx
that referenced
this pull request
Oct 19, 2026
An uninitialised send buffer in SuperLU_DIST's dist_symbLU makes ParSymbFact hang on non-power-of-two rank counts, which the int64 CI legs hit on 3 ranks. Fixed upstream in xiaoyeli/superlu_dist#231. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2
Contributor
Author
|
This is unrelated to #230. That issue is a separate ParSymbFact bug, a deterministic segfault in |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EMinsight
pushed a commit
to EMinsight/dolfinx
that referenced
this pull request
Oct 26, 2026
…iCS#4665) * Default SuperLU_DIST to PARMETIS column ordering With METIS_AT_PLUS_A every rank orders the global matrix itself and SuperLU_DIST assumes the orderings agree. Scotch's METIS emulation, used by the wheels, returns different orderings on different ranks for the same graph, so parallel factorisations deadlock. PARMETIS ordering is a single collective call and is also faster on larger 3D problems. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2 * Expose SuperLU_DIST ParSymbFact option ParSymbFact runs the symbolic factorisation in parallel. With RowPerm NOROWPERM it also avoids gathering the global matrix on every process. It needs PARMETIS ordering, so solve rejects other orderings before entering SuperLU_DIST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2 * Default SuperLU_DIST ParSymbFact to YES with PARMETIS Serial symbolic factorisation replicates the global matrix and L/U structure on every rank, which limits memory scalability. Turn on parallel symbolic factorisation wherever PARMETIS ordering is the default. ParSymbFact needs the ordering step, so setting ColPerm to another ordering turns it off. SamePattern skips the ordering and crashes in symbfact_dist, so set_A treats SamePattern as DOFACT under ParSymbFact. A DOFACT refactorisation in set_A now also releases the solve communication structure, which a new ordering invalidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2 * Trim SuperLU_DIST option comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2 * Leave ParSymbFact off by default An uninitialised send buffer in SuperLU_DIST's dist_symbLU makes ParSymbFact hang on non-power-of-two rank counts, which the int64 CI legs hit on 3 ranks. Fixed upstream in xiaoyeli/superlu_dist#231. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 26, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR description was written with the help of an LLM (Claude), and checked by me.
In
dist_symbLU(p[sdz]symbfact_distdata.c),snd_luindis never initialised. It is only allocated when the rank has something to send:but it is always passed as the send buffer to
MPI_Alltoallv. WithParSymbFact = YESon a non-power-of-two number of processes, the processes outside the symbolic factorisation have nothing to send. They pass whatever is on the stack, with all send counts zero.That is usually harmless, but if the garbage happens to equal
MPI_IN_PLACE((void *) 1in Open MPI), that process does an in-place all-to-all while the others do not. The job then hangs, or aborts with:Backtraces from a hung 3-process run show ranks 0 and 1 in
ompi_coll_base_alltoallv_intra_pairwiseand rank 2 inmca_coll_base_alltoallv_intra_basic_inplace, all called fromdist_symbLUviaddist_psymbtonum.This PR initialises
snd_luindtoNULLnext torcv_luind, in all three precisions.How it was found and reproduced
DOLFINx CI started hanging after we made
ParSymbFact = YESthe default. It hung on 3 processes and passed on 2 and 4. Only builds with 64-bitint_twere affected, presumably because the stack layout is different. It reproduces without DOLFINx. A C driver solving a 7-point Laplacian on a k^3 grid (ColPerm = PARMETIS,ParSymbFact = YES, onepdgssvxcall) hangs intermittently on 3 processes. The setup was SuperLU_DIST 9.2.1+int64+parmetis, ParMETIS 4.0.3 and Open MPI 5.0.10 on Linux x86_64 (Spack). It hung for k = 8, 12 and 16, and aborted withMPI_ERR_TRUNCATEfor k = 4. The same binary passed on another runner, as you would expect for an uninitialised read.Clang points at the same variable:
With this change the warning is gone in all three files. The C driver above, built against this branch with
XSDK_INDEX_SIZE=64and ParMETIS 4.0.3, passes on 2 to 7 processes for k = 4, 8 and 16. Because the bug depends on stack contents, the passing runs show that nothing regressed, not that the hang is fixed. The fix follows from the code and the backtraces.DOLFINx issue: FEniCS/dolfinx#4666