Skip to content

Initialise snd_luind in dist_symbLU to fix hang with ParSymbFact - #231

Merged
xiaoyeli merged 2 commits into
xiaoyeli:masterfrom
jhale:fix-psymbtonum-snd-luind
Oct 22, 2026
Merged

xiaoyeli merged 2 commits into
xiaoyeli:masterfrom
jhale:fix-psymbtonum-snd-luind

Conversation

@jhale

@jhale jhale commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

This PR description was written with the help of an LLM (Claude), and checked by me.

In dist_symbLU (p[sdz]symbfact_distdata.c), snd_luind is never initialised. It is only allocated when the rank has something to send:

if ( nprocs > 1 && (SendCnt_l || SendCnt_u) ) {
  if (!(snd_luind = intMalloc_dist(SUPERLU_MAX(SendCnt_l, SendCnt_u))) ) {

but it is always passed as the send buffer to MPI_Alltoallv. With ParSymbFact = YES on a non-power-of-two number of processes, the processes outside the symbolic factorisation have nothing to send. They pass whatever is on the stack, with all send counts zero.

That is usually harmless, but if the garbage happens to equal MPI_IN_PLACE ((void *) 1 in Open MPI), that process does an in-place all-to-all while the others do not. The job then hangs, or aborts with:

*** An error occurred in MPI_Alltoallv
*** MPI_ERR_TRUNCATE: message truncated

Backtraces from a hung 3-process run show ranks 0 and 1 in ompi_coll_base_alltoallv_intra_pairwise and rank 2 in mca_coll_base_alltoallv_intra_basic_inplace, all called from dist_symbLU via ddist_psymbtonum.

This PR initialises snd_luind to NULL next to rcv_luind, in all three precisions.

How it was found and reproduced

DOLFINx CI started hanging after we made ParSymbFact = YES the default. It hung on 3 processes and passed on 2 and 4. Only builds with 64-bit int_t were affected, presumably because the stack layout is different. It reproduces without DOLFINx. A C driver solving a 7-point Laplacian on a k^3 grid (ColPerm = PARMETIS, ParSymbFact = YES, one pdgssvx call) hangs intermittently on 3 processes. The setup was SuperLU_DIST 9.2.1 +int64+parmetis, ParMETIS 4.0.3 and Open MPI 5.0.10 on Linux x86_64 (Spack). It hung for k = 8, 12 and 16, and aborted with MPI_ERR_TRUNCATE for k = 4. The same binary passed on another runner, as you would expect for an uninitialised read.

Clang points at the same variable:

SRC/double/pdsymbfact_distdata.c:456:6: warning: variable 'snd_luind' may be uninitialized when used here [-Wconditional-uninitialized]
SRC/double/pdsymbfact_distdata.c:117:57: note: initialize the variable 'snd_luind' to silence this warning

With this change the warning is gone in all three files. The C driver above, built against this branch with XSDK_INDEX_SIZE=64 and ParMETIS 4.0.3, passes on 2 to 7 processes for k = 4, 8 and 16. Because the bug depends on stack contents, the passing runs show that nothing regressed, not that the hang is fixed. The fix follows from the code and the backtraces.

DOLFINx issue: FEniCS/dolfinx#4666

A process with nothing to send passed an uninitialised pointer to
MPI_Alltoallv. If it equalled MPI_IN_PLACE, the collective mismatched
and hung, as seen with ParSymbFact on non-power-of-two process counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2
jhale added a commit to FEniCS/dolfinx that referenced this pull request Oct 19, 2026
An uninitialised send buffer in SuperLU_DIST's dist_symbLU makes
ParSymbFact hang on non-power-of-two rank counts, which the int64 CI
legs hit on 3 ranks. Fixed upstream in xiaoyeli/superlu_dist#231.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2
@jhale

jhale commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

This is unrelated to #230. That issue is a separate ParSymbFact bug, a deterministic segfault in symbfact_dist when refactoring with Fact=SamePattern, and this PR doesn't fix it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xiaoyeli
xiaoyeli merged commit e542221 into xiaoyeli:master Oct 22, 2026
15 of 16 checks passed
EMinsight pushed a commit to EMinsight/dolfinx that referenced this pull request Oct 26, 2026
…iCS#4665)

* Default SuperLU_DIST to PARMETIS column ordering

With METIS_AT_PLUS_A every rank orders the global matrix itself and
SuperLU_DIST assumes the orderings agree. Scotch's METIS emulation, used
by the wheels, returns different orderings on different ranks for the
same graph, so parallel factorisations deadlock. PARMETIS ordering is a
single collective call and is also faster on larger 3D problems.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2

* Expose SuperLU_DIST ParSymbFact option

ParSymbFact runs the symbolic factorisation in parallel. With RowPerm
NOROWPERM it also avoids gathering the global matrix on every process.
It needs PARMETIS ordering, so solve rejects other orderings before
entering SuperLU_DIST.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2

* Default SuperLU_DIST ParSymbFact to YES with PARMETIS

Serial symbolic factorisation replicates the global matrix and L/U
structure on every rank, which limits memory scalability. Turn on
parallel symbolic factorisation wherever PARMETIS ordering is the
default.

ParSymbFact needs the ordering step, so setting ColPerm to another
ordering turns it off. SamePattern skips the ordering and crashes in
symbfact_dist, so set_A treats SamePattern as DOFACT under ParSymbFact.
A DOFACT refactorisation in set_A now also releases the solve
communication structure, which a new ordering invalidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2

* Trim SuperLU_DIST option comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2

* Leave ParSymbFact off by default

An uninitialised send buffer in SuperLU_DIST's dist_symbLU makes
ParSymbFact hang on non-power-of-two rank counts, which the int64 CI
legs hit on 3 ranks. Fixed upstream in xiaoyeli/superlu_dist#231.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants