[jhale]

When SuperLU_DIST has ParMETIS support (HAVE_PARMETIS, which is set for both real ParMETIS and Scotch's emulation), this PR makes ColPerm=PARMETIS the default in place of METIS_AT_PLUS_A. Builds without ParMETIS keep SuperLU_DIST's own default (MMD_AT_PLUS_A). It also exposes ParSymbFact through set_option, off by default.

Deadlock with the default ordering

With ColPerm=METIS_AT_PLUS_A, every rank computes the column ordering of the gathered global matrix independently (get_perm_c_dist in pdgssvx), with no broadcast, so all ranks must get the same permutation. The wheels (#4652) build SuperLU_DIST against Scotch's METIS/ParMETIS emulation. Its METIS_NodeND returns different permutations on different ranks for identical input. I checked this by interposing it in a 2-rank solve: the ranks had the same input hash and got different outputs. Resetting Scotch's RNG and running single-threaded didn't make the orderings consistent. The ranks then reach mismatched collectives and the factorisation deadlocks. The existing tests use 5×5 meshes, which happen to order consistently, so CI didn't catch this.

PARMETIS is a single collective call, so the ranks agree by construction. It's also faster on larger 3D problems. Solve times on the wheels, 4 ranks, Poisson on a unit cube:

| Problem | DOFs | MMD_AT_PLUS_A | PARMETIS | |---|---|---|---| | P1 | 69k | 3.6 s | 1.6 s | | P1 | 185k | 22.1 s | 6.6 s | | P2 | 118k | 10.0 s | 3.7 s |

On 2D problems MMD is about 1.5× faster: 0.69 s against 1.03 s at 263k DOFs. Sequential METIS couldn't be benchmarked on the wheels because it hangs.

Reported upstream as xiaoyeli/superlu_dist#229. Follow-up once it's fixed: #4666.

Parallel symbolic factorisation (opt-in)

Serial symbolic factorisation builds the gathered global matrix and the global L/U structure on every rank, so its memory per rank grows with the problem size whatever the rank count. ParSymbFact=YES distributes both. With RowPerm=NOROWPERM it also removes the remaining global gather (pdgssvx.c:789). On a 3D 7-point Laplacian, using a C driver linked against the wheel's SuperLU_DIST, it saved ~110 MB per rank at 373k unknowns on 8 ranks (1249 MB against 1139 MB). The cost in time ranged from 10% faster to 14% slower.

It isn't the default, because it can hang. In dist_symbLU, a process with nothing to send passes an uninitialised snd_luind to MPI_Alltoallv. With ParSymbFact on a non-power-of-two process count, the processes outside the symbolic factorisation have nothing to send. When the garbage equals MPI_IN_PLACE, the collective mismatches, and the job hangs or aborts with MPI_ERR_TRUNCATE. An earlier version of this PR made ParSymbFact=YES the default. It then hung on 3 ranks in both int64 CI legs, and passed on 2 and 4. Backtraces showed rank 2 in Open MPI's in-place Alltoallv and ranks 0 and 1 in the pairwise one. The bug is in 9.2.1, 9.3.0 and master, and the one-line fix is xiaoyeli/superlu_dist#231. #4666 tracks making ParSymbFact the default once a fixed release is the minimum.

For users who opt in:

• It needs ColPerm=PARMETIS. Setting ColPerm to another ordering through set_option turns ParSymbFact off. An explicit inconsistent combination makes solve throw before entering SuperLU_DIST.
• Fact=SamePattern crashes with it. SamePattern skips the ordering step, but parallel symbolic factorisation needs the separator tree from that step. SuperLU_DIST 9.3.0 segfaults in symbfact_dist, so under ParSymbFact set_A(A, "SamePattern") refactors with DOFACT, and solve rejects Fact=SamePattern set through set_option. Reported as Segfault in symbfact_dist with ParSymbFact=YES and Fact=SamePattern xiaoyeli/superlu_dist#230.
• A DOFACT refactorisation needs a fresh solve structure. DOFACT in set_A now also calls *SolveFinalize, because a new ordering invalidates the cached solve communication pattern. This also fixes the existing set_A(A, "DOFACT") path, which until now left that structure stale, whatever the ParSymbFact setting.

Using the C driver with ParSymbFact on and off, MC64 on and off, and 1–4 ranks, these refactorisation sequences give correct solutions for the latest matrix: DOFACT → DOFACT → FACTORED, SamePattern_SameRowPerm → FACTORED and DOFACT → SamePattern_SameRowPerm → FACTORED.

Tests

• A Poisson solve on a 32×32 P2 mesh, run with the default options, with RowPerm=NOROWPERM and with ColPerm=MMD_AT_PLUS_A. On the current wheels the default case hangs on 2 ranks. With ColPerm=PARMETIS set, the whole SuperLU_DIST test file passes on 2, 3 and 4 ranks.
• A check that ParSymbFact without PARMETIS is rejected.

The tests don't turn ParSymbFact on, because CI runs them on 3 ranks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JudYk8UecTQyirGsLsHUN2