Deadlock when METIS_NodeND returns different orderings on different processes (METIS_AT_PLUS_A) #229

This is a symptom of using SCOTCH, rather than proper METIS, but this is how any true FOSS binary must be built if they want to use SuperLU_DIST at scale.

Note: This issue text was generated by an LLM (Claude Code), working with a FEniCS developer, and checked against the 9.3.0 sources and the reproducer below.

With ColPerm=METIS_AT_PLUS_A (the default when built with ParMETIS support), pdgssvx has every process compute the column ordering of the gathered global matrix independently:

• SRC/double/pdgssvx.c (v9.3.0, line 1067): get_perm_c_dist(iam, permc_spec, &GA, perm_c);
• SRC/prec-independent/get_perm_c.c (line 96): METIS_NodeND(...) is called on each process

perm_c is not broadcast or checked afterwards, so correctness depends on METIS_NodeND returning identical permutations on all processes. When it doesn't, the processes reach mismatched collectives in pdgstrf/pddistribute and the factorisation deadlocks, with no error.

Real METIS is deterministic, so this works with METIS. It fails with Scotch's METIS compatibility layer (libscotchmetisv5), which SuperLU_DIST can be built against through TPL_PARMETIS_LIBRARIES (-lptscotchparmetisv3 ... -lscotchmetisv5). With its default SCOTCH_DETERMINISTIC=FIXED_SEED and threads enabled, Scotch documents that its orderings may vary between calls. We confirmed this by interposing METIS_NodeND during a 2-process solve: both processes passed the identical graph and got different permutations back.

Reproducer

The setup was:

• SuperLU_DIST 9.3.0, built with -DTPL_ENABLE_PARMETISLIB=ON -DTPL_PARMETIS_LIBRARIES="-lptscotchparmetisv3 -lptscotch -lptscotcherr -lscotchmetisv5 -lscotch -lscotcherr" -DCMAKE_C_FLAGS_INIT="-DSCOTCH_METIS_VERSION=5", against Scotch 7.0.17 (default SCOTCH_DETERMINISTIC, -DTHREADS=ON);
• macOS arm64 with Open MPI 5.0, and also Linux with MPICH;
• a 3D 7-point Laplacian distributed by rows (SLU_NR_loc), with default options except ColPerm, solved once with pdgssvx.

Results:

MMD_AT_PLUS_A also works, as it is deterministic.

Suggestion

Compute perm_c on one process and MPI_Bcast it, for any ordering computed by get_perm_c_dist. That costs n integers of communication. It makes the result independent of whether the ordering library is deterministic, and saves every other process from repeating the serial ordering. If replicated computation is preferred, a cheap checksum compared with MPI_Allreduce would at least turn the deadlock into an error.

For context, downstream in DOLFINx we now default to ColPerm=PARMETIS when ParMETIS support is available (FEniCS/dolfinx#4665), which avoids the problem because that ordering is computed collectively.