Skip to content

fix: keep raised conditional branches in depth-first order - #4

Closed
havogt wants to merge 1 commit into
mainfrom
fix/cfg-raising-branch-order
Closed

fix: keep raised conditional branches in depth-first order#4
havogt wants to merge 1 commit into
mainfrom
fix/cfg-raising-branch-order

Conversation

@havogt

@havogt havogt commented Aug 12, 2026

Copy link
Copy Markdown
Owner

What

ControlFlowRaising._lift_conditionals collected the blocks of a branch with

branch_nodes = set(dfs_conditional(graph, [oe.dst], lambda _, x: x is not merge_block))

dfs_conditional yields a deterministic depth-first ordering, but control flow
blocks do not override __hash__, so a plain set orders them by id() — which differs
between processes.

That order is not internal bookkeeping. It drives branch.add_nodes_from(branch_nodes)
and, through graph.all_edges(*branch_nodes), the order the branch's edges are re-added
in. Two identical lowerings of the same program could therefore produce SDFGs that
serialize differently and hash differently, and could pick a different one of the two
outgoing edges to collapse into the unconditional else.

Why the existing OrderedSet did not already cover this

Graph.all_edges already accumulates into an OrderedSet (dace/sdfg/graph.py:219).
That is not sufficient on its own: an ordered container only preserves the order it is
given, and here it is handed *branch_nodes, whose order was already discarded. The
ordering has to be kept at the point where it is first lost.

Worth noting because it is the natural place to look first, and it looks like the problem
is already solved there.

Difference from spcl#2445

That PR replaced sets whose elements are strings, where iteration order is governed by
PYTHONHASHSEED. This one is identity-hashed, so it varies with address space layout
and survives a fixed hash seed — it needs setarch -R (or an ordered container) to
pin down, not PYTHONHASHSEED.

Evidence

Found in gt4py, whose compile cache is keyed on the serialized SDFG. Across five runs of
the icon4py dycore + diffusion benchmarks on GH200, one program of 67
(compute_rayleigh_w) alternated between two forms in a 3-vs-2 split. The difference:

-  "string_data": "(rayleigh_type == 2)"
+  "string_data": "(not (rayleigh_type == 2))"

i.e. the two branches of the conditional swapping which one carried the negated
condition, taking a ~311-line block with them. It reaches the generated CUDA, not just
the SDFG.

Tests

test_raised_branch_preserves_dfs_order builds a conditional whose taken branch is a
chain of 8 blocks, raises it, and asserts the raised branch keeps the chain order. On
main the blocks come out scrambled (branch_block_6 first); with the fix they are in
order. The chain is long enough that the old behaviour cannot plausibly match by chance.

Note the 7 pre-existing CompilationError failures in that file in my environment are a
local toolchain issue (cmake -G Ninja, no ninja on PATH) and are identical with and
without this change: 8 failed before, 7 failed after, the difference being this test.

Scope

Only the one set that reaches the emitted SDFG is changed. The others in this pass
(added for membership, and the sets in the unstructured-lifting path, whose output is
built from cfg.edges() graph order) do not affect the serialized result and are left
alone.

`ControlFlowRaising._lift_conditionals` collected the blocks of a branch with
`branch_nodes = set(dfs_conditional(...))`. `dfs_conditional` yields a deterministic
depth-first ordering, but control flow blocks do not override `__hash__`, so a plain
`set` orders them by `id()` — which differs between processes.

That order is not merely internal: it drives `branch.add_nodes_from(branch_nodes)` and,
via `graph.all_edges(*branch_nodes)`, the order the branch's edges are re-added in. Two
identical lowerings of the same program could therefore produce SDFGs that serialize
differently and hash differently, and could pick a different one of the two outgoing
edges to collapse into the unconditional `else` branch.

Note that `all_edges` already builds its result in an `OrderedSet`. That is not enough
on its own, because an ordered container only preserves the order it is given; the
ordering has to be kept at the point where it is first discarded.

Unlike the `set` ordering addressed in spcl#2445, this one does not depend on
`PYTHONHASHSEED` — identity hashes vary with address space layout instead — so it
survives a fixed hash seed.

Found in gt4py, whose compile cache is keyed on the serialized SDFG: across five runs of
the icon4py dycore and diffusion benchmarks, one program of 67 alternated between two
forms, differing by which branch of a conditional carried the negated condition.

Only this one `set` is changed. The others in the pass are used for membership tests or
set arithmetic and do not reach the emitted SDFG.
@havogt

havogt commented Aug 12, 2026

Copy link
Copy Markdown
Owner Author

Verified end-to-end

Re-ran the icon4py dycore + diffusion benchmarks on GH200, 5 runs with distinct
PYTHONHASHSEEDs and isolated build caches, with this fix plus #3 applied.
compute_rayleigh_w now produces a single fingerprint across all five runs
(ac36e2c44cdde8ce), and the whole matrix is clean at all three levels:

level pairs result
build-folder fingerprints 10/10 IDENTICAL
translation-cache payloads (SDFG JSON) 10/10 0 of 84 entries differ
generated CUDA (dace_determinism.py check) 10/10 67/67 programs match

For contrast, the same workload before these two fixes: compute_rayleigh_w and
compute_scaling_factor_for_3d_divdamp each alternated between two forms, and
compute_rayleigh_w differed in the generated CUDA on 6 of 10 pairs.

One negative result worth recording, since it is the obvious first thing to try: running
with ASLR disabled (setarch -R) did not make it reproducible. That does not
contradict the identity-hash diagnosis — id() still depends on allocation sequence, so
any earlier divergence propagates — but it does mean a fixed address-space base is not a
usable way to isolate this class of bug. Only the ordered container fixes it.

@havogt

havogt commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Superseded by spcl#2495, which carries this fix plus the unstructured-lifting path.

@havogt havogt closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant