The Sustained gossipsub soak test step in the zig CI job intermittently hangs indefinitely (observed 90+ minutes) instead of its normal duration.
Evidence
Root cause (self-documented)
The test QuicRuntime: gossip saturation race repro (src/transport/quic/runtime.zig) carries this comment:
Soak-gated: a deliberately extreme multi-shard burst (heavier than live) used to REPRODUCE the advanceInboundStreams ist.conn UAF. It is heavy (~50s) and can flakily deadlock the harness teardown under the synthetic overload, so it is excluded from normal CI. Run with -Denable-soak-tests.
So the test itself acknowledges a flaky teardown deadlock under the synthetic overload. It is excluded from zig build test, but a dedicated CI step runs it with -Denable-soak-tests, where the deadlock surfaces and hangs the job (no per-step timeout → the run sits until the 6h GitHub default).
Impact
- Blocks merges (PR sits
UNSTABLE for hours).
- Wastes a CI runner for hours per occurrence.
- Trains reviewers to ignore/force-merge, eroding the signal.
Suggested remediation (any of)
- Fix the teardown deadlock in the soak harness (the cluster drainer thread join / shutdown ordering under sustained overload) so teardown is deterministic — this is the real fix; the test is valuable.
- Add a hard per-step timeout (e.g.
timeout-minutes: 10) on the soak CI step so a hang fails fast instead of hanging for hours, plus auto-retry-once.
- Add an internal watchdog to the soak test: if the burst/teardown does not complete within N×expected, abort with a clear diagnostic instead of deadlocking.
Until fixed, the practical workaround is cancel + re-trigger (it passes ~most runs).
The
Sustained gossipsub soak teststep in thezigCI job intermittently hangs indefinitely (observed 90+ minutes) instead of its normal duration.Evidence
zigjob (which includes this soak step) completes in ~7 minutes (e.g. runs at 09:22→09:28, 08:18→08:24, 22:51→22:57 onmain).Sustained gossipsub soak test (#235)90+ minutes later; cancelling + re-triggering the same commit passed cleanly in ~7 min. So it is flaky/non-deterministic, not a code defect in that PR.Root cause (self-documented)
The test
QuicRuntime: gossip saturation race repro(src/transport/quic/runtime.zig) carries this comment:So the test itself acknowledges a flaky teardown deadlock under the synthetic overload. It is excluded from
zig build test, but a dedicated CI step runs it with-Denable-soak-tests, where the deadlock surfaces and hangs the job (no per-step timeout → the run sits until the 6h GitHub default).Impact
UNSTABLEfor hours).Suggested remediation (any of)
timeout-minutes: 10) on the soak CI step so a hang fails fast instead of hanging for hours, plus auto-retry-once.Until fixed, the practical workaround is cancel + re-trigger (it passes ~most runs).