quic: residual handshake stalls + ACK-starvation under 3-node all-to-all load (post-#245)

[ch4r10t33r]

Follow-up to #245 (non-blocking dials), which fixed the permanent Initial-handshake deadlock that forked zeam devnets. After that fix a 3-node zeam devnet on a single box now justifies steadily (slot 0→18 in lockstep with wall-clock; previously stuck at finalized=slot 0). But residual transport churn remains and should be chased down separately.

Observed (zeam beam --is-aggregator true, 3 in-process nodes, all-to-all quic-v1, single box)

Over a 90s run:

• ~7× quic_runtime: dial handshake timed out after 20000ms; … stalled_phase=initial — recurring to 2 specific peers, throughout the run (not just boot).
• ~4× zquic: io: client declaring connection lost (no ACK for 60s …); marking draining, one with bif=2497811, ld=2048/2048, pending=5 — i.e. ~2.4 MB in flight, loss-detector tracking array fully saturated, zero ACKs for 60s on an established connection, then teardown.

Consensus stays live (a stable 2/3 supermajority keeps justifying), so this is degraded-but-live, not a hard fork.

Hypotheses to investigate

1. ACK generation / pacing under load — ld=2048/2048 with zero ACKs for 60s suggests the receiver side isn't ACKing (or the sender isn't processing ACKs). Is the receiver's driveLoop servicing inbound packets + emitting ACKs promptly when also doing gossip/req-resp/identify work each tick?
2. CPU starvation of the drive thread — zeam aggregator STARK/XMSS proving is multi-second and CPU-heavy; 3 full nodes + provers on one box may starve the QUIC drive thread enough to miss handshake/ACK deadlines. Need to repro on separate hosts / with proving disabled to isolate transport from environment.
3. advancePendingDials services pending outbound sockets with drive(buf, 0) (non-blocking) once per tick, gated by the listener's 5ms poll — confirm that's frequent enough for a handshake under load, or whether pending dials need their own poll budget.
4. Congestion controller behaviour when ACKs are absent — does cwnd/bytes-in-flight back off, and is the 60s idle teardown the right signal vs. earlier PTO-driven retransmit give-up?

Repro

ssh box, cd /mnt/data/zeam_repo (on feat/replace-libp2p-glue, dep = v0.1.98), ./zig-out/bin/zeam beam --is-aggregator true; watch stderr for stalled_phase / marking draining, and poll http://localhost:9667/lean/v0/checkpoints/justified for liveness.

[ch4r10t33r]

Concrete impact: this breaks delayed-node bulk sync → persistent zeam fork (NOT cosmetic)

Re-traced on the box with debug logging. The "residual churn" here is the direct cause of a permanent zeam devnet fork, via large req/resp responses:

Setup: zeam 3-node beam devnet. Node 3 is a deliberately-delayed sync node (DelayedNodeRunner): it stays offline until node 1 finalizes past genesis, then joins (~slot 12) and must bulk-sync via blocks_by_range.

Observed:

• n3 connects to n1/n2 (connection stays up for a full 60s — not a dial/disconnect problem).
• n3 → n1 status request: works (small, 1 packet).
• n3 → n1 blocks_by_range start_slot=1 count=12: n1 receives and serves it (debug: Handling blocks_by_range request, loads finalized blocks slots 1/3/4/6, writes the response into the connection). But n3 receives 0 chunks and times out after 8s.
• On the responder's connection at the time: bif=2497811, ld=2048/2048, pending=5 — 2.4 MB in flight, loss-detector array fully saturated, zero ACKs for 60s, then marking draining + idle teardown.
• blocks_by_root fallback delivers some blocks but with gaps (same mid-transfer stall) → orphans → n3 frozen at its solo slot-9 head → permanent fork.

Diagnosis: large req/resp responses stall because the receiver never ACKs (or doesn't ACK enough): the sender fills its congestion window, the loss detector saturates at 2048/2048, and the transfer halts after the first window. Anything larger than the initial window (status/single-block work; multi-MB blocks_by_range doesn't) cannot complete. n1↔n2 stay healthy only because they only ever exchange small per-slot gossip.

Next step: investigate the zquic ACK-generation / ack-eliciting path on the receive side (esp. on the client/outbound leg), and loss recovery / flow-control window extension for bulk transfers. This is the blocker for delayed-node and restart resync.

[ch4r10t33r]

Root-caused and fixed in zquic v1.7.41 → zig-libp2p v0.1.99.

The stall was NOT ACK pacing — it was zquic#177: a single sendRawStreamData larger than one 1-RTT packet was queued as one oversized pending entry, which drainPendingStreamSends could not serialize into a datagram and silently dropped (no retransmit). Bulk responses (blocks_by_range ~200 KB/block) were lost → the delayed node couldn't sync → permanent fork. Small writes fit one packet and were unaffected, which is why status and single-block blocks_by_root worked while the bulk path didn't. The error.KnownInvalidBlock zeam was seeing is the same bug one layer up: truncated block bytes fail signature/structural verify and get cached as invalid.

Fix splits oversized writes into per-packet entries at enqueue. Repro: a 2.5 MB raw-app transfer delivered 0 bytes before, completes in ~240 ms after. Closing once devnet revalidation on v0.1.99 confirms a single converged head.