Skip to content

fix(graph): resolve startup deadlock on liveliness query with many live tokens - #226

Merged
YuanYuYuan merged 2 commits into
mainfrom
dev/liveliness-deadlock
Sep 23, 2026
Merged

YuanYuYuan merged 2 commits into
mainfrom
dev/liveliness-deadlock

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Jul 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A node that joins a system with more than 256 already-live liveliness tokens hangs forever inside ZContextBuilder::build(). This PR gives the startup liveliness query an explicit FifoChannel of capacity 65536, so the replay burst can no longer fill the channel.

Root cause

The replay of live tokens completes before the caller's drain loop starts.

Fact Where
Startup issues a blocking liveliness query Graph::new_with_pattern
The caller drains replies only after get(..).wait() returns drain loop
The query registers its callback, then sends the interest on the calling thread Session::liveliness_query (zenoh 1.9.0)
A callback inside zenoh's DeclareToken handling pushes one Reply per matching token session.rs
The default handler capacity is 256 API_DATA_RECEPTION_CHANNEL_SIZE

No receiver runs during the replay. At token 257 the sender blocks, and startup never completes.

sequenceDiagram
    autonumber
    participant T as Startup thread
    participant Z as Zenoh session
    participant C as Reply channel (cap 256)
    T->>Z: liveliness().get(pattern).wait()
    activate Z
    Z->>Z: send_interest — replay live tokens
    loop each matching live token
        Z->>C: push Reply
    end
    Note over C: token 257 — channel full
    Z--xC: push blocks
    Note over T,C: the drain loop starts only after get() returns, so no receiver exists yet
    deactivate Z
    T-->>T: never reaches replies.recv()
Loading
  • Above the capacity the hang is deterministic, not a race.
  • In the field it looks intermittent, because the live-token count varies between runs around the 256 threshold.
  • That matches the reported hang rate of roughly 30% of startups on the affected system.

The defect sits in the zenoh reply-delivery path, not in how hiroz uses it. eclipse-zenoh/zenoh#2678 proposes the root-cause fix upstream. This PR removes hiroz's exposure without waiting for that change.

Fix

Graph::new_with_pattern passes an explicit handler to the liveliness query:

let replies = session
    .liveliness()
    .get(&c_liveliness_pattern)
    .with(zenoh::handlers::FifoChannel::new(65536))
    .timeout(std::time::Duration::from_secs(3))
    .wait()?;

65536 is a ceiling above any live-token count a realistic ROS graph reaches. A comment at the call site states the ordering, so a reader does not take the constant as arbitrary.

What fails without this

No failing test in CI. The reproduction needs more than 256 matching live tokens in a running system, which the test suite does not build.

  • Symptom without the fix: ZContextBuilder::build() blocks forever on a node that joins a large graph.
  • Evidence for the threshold: the constant and the delivery path linked in Root cause, both read at zenoh 1.9.0.
  • 25 checks green on 084f2d81e2ab9dbe27370fad1e16bab4163fa815, 0 failed, 0 pending.

Note

This change makes the deadlock unreachable at any practical token count. It does not make it impossible. A graph with more than 65536 matching live tokens hits the same ordering. eclipse-zenoh/zenoh#2678 is what removes the class.

Breaking Changes

None.

@YuanYuYuan

Copy link
Copy Markdown
Collaborator Author

The lyrical interop failure on this PR is unrelated to this diff — same class of image/apt-repo ABI skew as #303, on a different package. Fix: #356.

zenoh's liveliness_query() synchronously replays every currently-live
matching token onto the query's reply channel, on the calling thread,
before the call returns. The default handler is a bounded 256-slot
flume channel, so once more than 256 tokens match the pattern, the
synchronous replay blocks forever waiting for a receiver that cannot
run until the call itself returns, deadlocking the caller.
Graph::new_with_pattern replays every live token into the backfill
query's reply channel before anything drains it. The default channel
holds 256, so a session that already holds more matching tokens blocks
in Graph::new.

The test declares 300 tokens on the session that builds the Graph, first
checks with an oversized handler that the query really sees all 300, then
builds the Graph on a thread and fails if it is still blocked after 30 s.
A second session behind a router does not reproduce the hang, so the test
does not use one.
@YuanYuYuan
YuanYuYuan force-pushed the dev/liveliness-deadlock branch from 000924d to 5ec0e2c Compare September 22, 2026 11:45
@YuanYuYuan
YuanYuYuan merged commit 39f1a1b into main Sep 23, 2026
31 checks passed
@YuanYuYuan
YuanYuYuan deleted the dev/liveliness-deadlock branch September 23, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant