[Request] Standardize connection monitoring

[cafanasyev]

Title: §4.9 Lost connection: connection monitoring is required but unspecified — request to standardize a health-check mechanism and clarify detection semantics

Summary

§4.9 obliges every node to monitor its connection and reconnect on loss, but the specification
explicitly declines to define how connection loss is detected. Since the protocol guarantees no
server→node traffic outside of acks, a node cannot detect loss passively — every implementer must
invent their own out-of-spec detection mechanism, even though the specification already
standardizes the connection down to its byte-level framing (§4.2). This hurts interoperability directly, because the
2-minute re-registration rule in the same clause is anchored to the undefined moment of "noticing
loss of connection". We describe what we had to build, why it is unsatisfying, how mature protocols
solve this, and a set of options a future revision could adopt. We are not pushing a specific
solution — we would like the working group to pick one, so that all implementations can converge on
it.

What §4.9 says

4.9 Lost connection
The fusion node and every edge node shall monitor the connection to the message handling
application and shall attempt to reconnect if connection is lost.
If a node or fusion node is unable to connect to the message handling application, then it shall
attempt to connect every 10 seconds (s) until it is successful.
If reconnection occurs within 2 minutes (min) of the node noticing loss of connection, re-sending
a registration message shall not be necessary. If a longer time has passed, the ASM shall re-send
a registration message.
NOTE Monitoring of the network connectivity is not covered by BSI Flex 335.

The clause mandates an outcome — shall monitor, shall reconnect, shall re-register after 2
minutes — whose triggering event ("noticing loss of connection") the NOTE explicitly leaves
undefined.

Why this matters for interoperability

1. Loss detection is not trivial over TCP, and the hard cases are the common ones.

• Half-open connection. The server disappears silently (power loss, network partition, NAT/
  conntrack expiry) and no FIN ever arrives. The client's blocking read never returns; writes
  "succeed" into the local TCP send buffer for minutes. Without an active mechanism, the node
  never notices.
• Process suspension. Laptop sleep, container pause, VM migration. Threads freeze, the wall
  clock advances arbitrarily far, and on resume the OS often delivers a buffered FIN immediately —
  so the observed disconnect-to-reconnect gap is seconds regardless of how long the process was
  actually gone.

2. The 2-minute rule inherits the ambiguity. The re-registration deadline is measured from
"the node noticing loss of connection". Detection delay directly shifts that anchor: a real
2-minute-and-1-second outage with a 30-second detection delay looks like a 1.5-minute gap, and a
compliant client will skip a re-registration that the server-side state actually requires. Two
implementations that detect loss at different speeds will make different re-registration decisions
for the identical outage — both formally compliant, one of them wrong in practice. The
complementary server behaviour (close after 3 consecutive missed status reports, retain
registration state for 2 minutes) also only emerges from reading several clauses together; making
it explicit in one normative place would help.

3. The specification already standardizes the connection itself — framing is in scope, so
monitoring naturally is too. §4.2 (Message framing/structure) mandates a wire-level detail of
the TCP byte stream:

NOTE 1 The protobuf binary format is not self-delimiting. When two binary messages are
concatenated, there is no way to tell where one message ends and the next message begins.
To enable decoding of the protobuf messages when they are serialized over a continuous byte
stream, each message shall have the length of the message in bytes added as a prefix to the
message. This length shall take the form of a 32-bit (4-byte) little endian number (just the
length of the actual message, not including the 4-byte prefix).

The 4-byte length prefix already makes the TCP connection SAPIENT-specific: this is not a generic
transport carrying opaque protobuf, but a connection whose byte layout the specification defines.
Having taken ownership of how bytes flow on the connection, leaving whether the connection is
alive unspecified (the §4.9 NOTE) is an inconsistency rather than a scope boundary — connection
monitoring sits at exactly the same layer as the framing rule the specification already contains.

Our context: generic client SDKs, and why we cannot just pick a solution ourselves

From our experience adopting SAPIENT, the single most complicated part of getting another party to
integrate with us is that they must implement a client — framing, registration lifecycle,
keep-alive, reconnection, re-registration — from scratch, and then spend significant time debugging
and polishing it. To lower that barrier we are building open-source, generic client SDKs (starting
with java-sapient-sdk, with other popular
languages to follow), intended to work against any conformant SAPIENT implementation, not just
our own — deliberately avoiding any vendor lock-in, so other parties can reuse them with their own
protocol implementations. We want to help popularise the protocol this way.

This is exactly why the gap in §4.9 bites us: a generic SDK cannot quietly impose its author's
private connection-monitoring convention on every party that adopts it. Whatever mechanism we ship
becomes a de-facto standard for everyone who uses the SDK — without ever having been agreed by the
working group, and without any guarantee that other implementations behave compatibly. Connection
monitoring belongs in the specification precisely so that independent implementations can converge.

What implementers currently have to build (our workaround, and its drawbacks)

The obvious in-band mechanism — a read timeout (SO_TIMEOUT) on the socket — turned out to be
unreliable in practice:

• On SSLSocket, SO_TIMEOUT applies to the underlying raw socket, not the TLS record stream;
  partial TLS records can keep a read blocked in native code long past the configured timeout.
• The timeout applies per read() call, not per logical message. A length-prefixed protobuf frame
  is read in a loop, so a slow trickle of bytes resets the timer on every call and the effective
  timeout becomes unbounded.
• Empirically, with SO_TIMEOUT = 10 s we observed the read loop waking only after 30–40 s —
  enough to push the §4.9 re-registration decision past its window.

So we built an out-of-band watchdog
(SocketClient.startWatchdog,
design rationale in
CHANGELOG §3–§4): every
watchdogInterval it opens a fresh TCP socket to the server's host:port and closes it
immediately — effectively nc -z host port. If the probe fails, the watchdog closes the managed
socket, which unblocks the pending read and drives the reconnect path. Detection is bounded by
watchdogInterval + probeTimeout.

It works, but it has real drawbacks — which is much of why we would prefer a specified mechanism:

• It doesn't monitor the actual open connection. An nc -z-style probe tests whether the
  server is accepting new TCP connections in general, not whether our established connection
  is alive, and not whether the application behind the socket is healthy. Both error directions
  exist: a hung or crash-looping server process whose listen socket the OS still services passes
  the probe (false healthy), and our specific connection can be dead — e.g. an expired NAT
  mapping — while a fresh probe connect succeeds (also false healthy); conversely a transient
  refusal of new connections fails the probe and kills a perfectly healthy established
  connection (false dead).
• The server sees permanent connect/disconnect churn. One extra TCP connection per node per
  watchdogInterval, opened and closed without ever completing a TLS handshake. It pollutes
  server logs, looks like port scanning to security monitoring, and may be dropped or rate-limited
  by firewalls and load balancers.
• A calibration constant leaks into protocol arithmetic. Because detection is delayed by up to
  watchdogInterval + probeTimeout, we must carry that value into every place the §4.9 timing
  rules are evaluated — inflating the measured disconnect gap before comparing it against the
  2-minute grace, and deflating the server-retention deadline in the status loop. The protocol's
  timing rules end up parameterized by an implementation detail the specification knows nothing
  about.
• Every implementer repeats this. None of the above is SAPIENT-specific engineering — it is
  generic plumbing that every independent implementation must rediscover, tune, and debug, and
  each will land on slightly different behaviour.

How mature protocols handle connection health

This problem is old and has a well-converged industry answer: an application-level ping/pong with
a specified liveness rule.

• WebSocket (RFC 6455 §5.5.2–5.5.3): Ping and Pong control frames; either peer may ping, the
  other must pong "as soon as is practical".
• MQTT: PINGREQ/PINGRESP, with a Keep Alive interval negotiated at CONNECT; the broker
  disconnects a client after 1.5 × the keep-alive with no traffic.
• AMQP 0-9-1 / 1.0: heartbeat frames at a rate negotiated during connection tuning; a missed
  heartbeat window closes the connection.
• HTTP/2 (RFC 9113 §6.7): PING frames — this is what gRPC's keepalive is built on.

Notably, all of these run over TCP, which already has SO_KEEPALIVE — and all of them still added
an application-level mechanism, because TCP keepalive only proves the remote kernel is alive, is
not portably tunable from every language runtime, and is invisible to the application on some
stacks.

Options for a future revision

We list the options we see, with trade-offs, without advocating one — the value for the ecosystem
is in the working group picking some mechanism that all implementations share.

A. Application-level Ping/Pong message pair. Add Ping/Pong (or a single Heartbeat) to
the SapientMessage oneof; either peer may send Ping (with a nonce or timestamp echoed in
Pong); the spec defines the send interval, the response bound, and "N consecutive misses ⇒
connection lost".
Pros: the industry-standard solution; transport-agnostic; exercises the full stack (TCP + TLS +
framing + application), so it detects hung processes, not just dead sockets; symmetric — works for
both edge node and fusion node; precisely testable.
Cons: a new message type needs a compatibility story (a v2 peer would silently ignore the
unknown oneof case, so the spec must define behaviour when pings go unanswered by an older peer);
adds periodic traffic (negligible).

B. Fusion node acknowledges StatusReports. Introduce a StatusAck (symmetric to the existing
RegistrationAck/AlertAck). The node→server direction already carries traffic at the negotiated
reporting cadence; acking it gives the edge node passive liveness for free: "no ack for K
consecutive reports ⇒ connection lost" — the exact mirror of the server's existing 3-missed-reports
rule.
Pros: no new periodic messages; reuses the already-negotiated interval; elegant symmetry with
§4.9's server-side rule; additionally confirms the server actually processed the report, which
catches the server-crash-with-fast-restart case that a pure transport ping cannot.
Cons: a mandatory server-side change; detection latency is coupled to the status interval (a
node reporting every 60 s detects loss in minutes, not seconds); provides no coverage in the window
before registration completes.

C. Framing-layer keepalive frame, edge-node-initiated. Define a zero-length frame — a 4-byte
length prefix of 0 with no payload — as a keepalive at the §4.2 framing layer, below protobuf
entirely. The rule is directional: only the edge node may initiate a keepalive frame; the fusion
node shall reply with the same zero-length frame and shall never initiate one. "No reply within the
bound ⇒ connection lost."
The direction restriction is essential, for two reasons. First, it prevents an infinite loop: the
probe and the reply are identical bytes, so a symmetric rule ("whoever receives a zero-frame
answers with one") would have each peer answering the other's answer forever, flooding the
connection from the very first keepalive — this is why protocols with symmetric ping either use
distinct probe/reply frames (WebSocket Ping/Pong opcodes) or flag the reply (HTTP/2 PING ACK
bit), while MQTT instead restricts initiation to the client, exactly as proposed here. Second, the
direction matches the protocol's existing traffic asymmetry: the fusion node already receives
StatusReports at the negotiated cadence and already has its 3-missed-reports rule, so it needs no
probe of its own — it is the edge node that has no guaranteed inbound traffic and therefore no
feedback to detect loss with.
Pros: the SapientMessage schema is untouched, so no protobuf compatibility question arises;
sits at exactly the layer the specification already owns — the same §4.2 clause that mandates the
length prefix; four bytes each way; the round trip proves the peer's read/framing loop is alive,
not merely its kernel (unlike TCP keepalive, which the kernel answers even when the application is
deadlocked).
Cons: existing decoders were never told what a zero-length frame means — a current receiver
would hand an empty byte array to the protobuf parser, obtain a SapientMessage with every field
unset, and its behaviour (ignore, error, close) is undefined, so the mechanism needs an explicit
compatibility statement before a node may probe an un-upgraded server; the reply is naturally
answered from the framing loop, so it proves less than option A — a peer whose reader thread is
alive but whose application logic above it is hung still answers keepalives.

D. Framing-layer ping/pong via two reserved length values. A symmetric variant of C. The
32-bit length prefix can express message lengths up to ~4.29 GB, while real SAPIENT messages are
bytes to kilobytes — almost the entire value space of the field is dead. Reserve two values from
that dead space as control codes, neither followed by a payload: 0x00000000 = ping,
0xFFFFFFFF = pong. A peer receiving ping shall reply with pong; a peer receiving pong shall
not reply — probe and reply are now distinct, so the infinite-loop problem of a symmetric
zero-frame rule disappears without restricting who may probe. This is HTTP/2's PING ACK-flag
mechanism expressed in the length field; the framing loop gains two match arms and never invokes
the protobuf parser for either value.
A natural companion change: declare a maximum message size (the specification currently sets
none), so a receiver no longer has to trust an arbitrary 32-bit prefix — a corrupted or malicious
prefix can today demand a ~4 GB allocation. With a cap, the reserved values sit in
guaranteed-invalid space by rule rather than by accident, and every implementation gains a
bounds check it should have anyway. Both sentinel values are also endianness-palindromes
(00 00 00 00, FF FF FF FF), so even a peer with a byte-order bug recognizes them.
Pros: everything option C offers (schema untouched, same §4.2 layer, 4 bytes per frame,
proves the framing loop is alive), plus full symmetry — either peer may probe, including the
fusion node probing a connection that has connected but not yet registered, the one window C
cannot cover; the failure modes on an un-upgraded peer are favourably arranged — the mild frame
(ping, parsed as an empty message) is the only one an old peer can ever receive, since pong is
only ever sent in reply to a ping, i.e. only to a peer that has just proven it understands the
mechanism.
Cons: the same compatibility statement as C is still required before probing an un-upgraded
peer; two reserved values and response rules in both directions are more spec text than C's
single sentinel plus one directional rule; the pong is answered from the framing loop, so like C
it does not detect a peer whose application logic is hung above a live reader (option A does).

The ask

1. In a future revision, standardize some connection health-check mechanism (any of A–D, or an
   alternative the working group prefers) so that independent implementations converge on
   compatible liveness behaviour instead of each inventing their own.
2. Independently of the mechanism chosen, clarify §4.9: define "noticing loss of connection",
   the detection bound, and the anchor of the 2-minute re-registration rule.

We are happy to contribute — drafting proposed wording, prototyping any of the options in our
open-source SDK, and reporting back interoperability results.

[ArcheronTechnologies]

We hit the same gap from the sensor side and reached a similar conclusion, so one observation that might help narrow the option set.

Whichever detection mechanism is chosen, it may be worth the specification also bounding the detection interval relative to the 2-minute re-registration window, rather than leaving the two independent.

Our mesh layer has an analogous pair: a periodic advertisement that refreshes a neighbour's state, and a TTL after which that state expires. We specified them independently at first. The failure mode is quiet. An adaptive advertisement cadence (Trickle, RFC 6206) can back off past the point where a node still refreshes downstream state inside the TTL, and the sub-tree it feeds expires while every node believes it is behaving correctly. Nothing errors, and nothing logs. The fix was a hard clamp: the maximum advertisement interval is bounded to TTL/2, so a node sitting at the ceiling still refreshes at least twice inside the expiry window.

§4.9 has the same shape. The 2-minute rule is anchored to "the moment the node notices loss of connection", so if detection cadence is left to the implementer, two conforming implementations can disagree about whether re-registration was required, precisely in the slow-detection cases where it matters most. Bounding detection cadence relative to the 2-minute window removes that class of disagreement without mandating any particular mechanism, which seems compatible with what you are asking the working group to decide.

One adjacent trap, in case it is useful when specifying whatever gets chosen: be explicit about which peers count as evidence of liveness. We initially allowed any neighbour advertisement to suppress our own, and a node holding a distinct role could then be suppressed by a peer whose traffic said nothing about our own reachability. Restricting the count to same-role peers fixed it. If a SAPIENT health check ends up being passively observed rather than actively exchanged, the same question arises.

Happy to write either up in more detail if that would be useful.