Remote Debugging

Remote Debugging and Replay Inspection

Copper's remote debug API exposes a replaying application over Zenoh. A debugger such as Time Traveler can navigate the recorded timeline, replay CopperLists, inspect task state, query schemas, and read structured logs without adding ad hoc instrumentation to the runtime.

Starting a replay-backed server

Replay binaries should use the standard CLI contract:

--debug-base <zenoh-prefix>
--log-base <recorded-log>
--replay-log-base <replay-output>

cu29::replay::ReplayCli provides this contract for a dedicated replay binary. Use ReplayArgs with #[command(flatten)] when the binary has additional application-specific arguments. Each replay-server process should write to its own replay log path.

Run the application's replay target with --debug-base, --log-base, and --replay-log-base pointing to its Zenoh prefix, recording, and replay output. See the remote debug session example for a runnable host server.

Useful API methods

After session.open, a client can use:

Area	Methods
Navigation	nav.seek, nav.step, nav.replay
Timeline	timeline.get_cursor, timeline.get_cl, timeline.list
State	state.inspect, state.read, state.search
Schema	schema.get_stack, schema.get_type, schema.get_outputs
Logs	logs.strings, logs.list

Session lease

A replay server permits one active debug session. While otherwise idle, the debugger sends a session-scoped health.ping every second. Any request carrying the active session_id also renews the lease. If no session-scoped request arrives for 3 seconds, the server closes the session automatically and another debugger can connect. A health.ping without session_id checks whether the server is reachable but does not renew the lease.

While the lease is active, a second session.open fails with SessionBusy. Run another replay-server process with a different --debug-base and --replay-log-base when two independent logs or timeline cursors must be inspected concurrently.

schema.get_outputs accepts separate page and field_page values so tools can page both output slots and large field catalogs.

Debug values preserve Rust values that plain JSON cannot represent directly. This includes 128-bit integers, non-finite floats, and maps with non-string keys. Schema descriptors also distinguish structs, tuple structs, tuples, lists, arrays, maps, enums, and CuArrayVec containers so inspectors do not have to infer those shapes from display strings.

Handle-backed payloads

CuHandle contents are deferred by default. A payload contains a small handle descriptor, but the debugger does not read or copy the backing buffer unless the request sets:

{
  "include_handle_contents": true,
  "output_indices": [0, 2]
}

The selected contents are returned out of line in the response attachments array. Primitive buffers use raw little-endian bytes; other handle values use CBOR. Binary attachments require the CBOR wire codec. JSON requests that ask for attachments fail explicitly instead of converting the bytes into a large JSON array.

Linux shared-memory requirements

On Linux, the default local transport reserves a 1 GiB Zenoh shared-memory pool and sends replies of 256 KiB or larger through explicitly allocated SHM buffers. Startup checks both RLIMIT_MEMLOCK and available /dev/shm capacity. A large reply is rejected if it cannot use SHM; Copper will not silently copy it over the Unix socket.

Configure the shell that launches both Copper and the debugger before starting either process:

sudo prlimit --pid $$ --memlock=2147483648:2147483648
ulimit -l 2097152
df -h /dev/shm

The ulimit -l value is in KiB. A shell cannot raise its soft limit above its current hard limit, which is why changing only ulimit may fail. For a systemd service use LimitMEMLOCK=2G. For Docker, configure both memory locking and SHM capacity:

--ulimit memlock=2147483648:2147483648 --shm-size=2g

Tests and memory-constrained deployments can choose an explicit profile with RemoteDebugShmConfig, validate it with remote_debug_shm_preflight, pass it to RemoteDebugZenohServer::new_with_shm_config, and configure the client builder with the same profile.

Non-Linux platforms retain the existing 4 MiB pool and 3 KiB transport-optimization threshold. They do not run the Linux RLIMIT_MEMLOCK or /dev/shm checks.