feat(agent): cuinterpose coordinator

[galletas1712]

Summary

Layer 6 of the eleven-PR cuinterpose stack #293. Closed PR #214 is intentionally not in the active stack.

This PR replaces the coordinator placeholder with the global orchestration and validation engine. The coordinator never calls CUDA; it connects to each workload shim, validates the combined topology, and drives process-local CUDA work through barriers.

Endpoint discovery and durable identity are separate. The agent supplies observed/namespace PID pairs, from which the coordinator derives socket paths. In the normal agent path, prepare and restore run in the target mount namespace and address /snapshot-control directly; nonempty --proc-root remains available for diagnostics and isolated tests. HANDSHAKE returns each shim's stable 32-hex participant ID, and every later request/reply is bound to that ID. Restore reads expected participant IDs from cuinterpose.state, handshakes the PID-derived endpoints again, and requires an exact set match.

INSPECT combines unicast allocations/mappings and multicast objects/devices/bindings/mappings into one topology. Validation requires coherent creators, in-bounds mappings and members, complete multicast teams, valid access records, and allocation-content flags only on creators. Preflight refuses capture if any participant holds a live raw import or has successfully created an exportable CUDA resource with an unsupported handle type. Both refusals happen before the first destructive operation.

The capture sequence is:

PREPARE_MULTICAST → SAVE_ALLOCATIONS → PREPARE_UNICAST → state_write

The restore sequence is conceptually:

LOAD_ALLOCATIONS → RESTORE_UNICAST → RESTORE_MULTICAST → validate

LOAD_ALLOCATIONS is the creator barrier: every creator allocation and export endpoint is ready before importers run. Multicast restore remains four wire operations—creators, importers, devices, then bindings/mappings—because all devices must finish cuMulticastAddDevice before any binding begins. Capture needs only one concurrent PREPARE_MULTICAST: each shim can remove its local mappings, bindings, exports, and handle in dependency order, then one global barrier protects unicast teardown.

cuinterpose.state is written through a temporary file, file fsync, atomic rename, and directory fsync. Restore reparses it and finishes with a fresh inspection that must match the captured normalized topology exactly. Allocation save/load progress reports allocation_count, allocation_bytes, end-to-end gb_per_s, and device-copy copy_gb_per_s. Elapsed-time calculation is shared with the shim through util/time.c rather than duplicated in the coordinator.

Stack boundary

Based on #215. The shim does not serve the full lifecycle at this layer; #217 adds running-state tracking, #292 isolates current content storage, #218 implements unicast lifecycle, and #219 implements multicast lifecycle.

Validation

The pinned CUDA 13.1 builder passes 14 coordinator tests under ASan/UBSan, including operation ordering, concurrent multicast prepare, restore barriers, state-file corruption, participant mismatch, topology rejection, live raw-import refusal, and unsupported-exportable refusal before teardown. make test also passes in all three Go modules. The final published stack head 0eae9c4f9b66093332f00a332afaf56048e1e64a passed the full make check gate, the CUDA 13.1 production build, and all 73 sanitizer-backed native cuinterpose tests. Its physical-GPU suite passed all 3 tests with no skips on two DRA-assigned NVIDIA B200 GPUs. The unicast test kept an explicit allocation-ID ticket-backed peer mapping per worker live across capture and restore; the multicast test required a nonzero multicast VA and shim-logical handle, exercised BindAddr, and passed its collective and captured-graph replay. Detailed hardware evidence and measurements are in #220.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[galletas1712]

Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326–#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference.