test(cuinterpose): unit, integration, and GPU lifecycle coverage

[galletas1712]

Summary

Refs #295 (approved).

Collect frontend, Rust, fake-driver, GPU, agent/operator, and pod-contract coverage for the assembled stack. Add memory-IPC and jobfile-free lifecycle regressions.

Review boundary

Remove tests for the deleted shim PageBroker backend and launch-job path. Keep this as the final test PR. The fake CUDA driver checks call/ownership behavior; physical-GPU tests check real contents and mappings.

This PR contains one signed-off commit, da64c55c013d9b407a52e447d627dae621ea50f7. Diff: 59 files changed, 6988 insertions(+), 7 deletions(-).

Extend the existing identity test to check leading zeroes, byte order, maximum values, unchanged binary serialization, and malformed input. Cleanup regressions remain in this final test PR.

Stack and compatibility

The stack starts directly on main; it does not depend on the PageBroker GPU-transfer branch or #323. The shim only saves shared creator bytes through host carriers. Never-shared allocations remain native CUDA state. The stack removes launch-job/jobfile support, and older jobfile-dependent or draft shim artifacts are rejected rather than migrated.

PageBroker and native CustomStorage changes are separate. No PR in this stack adds that implementation.

Order	PR	Scope
1	#326	C preload frontend and private ABI
2	#327	typed identities and peer transport
3	#328	process identity and peer export service
4	#329	VMM ownership and allocation lifecycle
5	#342	implement CUDA memory IPC over tracked VMM
6	#330	host-carrier storage for shared allocation bytes
7	#331	multicast tracking and reconstruction
8	#332	Rust lifecycle coordinator
9	#333	package frontend, Rust backend, and coordinator
10	#334	deliver CUDA tools and enable opt-in preload
11	#335	coordinate cuinterpose capture and restore without jobfiles
12	#336	deliver CUDA tools without rewriting workload commands
13	#337	explain host carriers, memory IPC, and restore ordering
14	#338	unit, integration, and GPU lifecycle coverage

Validation

Validation of the assembled implementation:

• Full API, agent, and operator Go tests; key agent race tests.
• Rust workspace tests, strict Clippy, GNU/musl builds, ABI/ELF checks, packaged frontend/fake-driver tests, and memory-IPC regressions.
• Helm tests/lint, Python manifest/report tests, repository lint, and make check in a clean disposable worktree.
• A small real-agent CRIU cross-node test passed on two GPU nodes, covering memory IPC, private/shared VMM, multicast/graph replay, bytes, and original addresses. It preceded only the final manifest-format guard; that guard has local regression coverage.

GLM testing without CustomStorage was cancelled at the user's request and is not a pass. GLM qualification uses a separate composition with CustomStorage; previous experimental GLM results are not qualification of this rebuilt stack. The installed test driver is not claimed to be a stock-driver qualification.

Tests above were run on the assembled implementation, not claimed independently for every source-only intermediate PR. The standard-library cleanup was validated with Rust workspace tests and strict Clippy, GNU/musl builds and packaged fake-driver tests, plus targeted pod-contract and CUDA-agent Go tests. Physical-GPU tests were not rerun for this cleanup; the earlier GPU results above remain attributed to the pre-cleanup implementation.

Cross-node framework qualification with separate CustomStorage composition

The separate schwinns/cuinterpose-customstorage-glm-matrix branch composes this stack with native CustomStorage/PageBroker changes; those changes are not part of this PR. Fresh GLM-5.2-NVFP4 TP8/EP8 capture/restore ran from l9nsv to s2877, with eight B200s per node. Saved post-restore responses were inspected for coherent Paris, Tokyo, and Berlin answers; none of these passes cold-reloaded the model.

Framework	Result and required configuration	Capture	Restore
vLLM	Passed with the default FlashInfer MoE backend; sleep level 1, wake weights before capture	458.247 s	63.498 s
SGLang	Passed with an explicit test-package patch preserving memory-saver/cuinterpose preload chaining	346.889 s	67.255 s
TensorRT-LLM rc24	Passed with explicit NCCL all-reduce and MPI ob1/tcp,self	298.533 s	112.331 s

This PR now includes the tested MPI settings in both TensorRT guide manifests and a regression test; all 37 framework-manifest tests pass. Restricting UCX to TCP alone did not prevent post-restore MPI endpoint timeouts. This changes CPU-side MPI transport, not the CUDA collective strategy.

Limits: default TensorRT collectives remain unqualified: matched full-model source controls become corrupted with the shim, despite reduced eight-GPU collective tests passing. The SGLang result does not qualify its unmodified image. All runs used the existing patched driver, not stock-driver qualification. vLLM/SGLang were not rerun after the separate PageBroker cancellation fix; TensorRT used the fixed broker. No interposer implementation changed during the framework matrix.

The composition was rebased after porting the MPI settings into this PR; its final tree is identical to the tested composition plus those recorded settings. Timings are agent capture/restore wall times, excluding model startup and image pulls. The matrix does not establish 30-second restore parity.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}