feat(agent): track POSIX CUDA VMM state - #217
Closed
galletas1712 wants to merge 1 commit into
Closed
galletas1712 wants to merge 1 commit into
galletas1712 wants to merge 1 commit into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
This was referenced Sep 4, 2026
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 4, 2026 20:10
f043fae to
e27a00a
Compare
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 11, 2026 02:23
e27a00a to
de8e924
Compare
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 11, 2026 06:10
de8e924 to
5166f5e
Compare
galletas1712
removed this pull request from stack #221
September 11, 2026 06:11
galletas1712
added this pull request to stack #293
September 11, 2026 06:11
This was referenced Sep 11, 2026
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 11, 2026 06:16
5166f5e to
b4d890e
Compare
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 11, 2026 06:21
b4d890e to
570ee43
Compare
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
2 times, most recently
from
September 11, 2026 08:31
427541b to
c75bbc0
Compare
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 11, 2026 09:41
c75bbc0 to
8782781
Compare
Turn the forwarding shim into one that knows which CUDA allocations are shared between processes. Allocations created with exactly the POSIX file-descriptor handle type are tracked behind logical handles; every other handle type passes through untouched. Exports return sealed memfd tickets instead of the driver's descriptor; an import of a ticket asks the creator process for the real descriptor over its control socket. The creator answers from an export cache that holds the one real descriptor per allocation, so its listener never calls into the driver and never takes the shim's main lock, and a creator busy inside a long collective call cannot stall its peers. Bookkeeping lives in hash tables plus a sorted range index that shrink as well as grow; records are freed when no handle and no mapping remains, so a server that maps and unmaps for days does not accumulate dead entries or slow down. One driver handle backs each allocation per process: repeated imports and cuMemRetainAllocationHandle alias it, and the last logical handle releases it. A ticket lives as long as the creator holds a handle or mapping, so freeing memory frees it. cuMemSetAccess keeps the union of access per location so one call per peer GPU survives; a call that partly overlaps a tracked mapping, or would grant more than 32 locations, is refused before the driver sees it. cuMemMap refuses overlapping a tracked mapping and cuMemUnmap may cover several whole mappings but not cut through one. Imports of descriptors that are not tickets are counted so the coordinator can refuse to checkpoint while one is alive. Fork children drop inherited records and register on their first CUDA activity, including through the driver's own symbol resolution, so a child that initializes CUDA without VMM calls is still visible. The multicast wrappers translate tracked member handles for the bind calls and otherwise still forward. The shim does not yet answer the coordinator's lifecycle requests; that is the next change. Compared with the earlier tracking layers (#152, #166): the export happens once at ticket time instead of on demand under the main lock; access sets are merged per location instead of overwritten by the last call; lists that never shrank are replaced by tables that do; the CUDA context is captured at creation rather than only at export; unmap and map are range-checked; the listener retries transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state tests, table and export-cache unit tests, and the earlier suites run under AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
force-pushed
the
feat/cuinterpose-posix-tracking
branch
from
September 11, 2026 10:08
8782781 to
3d32af6
Compare
This was referenced Sep 12, 2026
Contributor
Author
|
Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326–#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Layer 7 of the eleven-PR cuinterpose stack #293. Closed PR #214 is intentionally not in the active stack.
This PR implements running-state POSIX CUDA VMM tracking without checkpoint/restore lifecycle mechanics.
cuMemCreatehas an explicit three-way gate:0CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOROnly successful unsupported exportable creations increment the process-wide diagnostic. It is intentionally sticky for that process generation because cuinterpose did not track the resource and cannot prove that releasing one handle eliminated every derived sharing dependency. FABRIC logs once that checkpoint will be refused. A fork child discards inherited CUDA state and starts with a fresh count.
Tracked allocations receive stable random allocation IDs and tagged logical handles. Repeated imports and retained handles collapse onto one real driver handle per allocation per process. Mappings are indexed by address range; access grants merge by location; partial/ambiguous operations are refused; a possibly partial driver access failure marks state unknown so inspection fails closed.
On first POSIX export, the creator performs one real CUDA export, keeps that descriptor in the export cache under the allocation ID, and returns a sealed ticket FD to the application. An importing shim reads the ticket, connects to the original creator endpoint, requests the ticket's participant/resource/allocation identity, receives a fresh descriptor over
SCM_RIGHTS, imports it, and returns another logical handle. The random allocation ID is the opaque resource identity and export-cache key. Re-export by an importer still names the original creator.A non-ticket descriptor import passes through and is counted as live raw sharing until its driver handle is released. #216 refuses checkpoint while that count is nonzero. The export cache has independent locking, in-flight pinning, drain-on-drop, and lifecycle quiesce/resume so peer requests remain serviceable without the main state lock.
The library constructor establishes the participant ID and
/snapshot-control/cuinterpose-<namespace-pid>.sock.CUINTERPOSE_PARTICIPANT_IDis the explicit identity override used by tests and controlled launchers. The socket is mode0600; the Pod-local controlemptyDiris mounted through a container-namesubPath, so ordinary other Pods have no filesystem path to it and Snapshot-managed target containers receive isolated views. The workload Pod remains one trust domain if its author deliberately gives a sidecar the reserved volume. This boundary does not claim protection from node root or an equivalently privileged workload. Fork children discard inherited CUDA bookkeeping and lazily create a new identity/socket on first CUDA activity. Control-endpoint ownership uses scoped cleanup and direct returns rather thangoto.Stack boundary
Based on #216. #292 adds the isolated current allocation-content storage module; #218 invokes it from
SAVE_ALLOCATIONS/LOAD_ALLOCATIONSand implements unicast teardown/rebuild.Validation
The pinned CUDA 13.1 builder passes 13 tracking tests, 9 table/cache tests, and 9 protocol tests with ASan/UBSan where configured. Coverage includes private/POSIX/FABRIC gate behavior, handle alias collapse, range mapping/access semantics, ticket exchange through a forked child, raw import accounting, listener concurrency, descriptor exhaustion, and table churn.
make testpasses in all Go modules. The final published stack head0eae9c4f9b66093332f00a332afaf56048e1e64apassed the fullmake checkgate, the CUDA 13.1 production build, and all 73 sanitizer-backed native cuinterpose tests. Its physical-GPU suite passed all 3 tests with no skips on two DRA-assigned NVIDIA B200 GPUs. The unicast test kept an explicit allocation-ID ticket-backed peer mapping per worker live across capture and restore; the multicast test required a nonzero multicast VA and shim-logical handle, exercisedBindAddr, and passed its collective and captured-graph replay. Detailed hardware evidence and measurements are in #220.