Repository navigation
Conversation
Derive API pod UIDs from cgroupfs and systemd paths so GPU metrics and NVTX telemetry can use resource-based pod association without Kubernetes clients. Document collector requirements for missing identities and node-wide metrics. Refs DAT-339 Amp-Thread-ID: https://ampcode.com/threads/T-d0e575f3-a125-4b17-9005-97aaa7431ac6 Co-authored-by: Amp <amp@ampcode.com>
gnurizen
approved these changes
Oct 29, 2026
brancz
added a commit
that referenced
this pull request
Oct 30, 2026
GPU metrics previously resolved process resources only at export, and NVTX only at flush, so a process exiting in between could lose its container ID and pod UID. Capture each GPU metric batch resource at recording time, and capture NVTX resources when processing events while the process still exists. Retain the last NVTX snapshot for queued events and final export after exit, refresh it in later active flush intervals, and discard snapshots with their existing buffer/process state. Host/node resources, relabel drops, sampling, debuginfo behavior, and Go symbolization remain unchanged. Identity is still best effort if the process has already disappeared before its first metadata lookup. The regression passes for live processes, fails on the unchanged code for exit-before-export, and passes with this fix for metrics, spans, logs, empty flushes, queued events after exit, and fresh metadata in later batches with the same PID. Both `GOMAXPROCS=4 go test -race -p=4 ./...` and the GPU-tagged suite with the release lazy-linking flags pass. A controlled test against the actual k8sattributesprocessor v1.2.0 confirms workload association after metadata disappearance and confirms host metrics remain unassociated under resource-only rules, using a fake pod cache rather than a live cluster. Independent of #3258, which adds pod UID discovery from local cgroups; the collector requirements documented there still apply. Refs DAT-339.
brancz
added a commit
that referenced
this pull request
Oct 30, 2026
Reapplies the reviewed external-label change from #3260 onto current `main`, because that PR was merged into its stacked feature branch rather than the default branch. The branch preserves the merged pod-UID discovery and GPU/NVTX resource-lifetime fixes from #3258 and #3259. The existing string-valued `--metadata-external-labels` setting now reaches OTLP CPU/memory profiles, host/per-process GPU metrics, NVTX spans/logs, and agent diagnostic logs/traces through both tracing paths. External labels remain defaults, preserving existing workload service names, typed resource identity, host/process separation, and Arrow profile-label behavior. A source marker such as `parca.agent="true"` can be used for shared-receiver routing before Kubernetes enrichment, but the collector still needs resource-based workload association without profiler sender-IP fallback and routing that preserves node telemetry. Closes #3261; tracked in DAT-339. Verified on this main-based branch with `go test -race -p=4 ./...` and the full GPU-tagged race suite using `-ldflags="-extldflags=-Wl,-z,lazy" -tags=osusergo,netgo,gpumetrics`. Formatting and diff checks passed, and the pod-UID parser plus GPU/NVTX buffering implementations are unchanged from `main`. No live Kubernetes/GPU or shared-receiver routing test was performed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local metadata discovery now extracts API pod UIDs from cgroupfs and systemd paths, including custom roots, without Kubernetes API clients or runtime sockets.
This lets per-process GPU metrics and NVTX spans/logs associate with the workload pod instead of the profiler connection.
Static-pod config hashes are not treated as API UIDs, and node-wide metrics remain host/node resources.
The README documents the remaining collector requirements: resource-only association with container-ID support, no sender-IP fallback, and preservation of node telemetry through namespace filtering and routing.
Sampling, debuginfo startup, and Go symbolization are unchanged.
Verified with
GOMAXPROCS=4 go test -race -p=4 ./...and the GPU-tagged equivalent using the existing lazy-linking flags.A controlled test against the actual k8sattributesprocessor v1.2.0 exercised GPU metrics plus NVTX spans/logs, confirmed the old profiler-pod misassociation, and selected the workload with the discovered UID while leaving host metrics unassociated under resource-only rules.
No live GPU or Kubernetes cluster was exercised.
Refs DAT-339.