Repository navigation
fix: retain GPU and NVTX resources until export - #3259
Merged
Merged
Conversation
Capture per-process resource attributes when buffering telemetry rather than resolving them for the first time after a process exits. Refresh NVTX snapshots on subsequent flush intervals while alive, and retain them for queued events and final export. Refs DAT-339 Amp-Thread-ID: https://ampcode.com/threads/T-d0e575f3-a125-4b17-9005-97aaa7431ac6 Co-authored-by: Amp <amp@ampcode.com>
gnurizen
approved these changes
Oct 29, 2026
brancz
added a commit
that referenced
this pull request
Oct 30, 2026
Extends the existing string-valued `--metadata-external-labels` setting to OTLP CPU/memory profiles, host/per-process GPU metrics, NVTX spans/logs, and both agent diagnostic tracing paths plus diagnostic logs, for DAT-339. A source marker such as `--metadata-external-labels=parca.agent=true` lets a shared collector receiver route profiler telemetry before Kubernetes enrichment. External labels fill missing resource attributes only, preserving workload `service.name`, existing typed identity, and Arrow profile-label behavior without copying the agent environment onto host or workload resources. Use a nonconflicting source key and match the string `"true"`, not a boolean; global pod/container/process labels would also apply to node-wide metrics. The collector still needs resource-based workload association including `container.id`, no sender-IP fallback for profiler traffic, and namespace routing that preserves node telemetry. Stacked on #3259 to keep this diff limited to external-label propagation; retarget to `main` after that PR merges. Verified with `go test -race -p=4 ./...` and the full GPU-tagged race suite using `-ldflags="-extldflags=-Wl,-z,lazy" -tags=osusergo,netgo,gpumetrics`. Regression tests cover every distinct resource-construction path, string markers, collision precedence, and host/process separation, including real gRPC export captures; disabling propagation makes each path fail as expected. No live Kubernetes/GPU or shared-receiver routing test was performed.
brancz
added a commit
that referenced
this pull request
Oct 30, 2026
Reapplies the reviewed external-label change from #3260 onto current `main`, because that PR was merged into its stacked feature branch rather than the default branch. The branch preserves the merged pod-UID discovery and GPU/NVTX resource-lifetime fixes from #3258 and #3259. The existing string-valued `--metadata-external-labels` setting now reaches OTLP CPU/memory profiles, host/per-process GPU metrics, NVTX spans/logs, and agent diagnostic logs/traces through both tracing paths. External labels remain defaults, preserving existing workload service names, typed resource identity, host/process separation, and Arrow profile-label behavior. A source marker such as `parca.agent="true"` can be used for shared-receiver routing before Kubernetes enrichment, but the collector still needs resource-based workload association without profiler sender-IP fallback and routing that preserves node telemetry. Closes #3261; tracked in DAT-339. Verified on this main-based branch with `go test -race -p=4 ./...` and the full GPU-tagged race suite using `-ldflags="-extldflags=-Wl,-z,lazy" -tags=osusergo,netgo,gpumetrics`. Formatting and diff checks passed, and the pod-UID parser plus GPU/NVTX buffering implementations are unchanged from `main`. No live Kubernetes/GPU or shared-receiver routing test was performed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPU metrics previously resolved process resources only at export, and NVTX only at flush, so a process exiting in between could lose its container ID and pod UID.
Capture each GPU metric batch resource at recording time, and capture NVTX resources when processing events while the process still exists.
Retain the last NVTX snapshot for queued events and final export after exit, refresh it in later active flush intervals, and discard snapshots with their existing buffer/process state.
Host/node resources, relabel drops, sampling, debuginfo behavior, and Go symbolization remain unchanged.
Identity is still best effort if the process has already disappeared before its first metadata lookup.
The regression passes for live processes, fails on the unchanged code for exit-before-export, and passes with this fix for metrics, spans, logs, empty flushes, queued events after exit, and fresh metadata in later batches with the same PID.
Both
GOMAXPROCS=4 go test -race -p=4 ./...and the GPU-tagged suite with the release lazy-linking flags pass.A controlled test against the actual k8sattributesprocessor v1.2.0 confirms workload association after metadata disappearance and confirms host metrics remain unassociated under resource-only rules, using a fake pod cache rather than a live cluster.
Independent of #3258, which adds pod UID discovery from local cgroups; the collector requirements documented there still apply.
Refs DAT-339.