Skip to content

fix: retain GPU and NVTX resources until export - #3259

Merged
brancz merged 1 commit into
mainfrom
dat-339-retain-gpu-process-resources
Oct 30, 2026
Merged

brancz merged 1 commit into
mainfrom
dat-339-retain-gpu-process-resources

Conversation

@brancz

@brancz brancz commented Oct 29, 2026

Copy link
Copy Markdown
Member

GPU metrics previously resolved process resources only at export, and NVTX only at flush, so a process exiting in between could lose its container ID and pod UID.
Capture each GPU metric batch resource at recording time, and capture NVTX resources when processing events while the process still exists.
Retain the last NVTX snapshot for queued events and final export after exit, refresh it in later active flush intervals, and discard snapshots with their existing buffer/process state.
Host/node resources, relabel drops, sampling, debuginfo behavior, and Go symbolization remain unchanged.
Identity is still best effort if the process has already disappeared before its first metadata lookup.

The regression passes for live processes, fails on the unchanged code for exit-before-export, and passes with this fix for metrics, spans, logs, empty flushes, queued events after exit, and fresh metadata in later batches with the same PID.
Both GOMAXPROCS=4 go test -race -p=4 ./... and the GPU-tagged suite with the release lazy-linking flags pass.
A controlled test against the actual k8sattributesprocessor v1.2.0 confirms workload association after metadata disappearance and confirms host metrics remain unassociated under resource-only rules, using a fake pod cache rather than a live cluster.

Independent of #3258, which adds pod UID discovery from local cgroups; the collector requirements documented there still apply.
Refs DAT-339.

Capture per-process resource attributes when buffering telemetry rather than resolving them for the first time after a process exits. Refresh NVTX snapshots on subsequent flush intervals while alive, and retain them for queued events and final export.

Refs DAT-339
Amp-Thread-ID: https://ampcode.com/threads/T-d0e575f3-a125-4b17-9005-97aaa7431ac6
Co-authored-by: Amp <amp@ampcode.com>
@brancz
brancz merged commit 4a3435f into main Oct 30, 2026
11 checks passed
brancz added a commit that referenced this pull request Oct 30, 2026
Extends the existing string-valued `--metadata-external-labels` setting
to OTLP CPU/memory profiles, host/per-process GPU metrics, NVTX
spans/logs, and both agent diagnostic tracing paths plus diagnostic
logs, for DAT-339.
A source marker such as `--metadata-external-labels=parca.agent=true`
lets a shared collector receiver route profiler telemetry before
Kubernetes enrichment.
External labels fill missing resource attributes only, preserving
workload `service.name`, existing typed identity, and Arrow
profile-label behavior without copying the agent environment onto host
or workload resources.
Use a nonconflicting source key and match the string `"true"`, not a
boolean; global pod/container/process labels would also apply to
node-wide metrics.
The collector still needs resource-based workload association including
`container.id`, no sender-IP fallback for profiler traffic, and
namespace routing that preserves node telemetry.

Stacked on #3259 to keep this diff limited to external-label
propagation; retarget to `main` after that PR merges.

Verified with `go test -race -p=4 ./...` and the full GPU-tagged race
suite using `-ldflags="-extldflags=-Wl,-z,lazy"
-tags=osusergo,netgo,gpumetrics`.
Regression tests cover every distinct resource-construction path, string
markers, collision precedence, and host/process separation, including
real gRPC export captures; disabling propagation makes each path fail as
expected.
No live Kubernetes/GPU or shared-receiver routing test was performed.
brancz added a commit that referenced this pull request Oct 30, 2026
Reapplies the reviewed external-label change from #3260 onto current
`main`, because that PR was merged into its stacked feature branch
rather than the default branch.
The branch preserves the merged pod-UID discovery and GPU/NVTX
resource-lifetime fixes from #3258 and #3259.

The existing string-valued `--metadata-external-labels` setting now
reaches OTLP CPU/memory profiles, host/per-process GPU metrics, NVTX
spans/logs, and agent diagnostic logs/traces through both tracing paths.
External labels remain defaults, preserving existing workload service
names, typed resource identity, host/process separation, and Arrow
profile-label behavior.
A source marker such as `parca.agent="true"` can be used for
shared-receiver routing before Kubernetes enrichment, but the collector
still needs resource-based workload association without profiler
sender-IP fallback and routing that preserves node telemetry.

Closes #3261; tracked in DAT-339.

Verified on this main-based branch with `go test -race -p=4 ./...` and
the full GPU-tagged race suite using `-ldflags="-extldflags=-Wl,-z,lazy"
-tags=osusergo,netgo,gpumetrics`.
Formatting and diff checks passed, and the pod-UID parser plus GPU/NVTX
buffering implementations are unchanged from `main`.
No live Kubernetes/GPU or shared-receiver routing test was performed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants