Skip to content

fix: discover Kubernetes pod UIDs from local cgroups - #3258

Merged
brancz merged 1 commit into
mainfrom
dat-339-prepare-parca-agent-release-for-dash0-operator-integration
Oct 30, 2026
Merged

brancz merged 1 commit into
mainfrom
dat-339-prepare-parca-agent-release-for-dash0-operator-integration

Conversation

@brancz

@brancz brancz commented Oct 29, 2026

Copy link
Copy Markdown
Member

Local metadata discovery now extracts API pod UIDs from cgroupfs and systemd paths, including custom roots, without Kubernetes API clients or runtime sockets.
This lets per-process GPU metrics and NVTX spans/logs associate with the workload pod instead of the profiler connection.
Static-pod config hashes are not treated as API UIDs, and node-wide metrics remain host/node resources.
The README documents the remaining collector requirements: resource-only association with container-ID support, no sender-IP fallback, and preservation of node telemetry through namespace filtering and routing.
Sampling, debuginfo startup, and Go symbolization are unchanged.

Verified with GOMAXPROCS=4 go test -race -p=4 ./... and the GPU-tagged equivalent using the existing lazy-linking flags.
A controlled test against the actual k8sattributesprocessor v1.2.0 exercised GPU metrics plus NVTX spans/logs, confirmed the old profiler-pod misassociation, and selected the workload with the discovered UID while leaving host metrics unassociated under resource-only rules.
No live GPU or Kubernetes cluster was exercised.

Refs DAT-339.

Derive API pod UIDs from cgroupfs and systemd paths so GPU metrics and NVTX telemetry can use resource-based pod association without Kubernetes clients. Document collector requirements for missing identities and node-wide metrics.

Refs DAT-339
Amp-Thread-ID: https://ampcode.com/threads/T-d0e575f3-a125-4b17-9005-97aaa7431ac6
Co-authored-by: Amp <amp@ampcode.com>
@brancz
brancz merged commit d2a770f into main Oct 30, 2026
11 checks passed
brancz added a commit that referenced this pull request Oct 30, 2026
GPU metrics previously resolved process resources only at export, and
NVTX only at flush, so a process exiting in between could lose its
container ID and pod UID.
Capture each GPU metric batch resource at recording time, and capture
NVTX resources when processing events while the process still exists.
Retain the last NVTX snapshot for queued events and final export after
exit, refresh it in later active flush intervals, and discard snapshots
with their existing buffer/process state.
Host/node resources, relabel drops, sampling, debuginfo behavior, and Go
symbolization remain unchanged.
Identity is still best effort if the process has already disappeared
before its first metadata lookup.

The regression passes for live processes, fails on the unchanged code
for exit-before-export, and passes with this fix for metrics, spans,
logs, empty flushes, queued events after exit, and fresh metadata in
later batches with the same PID.
Both `GOMAXPROCS=4 go test -race -p=4 ./...` and the GPU-tagged suite
with the release lazy-linking flags pass.
A controlled test against the actual k8sattributesprocessor v1.2.0
confirms workload association after metadata disappearance and confirms
host metrics remain unassociated under resource-only rules, using a fake
pod cache rather than a live cluster.

Independent of #3258, which adds pod UID discovery from local cgroups;
the collector requirements documented there still apply.
Refs DAT-339.
brancz added a commit that referenced this pull request Oct 30, 2026
Reapplies the reviewed external-label change from #3260 onto current
`main`, because that PR was merged into its stacked feature branch
rather than the default branch.
The branch preserves the merged pod-UID discovery and GPU/NVTX
resource-lifetime fixes from #3258 and #3259.

The existing string-valued `--metadata-external-labels` setting now
reaches OTLP CPU/memory profiles, host/per-process GPU metrics, NVTX
spans/logs, and agent diagnostic logs/traces through both tracing paths.
External labels remain defaults, preserving existing workload service
names, typed resource identity, host/process separation, and Arrow
profile-label behavior.
A source marker such as `parca.agent="true"` can be used for
shared-receiver routing before Kubernetes enrichment, but the collector
still needs resource-based workload association without profiler
sender-IP fallback and routing that preserves node telemetry.

Closes #3261; tracked in DAT-339.

Verified on this main-based branch with `go test -race -p=4 ./...` and
the full GPU-tagged race suite using `-ldflags="-extldflags=-Wl,-z,lazy"
-tags=osusergo,netgo,gpumetrics`.
Formatting and diff checks passed, and the pod-UID parser plus GPU/NVTX
buffering implementations are unchanged from `main`.
No live Kubernetes/GPU or shared-receiver routing test was performed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants