Add Talos provider support - #3406
Conversation
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The Talos provider removes unavailable host volumes from the affected Agent features. The static review finds no concrete failure in the changed paths.
🤖 Datadog Autotest · Commit de64bf8 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
🎯 Code Coverage (details) 🔗 Commit SHA: fda2d2b | Docs | View more details | Give us feedback! |
tbavelier
left a comment
tbavelier
left a comment
There was a problem hiding this comment.
nit: Should we report incompatible features as unsupported instead of "simply" dropping host volumes that are not compatible ?
@tbavelier I took another look at this, it was left as is because inherently it was Operator limitation, not Talos, which prevented these features to work. I have a POC now I'm validating and will add changes to the PR. |
npm mounts debugfs into system-probe like oomKill, tcpQueueLength, and cws already do, but it was the only one of the four missing from the Talos renderer fixture. Enable it in talos-dda.yaml and regenerate the goldens so the feature is covered end-to-end before any behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oomKill, tcpQueueLength, npm, and cws each mount debugfs (/sys/kernel/debug) into system-probe, but none mount tracefs. On Talos the kernel exposes tracefs as a standalone mount at /sys/kernel/tracing rather than nesting it under debugfs, and a hostPath mount is not recursive across sibling mount points, so CO-RE eBPF probe init failed for all four features: tracefs: /sys/kernel/tracing is not mounted with tracefs filesystem type Add the mount through the existing provider-capabilities mechanism, gated on TalosProvider, so only clusters declaring that provider are affected. The mount is read-write to match the existing debugfs mount, since system-probe writes to kprobe_events to register kprobes. npm did not implement ProviderAwareFeature before; it is discovered by type assertion, so the new method needs no registration. Verified on a Talos v1.13.7 EC2 cluster (kernel 6.18.39-talos): the error is gone and all four modules register (network_tracer, tcp_queue_length_tracer, oom_kill_probe, event_monitor), with oom_kill and tcp_queue_length reporting [OK]. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Talos tracefs support exists only in some feature hooks. USM, Dynamic Instrumentation, and eBPF Check can omit the tracefs mount, so the related system-probe module cannot start.
🤖 Datadog Autotest · Commit bfdb709 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| // mutations for the node agent. Talos exposes tracefs as a standalone mount | ||
| // rather than nesting it under the debugfs mount this feature already adds, | ||
| // so it must be mounted explicitly. | ||
| func (f *npmFeature) NodeAgentProviderCapabilities() providercaps.ProviderCapabilityMap { |
There was a problem hiding this comment.
USM, Dynamic Instrumentation, and eBPF Check can fail on Talos.
Assertion details
- Input: Enable the Talos provider and enable USM, Dynamic Instrumentation, or eBPF Check without NPM, OOM Kill, TCP Queue Length, or CWS.
- Expected:
Each Talos system-probe feature must mount /sys/kernel/tracing. - Actual:
The system-probe container has the debugfs mount but no tracefs mount. The module cannot use Talos tracefs and does not start.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
npm already declared the Talos tracefs capability; usm, dyninst, ebpfcheck, sbom (enrichment) and gpu (privileged mode) mount debugfs into system-probe the same way and needed the same capability. sbom and gpu gate it on the condition that adds their debugfs mount. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both Talos goldens carried spec hashes that no longer reproduce, so TestRender_Golden failed on this branch. Rendered content is unchanged; only the hash annotation moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
More details
The Talos capability changes remove the host files that Talos lacks. They also mount standalone tracefs for the Agent features that use system-probe.
🤖 Datadog Autotest · Commit 545b7fc · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
tbavelier
left a comment
tbavelier
left a comment
There was a problem hiding this comment.
============
System Probe
============
Status: Running
Uptime: 2m30.000369542s
Last Updated: 2026-09-10 13:39:03 UTC (1789047543000)
USM
===
Status: Running
Last Check: 2026-09-10 13:38:58 UTC (1789047538000)
Blocked Processes: [
{
"PathIdentifiers": [
{
"Dev": 34,
"Inode": 111,
"Reason": "no-symbols",
"SamplePath": "/host/proc/1/root/usr/bin/init"
},
{
"Dev": 34,
"Inode": 86,
"Reason": "no-symbols",
"SamplePath": "/host/proc/8/root/usr/bin/containerd"
},
{
"Dev": 51,
"Inode": 12584461,
"Reason": "failed to find symbols []string{\"crypto/tls.(*Conn).Close\", \"crypto/tls.(*Conn).Read\", \"crypto/tls.(*Conn).Write\"}",
"SamplePath": "/host/proc/68/root/usr/local/bin/kubelet"
},
{
"Dev": 64,
"Inode": 40800,
"Reason": "not-go",
"SamplePath": "/host/proc/172/root/pause"
},
{
"Dev": 72,
"Inode": 40800,
"Reason": "not-go",
"SamplePath": "/host/proc/182/root/pause"
},
{
"Dev": 76,
"Inode": 8547068,
"Reason": "no-symbols",
"SamplePath": "/host/proc/227/root/usr/local/bin/kube-proxy"
},
{
"Dev": 78,
"Inode": 30009,
"Reason": "no-symbols",
"SamplePath": "/host/proc/461/root/opt/bin/flanneld"
},
{
"Dev": 83,
"Inode": 40800,
"Reason": "not-go",
"SamplePath": "/host/proc/601/root/pause"
},
{
"Dev": 95,
"Inode": 12747739,
"Reason": "no-symbols",
"SamplePath": "/host/proc/635/root/manager"
},
{
"Dev": 105,
"Inode": 40800,
"Reason": "not-go",
"SamplePath": "/host/proc/743/root/pause"
},
{
"Dev": 34,
"Inode": 87,
"Reason": "no-symbols",
"SamplePath": "/host/proc/43/root/usr/bin/containerd-shim-runc-v2"
},
{
"Dev": 117,
"Inode": 40800,
"Reason": "not-go",
"SamplePath": "/host/proc/841/root/pause"
},
{
"Dev": 119,
"Inode": 263994,
"Reason": "failed to find symbols []string{\"crypto/tls.(*Conn).Read\", \"crypto/tls.(*Conn).Write\", \"crypto/tls.(*Conn).Close\"}",
"SamplePath": "/host/proc/872/root/opt/datadog-agent/bin/datadog-cluster-agent"
},
{
"Dev": 129,
"Inode": 9144337,
"Reason": "failed to find symbols []string{\"crypto/tls.(*Conn).Write\", \"crypto/tls.(*Conn).Close\", \"crypto/tls.(*Conn).Read\"}",
"SamplePath": "/host/proc/1102/root/opt/datadog-agent/embedded/bin/trace-loader"
}
],
"ProgramType": "go-tls"
}
]
Discovery Service Map
=====================
Status: Not running
NPM
===
Status: Running
Last Check: 2026-09-10 13:38:58 UTC (1789047538000)
OOM Kill
========
Status: Running
Last Check: 2026-09-10 13:38:58 UTC (1789047538000)
Event Monitor
================
Status: Runningfrom system-probe on my talos qemu cluster 🔥
4 participants
What does this PR do?
CONTP-2027
Add Talos provider support.
Motivation
Get Operator in parity with helm chart
providerssupport.Additional Notes
Anything else we should know when reviewing?
Minimum Agent Versions
Are there minimum versions of the Datadog Agent and/or Cluster Agent required?
Describe your test plan
Tested both using Docker and EC2 with Talos AMI, most of the feature work. APM with UDS worked but had to add a socket mount to application pod.
system-probelikenpm,tcpQueueLength,oomKilldon't work; neither doessecurity-agentfeatures due toConfirmed iveprocesses and sysprobe, security features work by looking
datadog.runtime_security.process_resolver.cache_sizetcp_queue.read_buffer_max_usage_pctdatadog.network_tracer.usm.http.total_hitsdatadog.security_agent.compliance.runningforhaven't reproduced oom_kill but should work too.
pod-security.kubernetes.io/enforce: privilegedto agent namespace.dda
Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel