feat(diagnostics): identify the agents and exec commands behind sustained CPU or memory load

[coygeek]

Summary

Make T3 Code's existing resource diagnostics answer "which thread, agent, or exec command is consuming this machine's resources?" Surface sustained heavy activity with a direct link to the responsible workload and enough process detail to investigate it.

Problem to solve

Running many agents on one local machine also runs their compilers, tests, shell commands, and background services. An agent waiting for a command can appear quiet while its descendants consume substantial CPU or memory. A list of busy executables such as node, rustc, or a test binary does not tell the user which conversation started them, whether several worker batches are competing, or where to intervene.

In one observed macOS incident involving T3 Code Nightly and Codex, the whole-machine CPU display reached 100%. A live process snapshot contained 33 test-binary processes and 17 processes matching rustc; several inspected tests had been running for roughly 23–45 minutes. Parent-chain inspection traced named headless workers through codex exec to T3's shared Codex app-server. At least two worker batches were active. The busiest individual processes also included macOS filesystem and security services. These observations establish an attribution problem; they do not establish that an individual agent caused all system activity or that the long-running tests were stuck.

This becomes harder as users run dozens of agents on CPU- and memory-constrained machines. Cloud-hosted execution can move the capacity constraint, but users still need to identify the workload and the environment where it runs.

Proposed behavior

• Provide a discoverable resource view that ranks workloads by CPU and memory consumption and connects measured processes to their originating project, thread, turn, delegated agent or headless worker, and exec/terminal command wherever that ownership is known. Let the user open the owning conversation and relevant command activity directly.
• Expand a workload to show its observed process descendants, executable or command, PID, elapsed time, current counters, and a bounded recent history. Include test/build processes launched through an agent, rather than displaying only the provider executable's own consumption.
• Distinguish confirmed ownership, inferred ownership, shared provider overhead, and unknown ownership. Preserve known ownership for observed descendants that outlive their launcher where supported; explain coverage limits for detached, reparented, or externally launched work. A shared provider PID alone must not assign all of its children to one thread.
• Offer configurable sustained-load warnings, including CPU saturation and high memory use or pressure where the host exposes it. A warning should identify the environment, rank the attributable workloads, and link to the same details. Include workloads that remain alive after their initiating turn completes. Coalesce repeated warnings and retain a bounded recent incident summary so users can inspect what happened after the spike stops.
• Let users inspect and navigate to existing interrupt or process controls with the target and action scope clear. Detection should not automatically terminate work. Do not imply that interrupting an agent necessarily stops a detached command.

Acceptance criteria

1. With two threads launching distinct CPU-heavy commands and a delegated/headless child launching a build, the resource view shows the observed workload trees, ranks their measured use, and links each supported workload to the correct thread and command. Shared provider overhead stays separate, and each observed process contributes at most once to CPU totals at a given aggregation level.
2. With a known child continuing after its launcher or turn exits, its consumption remains visible. The view retains previously established ownership where supported and explicitly reports unknown or lost ownership otherwise. Reused PIDs cannot inherit another process's ownership.
3. Sustained activity beyond the configured threshold and duration produces a warning with the responsible environment and attributable workload details. A spike shorter than the configured duration does not produce that warning. Dismissal, recovery, and repeat suppression behave consistently, and the recent incident remains inspectable within a documented retention bound.
4. CPU values state whether 100% means one core or the whole machine. Memory values name their metric and limitations; summed RSS must not be presented as unique physical memory or guaranteed memory recovery. Unavailable pressure or ownership data is shown as unavailable rather than zero.
5. A remote client sees measurements for the execution environment, not its own device. Monitoring applies independently of which provider launched the work; unsupported adapters or platforms state their coverage limits. Existing client surfaces have a usable way to reach the warning and owner details, and an agent-accessible read-only diagnostic query exposes the same supported ownership information.
6. Collection, history, and alert delivery have bounded overhead and retention. Reuse existing telemetry where possible, provide monitoring controls, and validate collection cost with dozens of active worker trees and while idle. Permission failures or collector failure leave agent execution usable and diagnostics visibly degraded.

Affected area

Resource diagnostics, provider and exec/terminal ownership reporting, delegated-agent visibility, and client navigation from heavy-workload warnings to the originating thread or command.

Non-goals

Automatic killing or throttling, CPU quotas, memory limits, a new scheduler, Linux cgroup isolation, cloud migration, or monitoring unrelated agents outside the execution environment's observable ownership. OS services may appear as host-level context, but high activity in a filesystem or security daemon must not be attributed to a particular agent without supporting evidence.

Alternatives considered

• OS task managers and manual parent-chain inspection already identify executables and some worker command lines. They remain useful for corroboration, but shared provider processes and detached children make conversation-level attribution difficult.
• Concurrency caps and OS resource isolation can reduce contention and contain failures. They do not explain which existing workload is responsible, so they are separate follow-ups rather than prerequisites for this feature.
• Extend T3's existing diagnostics with ownership and sustained-load warnings. This is the preferred approach because it connects measured load to the conversation and command where the user can act, without introducing a second general-purpose system monitor.

Supporting context

Classification is a feature request. No failure of a promised resource-attribution or warning contract was established, and the incident does not demonstrate a T3 CPU leak.

Source inspection used upstream main at 737993303d36e10674c54b95e5bd3826682c99c7, not a verified commit of the affected Nightly installation. The existing resource telemetry design describes native process counters and bounded history. The resource telemetry contract exposes process identities, parent PIDs, categories, and counters without thread/turn/exec ownership fields. This proposal builds on existing diagnostics rather than requesting the first resource monitor.

Related proposals cover thread-aware process launch/disposal hooks, CLI-delegated child agents in the Agents panel, and launch-environment thread lineage. Related reports cover Linux OOM containment and collector overhead. They overlap with parts of the problem but do not individually cover owner-linked resource diagnosis and sustained-load warnings. No complete duplicate was found in a bounded search of open and closed issues and discussions.

[only21mil]

From a Linux systemd service before #17662. The service's memory.current was 5.08 GB, while t3 serve had 940 MiB RSS. Process RSS totals were 1,451 MiB for the shared Codex app-server plus wrapper, 1,757 MiB across 114 cua-repl and 341 MiB across 228 node_repl processes. These RSS totals are not a partition of cgroup memory. memory.stat reported 3.94 GB anon, 0.65 GB file and 0.49 GB kernel memory. An age census found 195 of 236 helpers older than one hour.
The cgroup-total monitor produced 81 memory warnings since October 1. Alert records since October 8 at 19:30 showed high=max=oom=oom_kill=0. A service-total warning needs a breakdown by workload.
On Linux with a working systemd user manager, #17662 isolates supported provider and terminal launches under app-t3code-agents.slice. A scope identifies the launched runtime, but Codex can share that runtime across threads, so thread, turn and command ownership still needs a recorded mapping. Our helper totals were not attributed to individual threads. Recording memory.stat's anon and file counters alongside memory.current would make future changes easier to attribute.

Drafted with Claude Opus 5.5 in Claude Code (T3 Code harness).

🤖 Generated with Claude Code

[satyalyadav]

Adding the Linux shared-budget details requested in the closure of #18357.

Our WSL incident in #18350 is an example where memory use alone did not explain the apparent lack of progress. In the October 11 capture, the shared agents slice used about 3.43 GiB against a 3.35 GiB soft threshold and a 5.03 GiB hard limit, plus about 1.46 GiB of swap. The guest had all 2 GiB of swap in use and severe memory stalls. The captured cgroup counters recorded no OOM kill.

Two concurrent dependency installs ran in one Codex scope. The client showed reconnecting, and independent ordinary WSL launches also hung. Kernel records showed communication-ring allocation failures, tracked upstream in WSL #40795. The T3 triage reply confirmed that soft-limit throttling is intentional. These observations do not establish that the soft limit alone caused the WSL failure.

For this diagnostics proposal, it would help to:

• Show the shared slice's memory use, soft threshold, hard limit, and swap use. Explain that the allowance covers multiple agent scopes and is not a separate budget for each thread.
• Distinguish sustained stalls or repeated throttling from an OOM kill. Use pressure measurements and changes in counters where available. A cumulative high-event count alone should not trigger a current warning.
• Keep shared provider overhead separate and show command ownership only where it is known. A shared scope does not identify one responsible thread.
• After a disconnect, label retained measurements as last known and show their age. Unavailable current measurements should remain unavailable, rather than appear as zero or establish the cause of the disconnect.

That would let a user understand why a running command may progress slowly and decide whether to run expensive commands one at a time. Detection should preserve active work and avoid automatic termination or limit changes. Optional resource-budget configuration can remain a separate follow-up to visibility, with the current defaults retained.

Sequential-install testing is still pending. No threshold change has been tested, so this case supports better diagnosis rather than a claim that a new setting would fix WSL.