Skip to content

tracking(serve): Split daemon resource protection into reviewable PRs #8091

Description

@doudouOUC

What would you like to be added?

Track the delivery of #8051 as a sequence of small, independently reviewable changes. This replaces the original seven-PR decomposition: #8093 is closed as superseded because its process-wide heap-proxy ledger and generic schedulers do not match the daemon's multi-process memory boundary, were not wired to concrete consumers, and contain confirmed accounting defects.

Current delivery status:

  • Part 1 — feat(serve): resolve and report the daemon memory budget #8245: resolve and report one honest denominator. Configured/effective budgets, constrained/host source, explicitly modeled headroom, active ACP child count, and precise RSS coverage are additive status only; child spawn arguments are unchanged.
  • Part 2a — feat(serve): observe daemon and child memory against real denominators #8423 and feat(serve): report aggregate ACP child RSS, not just the primary's #8462: observe root and active ACP child memory without taking action. The daemon reports root pressure against real RSS/heap denominators and aggregates self-reported RSS for active ACP children with explicit sampled-count, age, and coverage caveats. It does not trigger GC, eviction, session closure, admission refusal, or process termination.
  • Part 2b — complete process-tree observation. Add OS/PID-backed attribution for hung children, channel workers, and managed descendants; report cgroup current usage separately from process-tree RSS; add sampling quality/staleness and recovery hysteresis without remediation. The existing active-child sum remains intentionally partial until this lands.
  • Part 3a — refactor(serve): model a per-child heap partition of the daemon budget #8508: model a fixed ACP child-heap partition. Report a constant per-child ceiling and maximum concurrent-child count whose product fits the modeled pool. off | observe remains non-enforcing; no spawn argument or refusal behavior changes.
  • Part 3b — bug(serve): daemon authorises each ACP child 50% of host memory, never divided by child count #8182: measure peak old-space and apply the child-capacity policy. Before introducing enforce, measure peak old-generation usage inside each ACP child and compare it with the modeled ceiling. The behavioral PR must apply one fixed partition, account for active plus in-flight spawns without races, never raise the legacy ceiling, and document the V8 old-space compatibility change separately from RSS guarantees.
  • Bound ACP and transport containers. Add wire-byte and queue bounds at the actual NDJSON, connection, pre-attach, HTTP, WebSocket, SSE, child-RPC, and IPC containers. Preserve standalone/public defaults until each daemon-owned path has a complete error and ownership contract.
  • Bound event, replay, and prompt containers. Add single-event and retained-byte limits, byte-aware replay behavior, authoritative reload signalling, and active-turn failure semantics without silently truncating legacy full replay.
  • Bound persistence and workspace-supplied inputs. Use bounded or streaming readers for session restore, catalogs, virtual transcripts, branch/rewind, export, settings, QWEN.md, skills, extensions, MCP configuration, and buffered external-process output.
  • Add narrow aggregate retained-byte quotas. Maintain per-workspace and process-wide totals at the insertion/removal points of rings, queues, caches, and concurrent large operations. Count bytes actually retained by each container; do not reintroduce estimated V8 object-cost accounting or caller-declared universal leases.
  • Close generation-scoped lifecycle gaps. Ensure failed initialization, replacement, removal, in-flight settlement, cache cleanup, and background jobs release the exact workspace generation and cannot resurrect a tombstoned runtime.
  • Finish writer-safe process-tree shutdown. Build on merged fix(serve): Isolate daemon session maintenance writers #7975 to seal admission, drain writers, terminate the daemon-owned process group/tree, rescan descendants, clean sockets/streams/timers, and report an unclean non-zero exit when owned work remains.

Each unchecked line may be split into more than one PR when it crosses ownership or compatibility boundaries. The behavioral and container-bound work should be prioritized by measured or directly reproducible risk rather than by a fixed PR number. Every PR must build and test independently against its declared base.

Guardrails:

  • Do not advertise a protected path until its complete production owner, byte/count boundary, failure contract, and release path are wired.
  • Treat every modeled share or reserve as advisory until a dedicated behavioral PR applies it; status must say enforced: false while that remains true.
  • Registration is not allocation. Do not reject or penalize dormant workspaces using registered workspace count as a memory denominator.
  • A changing per-spawn share is not an aggregate bound because existing V8 heaps cannot be shrunk. Any enforced child policy must use the fixed partition modeled by refactor(serve): model a per-child heap partition of the daemon budget #8508.
  • Keep workspace/session count defaults unchanged, and do not add workspace/session capacity LRU, active-session eviction, or generic exactly-once receipts in this phase.
  • Keep observation side-effect free. Report daemon process-tree RSS and cgroup usage separately rather than summing them.
  • Prefer bounds enforced by the container that retains the bytes. Add shared helpers only when a second concrete consumer exists.
  • Preserve standalone ACP and public ndJsonStream behavior unless a PR explicitly scopes, documents, and tests a compatibility change.

Why is this needed?

The daemon's memory risk crosses multiple processes and several distinct ownership boundaries. A single root-heap ledger cannot observe or bound ACP children, channel workers, MCP descendants, native/external memory, or unbounded containers that bypass the ledger. Conversely, changing child heap arguments before measuring peak old-space usage would introduce a compatibility policy without evidence that normal workloads fit the modeled ceiling.

The revised sequence establishes an honest denominator, measures the scopes it can name precisely, and applies focused limits at the places that allocate or retain memory. This keeps partial delivery useful and truthful: every merged PR can state exactly which path it observes or protects without claiming a daemon-wide guarantee prematurely.

Additional context

Completion requires every workstream above to merge or be explicitly deferred with its residual risk recorded, every advertised path to have precise ownership and coverage semantics, and the final integration to pass focused package tests, real daemon E2E coverage, build, typecheck, and lint. No phase is complete while status overstates what is measured or enforced.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions