Skip to content

fix: warn at start-up when a container limit pins memory at critical - #38371

Open
bolichen97 wants to merge 1 commit into
mainfrom
fix/37490-small-container-memory-warning
Open

bolichen97 wants to merge 1 commit into
mainfrom
fix/37490-small-container-memory-warning

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway running in a container whose memory limit is at or under agent.resource_critical_gb says so once at start-up and names the settings to lower.

In a container limited to 2 GB, the free-memory reading can never rise above 2 GB, and agent.resource_critical_gb is 2 GB by default. So the memory posture reads critical from the first second, background helpers are refused, the spawn gate sits at its floor, and with the default agent.spawn_min_memory_gb no sub-agent can start. The log never said why, or which settings to change.

Why it matters

In a 2 GB container the memory reading can never rise above the critical threshold, so the posture reads critical for good and, with the defaults, no sub-agent can start; nothing told the operator why or what to change.

Not a goal

  • Scaling the thresholds to the container's limit.
  • Changing when anything is held back (cron, background helpers, the adaptive gate, the spawn floor).
  • Changing the "retry when memory frees" refusal wording.

What changed (motivation → approach → change)

The operator gets one warning at start-up that names the cause and the fix. Nothing else behaves differently.

  • Cause: a fixed threshold in GB is compared with a reading that is capped at the container's limit. When the limit is at or under the threshold, the reading can never get above it.
  • Approach: the maintainer decision on the issue keeps the thresholds fixed and asks for one start-up line instead of scaling them.
  • Change: the new resource_status_container_limit.py module (a sibling of resource_status_agent_ceiling.py, since resource_status.py is past the 1,600-line target) checks the tightest container memory limit against the raw agent.resource_critical_gb, the value the spawn gate reads. When the limit is at or under it, it logs: "The container memory limit (2.0 GB) is at or under agent.resource_critical_gb (2.0 GB), so available memory can never rise above it. For a container this small, lower agent.resource_critical_gb, agent.resource_pressure_gb and agent.spawn_min_memory_gb."
  • Boot path: the gateway's start-up entry point (gateway_runtime/process_entry.py) schedules it the way the plain-text credentials notice is scheduled, in the background and never awaited, so the dashboard binds no later.
  • Docs: the resource_critical_gb help text in config/sections_agent.py (and config-baseline.json) and docs/architecture/resource-protection.md say a container at or under the threshold gets this warning.

Backwards compatibility

Compatible: the only addition is one log line at start-up; no threshold, gate, config key or refusal text changes.

Removes nothing: the diff only adds a background log notice, a help-text sentence and a doc sentence.

Tests

test/test_resource_status.py::test_container_limit_at_critical_logs_one_startup_warning drives the start-up scheduler with the cgroup reading patched, so the real host is never read:

  • 2 GB limit with the defaults: exactly one warning, with the limit and all three setting names;
  • 8 GB limit, no finite limit, or resource_critical_gb set to 0: no warning;
  • resource_pressure_gb 1.0 with resource_critical_gb 2.0 under a 2 GB limit: still one warning, because the spawn gate compares the raw critical value.

Proof: PROVEN test/test_resource_status.py::test_container_limit_at_critical_logs_one_startup_warning[2-None-None-True], test/test_resource_status.py::test_container_limit_at_critical_logs_one_startup_warning[8-None-None-False], test/test_resource_status.py::test_container_limit_at_critical_logs_one_startup_warning[None-None-None-False], test/test_resource_status.py::test_container_limit_at_critical_logs_one_startup_warning[2-1.0-2.0-True], test/test_resource_status.py::test_container_limit_at_critical_logs_one_startup_warning[2-None-0.0-False] (head 191abd4)

prove.py ran on a Linux host against tree 2ed5359c1ea2, the tree of this head.

Determinism: the five cases passed 5 runs in a row, the whole file passed in reversed node order (160 passed), and it passed again under -n auto --dist loadgroup --max-worker-restart=2 (160 passed). No sleeps and no fixed short wait: the test awaits the notice task with kiro_crew.testing.wait.await_until. The flake-pattern ratchet passed, and so did the wait-discipline ratchet in its committed-baseline and pull-request modes.

Gates on that host: all 66 entries of the repository's gate profile, each run as written, exited 0. They include flake8, isort, mypy src/kiro_crew/ (no issues in 4899 source files), the docs lint, the website build, tsc, eslint and the bundle checks, the backend related tests (22 files: 1917 passed, 1 skipped) and the frontend related tests (15625 passed). Also clean: the semgrep rules on the diff (0 findings), scripts/generate_config_baseline.py with no diff, and a merge check against current main (clean).

Manual verification

Host: Ubuntu 24.04.5 LTS, cgroup v2. Each run was a real isolated gateway (its own KIROCREW_HOME, HOME and KIRO_HOME under mktemp -d, a free 127.0.0.1 port) started inside systemd-run --user --scope -p MemoryMax=<N> -p MemorySwapMax=0, and stopped by its own PID.

Before the fix, 2 GB scope: resource_status.probe() read available_gb=1.87, posture='critical'. The gateway became ready (health 200) and logged no line naming the limit or any setting.

With the fix, 2 GB scope, exactly one line:

WARNING kiro_crew.resource_status_container_limit: The container memory limit (2.0 GB) is at or under agent.resource_critical_gb (2.0 GB), so available memory can never rise above it. For a container this small, lower agent.resource_critical_gb, agent.resource_pressure_gb and agent.spawn_min_memory_gb.

With the fix, 8 GB scope (happy path): the gateway became ready and logged no such line (probe: available_gb 7.91, posture ample).

Related Issues

Closes #37490

Sibling sweep

Swept: git grep -nE "resource_critical_gb|spawn_min_memory_gb|_container_cgroup_reading" -- src/kiro_crew
Siblings: none -- the posture, the adaptive gate and the spawn floor all read the same container-capped figure, and this one notice names all three settings

Pattern harvest

Not generalizable: a one-off mismatch between a fixed GB default and a container limit at or under it; the maintainer decision keeps the thresholds fixed, so the remedy is this one start-up notice rather than a rule.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 191abd497cab925c2cc71c4159d94d9b2ac69de0 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 191abd497cab925c2cc71c4159d94d9b2ac69de0 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. This is one cause-naming start-up log line that changes no behaviour. The fix that scales the thresholds was ruled out by the maintainer decision on #37490 and is listed under Not a goal.

What this change ships

Inventory (3 items) — 3 justified

Intent: an operator running in a small container learns once at start-up why memory reads critical and which settings to lower. FIX.

  1. At start-up, the gateway logs one WARNING when the container's memory limit is at or under agent.resource_critical_gb, and names the three settings to lower. — justified
  2. The help text for the resource_critical_gb setting (in Settings and config-baseline.json) now says this warning exists. — justified
  3. resource-protection.md gains a sentence about the warning. — justified

[FIRST-PRINCIPLES-REVIEWED] 191abd4

Description read

[DESCRIPTION-READ] e769131d112ad2071a65b717b8022a517ff0ddc0d0ac6687d6c70e4f0cda307d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 191abd497cab925c2cc71c4159d94d9b2ac69de0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is one log line at start-up, read-only and never awaited, and it takes nothing away.

Notes

  • warn_container_limit_at_critical imports the private subagent._container_cgroup_reading. Since this second reader now depends on it, consider making that reading a public helper in its own module, so a rename in subagent can't silently turn the notice off. The broad except only logs at debug, so a broken import would show nothing.

[DESIGN-REVIEWED] 191abd4

Description read

[DESCRIPTION-READ] be9a24eda90289fdb1d3923e16a22bbb0e9bb54e2d4f047c60424a25984a97b7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 191abd497cab925c2cc71c4159d94d9b2ac69de0 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 191abd4

Verdict parsed from the review's SHA-scoped output markers for commit 191abd497cab925c2cc71c4159d94d9b2ac69de0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 191abd497cab925c2cc71c4159d94d9b2ac69de0 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 191abd497cab925c2cc71c4159d94d9b2ac69de0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 191abd4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 191abd497cab925c2cc71c4159d94d9b2ac69de0 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 17, 2026
The available-memory figure is clamped to the container's cgroup
headroom, so in a container whose memory limit is at or under the
absolute agent.resource_critical_gb it can never rise above that
threshold: with the defaults the posture reads critical for good and
no sub-agent can start. Nothing told the operator why or what to
change.

At start-up the gateway logs one warning, in the background and never
awaited, when the tightest container memory limit is at or under the
raw configured critical threshold, naming the limit and the settings
to lower. No threshold or gate changes.
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in config-baseline.json, docs/architecture/resource-protection.md, src/kiro_crew/config/sections_agent.py and test/test_resource_status.py, and the branch conflicted with it (8cd1bac -> a065e64); CI re-running.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/37490-small-container-memory-warning branch from 069d607 to 191abd4 Compare November 19, 2026 10:14
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

In a container limited to 2 GB, memory posture is permanently critical: background helpers are refused and the sub-agent gate stays paused

1 participant