Skip to content

fix: spawned app backends write audit rows through the gateway and cannot read the key - #38807

Draft
chenmingwei23 wants to merge 2 commits into
mainfrom
mkt-app/sel-gateway-client
Draft

chenmingwei23 wants to merge 2 commits into
mainfrom
mkt-app/sel-gateway-client

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A spawned app backend can no longer read the key that signs the audit log, and its audit rows still get written.

Today a spawned app backend loads trust/sel_hmac.key through sel(), so a compromised backend can forge or rewrite audit rows (the second half of residual (b) of the app-backend HIGH). #38356 adds the gateway route that signs rows on a backend's behalf; nothing uses it yet.

Why it matters

The audit log is how an owner reconstructs what an app did. If any app backend can sign rows, the log proves nothing about that app.

Not a goal

  • No change for backends that have no gateway route (no valid bound port, a gateway bound to a specific interface, or no secret): they keep the real writer and the key, with a warning. Making the route reachable there is a follow-up.
  • No new method on the route: a tool use is an ordinary api_access row.
  • Rotation and tool_invocation events stay gateway-only. A governance decision or a degraded event from a backend is written as an app-attributed api_access row (operation governance.decision or governance.degraded): the app's own claim, not gateway policy evidence.

What changed (motivation -> approach -> change)

Approach: put the signing key out of reach of the backend and send its rows to the gateway, which signs them as app:<name>.

  • sel() returns a SelClient inside a spawned app backend. Builtins that run inside the gateway process keep the real sel().
  • trust/sel_hmac.key is hidden in the spawn mask of every app backend that has the gateway route (an origin and a secret). A backend that clears its spawn marker does not get the key back: the mask hides it. The mask follows the route (see the trade-off below).
  • AuditSDK.record (ctx.audit, audit_log()) reaches the log through the route, as an api_access row for app:<name>. A failed write is logged at warning level, not debug.
  • The route (sel_routes.py, from feat(apps): add a gateway route for app backend audit writes #38356) stores the operation as <app>.<operation>; operation is normalized so no newline or control character reaches a text view of the log, and secrets are redacted before normalization. A tool use is the ordinary row with the operation tool.<name>.
  • SelClient sends through loopback_urlopen (no system proxy, no redirect). Any gateway-only log_* method warns once and writes nothing.
  • Adds the end-to-end client test that feat(apps): add a gateway route for app backend audit writes #38356 does not have: a real token exchange, a real route, no key in the backend.
  • Docs: docs/app-kit/api-reference.md and docs/system-specs/modules/app-kit-platform.md describe the new audit_log() path; AGENTS.md and security.md update the SEL key residual note (the key is OS-masked for app backends that have the route).

Trade-off, the mask follows the route: a backend with no gateway origin or no secret keeps the real writer and its key, and the spawn logs a warning. Masking it anyway lost every audit row from such a backend (test_sel_backend_no_origin.py). So for non-loopback binds the key stays readable by those backends and the HIGH stays open there; a loopback listener or a CSRF exemption for the one route is the follow-up.

The owner-enable marker (host_exec_granted): the app token that SelClient and other callbacks need is minted only for an app that holds the owner-enable marker (validate_app_secret). An app enabled before the marker existed has none, so when an app has no marker the spawn now omits KIROCREW_GATEWAY_ORIGIN and its proof (the same dormancy as a non-loopback bind), leaves the key unmasked and logs a warning, so its audit rows are still written in-process. Without this the key was masked and the route unusable, and every row from that app was lost. The only consumer of the origin in the tree, the dev_fleet pointer broker, also needs that token, so it cannot work without the marker either.

A credential-shaped operation is not refused: the writer already redacts operation (sel.py _REDACTED_TEXT_FIELDS), shown by test_sel_operation_redaction.py, so the row is stored with [REDACTED: credential].

Choices kept: sel() still returns SecurityEventLog through a cast (about 288 typed call sites under src/kiro_crew/apps; a narrower type would break them), and sel_client.py no longer needs a nosemgrep comment now that it uses loopback_urlopen. grep -rnE "\bsel\(\)|\b_sel\(\)" src/kiro_crew/apps | wc -l gives 288.

Closes: the second half of residual (b) of the app-backend HIGH for backends that have the route.

Known limitations

  • The backend route client is pinned to the first process that loads sel_client with the four backend variables (KIROCREW_SEL_CLIENT_PID). A backend that never loads it before starting a Python child lets that child claim the client. The child already holds the app's secret and writes as the same app identity, so this is attribution, not escalation; the security review accepted it as LOW.

Backwards compatibility

Writer sweep on origin/main: the _sel route is new in #38356 and has no caller on main; the only writers are SelClient and AuditSDK.record, both in this PR. An app's ctx.audit.record("Publish Doc", ...) in a spawned backend is stored as Publish_Doc; the docs describe the normalization. Existing builtin names such as md_notebook.settings.autoSync pass as written (test_sel_builtin_operations.py scans the builtins), so row history is unchanged.

Reader: src/kiro_crew/sel.py:sel -- app (spawned backend) -- returns SelClient; test_sel_backend_switch.py::test_a_spawned_backend_gets_the_gateway_client_and_never_the_key
Reader: src/kiro_crew/apps/audit_sdk.py:AuditSDK.record -- app -- goes through the route; test_audit_sdk_route.py::test_audit_log_from_a_key_masked_backend_lands_as_an_api_access_row
Reader: src/kiro_crew/apps/backend.py:_start_app_backend_body (key mask) -- app spawn -- masked only with a route; test_sel_backend_no_origin.py::test_audit_log_from_a_backend_with_no_origin_lands_as_an_api_access_row
Reader: any shared code that calls a gateway-only sel() method inside a backend (for example platform/policy_distribution.py log_governance_decision) -- app -- written as an app-attributed api_access row, operation governance. (decision or degraded), the app's own claim and not gateway policy evidence; test_sel_client.py::test_a_governance_decision_becomes_an_app_attributed_api_access_row. Any other gateway-only method warns once and writes nothing (test_an_unsupported_method_warns_once_and_does_not_raise).
Reader: signed session identity (session_token_sig, session_pid_sig) read the same key -- app -- not covered: a spawned backend that starts sessions with the key masked loses signed identity; not verified, the one builtin that spawns sessions runs a stub agent today.

Tests

Red first: commit 1 is tests only and commit 2 is the fix. The suite runs in CI; the tests were not run locally on this head.

  • test_sel_client.py: no system proxy is used; an operation such as Publish Doc is stored as Publish_Doc, a newline or control character becomes _, a 300-character operation is clipped to 64 and an empty one is stored as unnamed; an unsupported method warns once; an oversized outcome, resources or error is redacted then clipped to 500 characters and the row still lands.
  • test_sel_routes.py: the same normalization through the route; test_sel_tool_route.py: tool rows; no other method accepted.
  • test_sel_backend_no_origin.py: with no origin the key is not masked and the row lands.
  • test_audit_sdk_route.py: ctx.audit from a key-masked backend lands as an api_access row; a route failure logs at warning.

Proof: not provable -- other: prove.py was not run on this crew; commit 1 holds the new tests and they fail on the base, shown above.

Manual verification

N/A -- unit coverage sufficient: no UI and no external service; the route and client are exercised end to end in tests. Not checked: the macOS Seatbelt mask path.

Related Issues

Part of the app-backend HIGH; follows #38356.

Sibling sweep

Swept: git grep -n "extra_hidden_dirs" -- src
Siblings: none -- the only app-backend spawn mask site is apps/backend.py; the other hits are agent harness spawns, not app backends.

Pattern harvest

Rule candidate: review-prompt
Pattern: a seal or mask applied unconditionally while the replacement path it relies on is conditional.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b399a28bee3f502be9d1492e25d874eb8e67c2e5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b399a28bee3f502be9d1492e25d874eb8e67c2e5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The riders exist to keep audit rows from being lost once the key is masked. The leftover SecurityEventLog() callers run in the gateway or its daemon, not in a backend.

Not checked: whether the macOS Seatbelt mask actually hides trust/sel_hmac.key from the backend.

Notes

  • Item 7 — undeclared: the cli.py hunk "Only the gateway command drops them" doesn't appear in the visible description. Its harm, a restarted gateway posting rows to its own route, is named only in a code comment.
  • Item 5 — inherited: "Masking it anyway lost every audit row" covers backends with no route. That leaves the HIGH open on non-loopback binds, and the PR says so in its Trade-off section.

What this change ships

Inventory (10 items) — 8 justified

Intent: stop a spawned app backend from reading the audit-log signing key while still writing its audit rows. FIX.

  1. Inside a spawned backend, sel() sends rows to the gateway _sel route instead of signing them itself — justified
  2. The SEL key is OS-masked for every backend that has a route — justified
  3. Backends with no owner-enable marker get no gateway origin and keep the key, with a warning — justified
  4. An enable that creates the marker restarts the backend so it gets an origin — rides along
  5. Backends with no route keep the key and log a warning — inherited
  6. The route normalizes an operation instead of refusing it: over 200 characters was refused, now it is clipped to 64 — justified
  7. A gateway command drops the app-backend env vars it inherited — undeclared
  8. Shared emitters (deny_gate, exfil_oauth, mcp_mount_check) go through sel() instead of SecurityEventLog() — justified
  9. Governance decision and degraded events become app-attributed api_access rows; any other gateway-only method warns once and writes nothing — justified
  10. A failed ctx.audit write logs at warning instead of debug, plus one new posture sink row — justified

[FIRST-PRINCIPLES-REVIEWED] b399a28

Description read

[DESCRIPTION-READ] e0ed3d92827f7d6bb1a127b0f585f213ce7d4157829868a770da1294758624f3

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b399a28bee3f502be9d1492e25d874eb8e67c2e5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the key is masked by the OS whenever the backend has a route that works, and backends without a route keep writing rows, with a warning.

Notes

  • Session signing (session_pid_sig._load_hmac_key) reads the masked key. The PR lists this but leaves it unverified. A spawned backend that starts sessions loses signed identity without any error. Make that a hard failure or a pinning test before an app relies on it.
  • sel() returns a client cast to SecurityEventLog. Inside a backend, any method that isn't log_* raises AttributeError when called, not when type-checked. Today's app call sites use only the log_* methods, but the cast gives no protection for the next one.

[DESIGN-REVIEWED] b399a28

Description read

[DESCRIPTION-READ] 820f3912bdea72404553f662a81cde105c3cb6011b918d46d567b2e90ae29f65

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@chenmingwei23 chenmingwei23 changed the title SEL: spawned app backends write audit rows through the gateway and cannot read the key fix: spawned app backends write audit rows through the gateway and cannot read the key Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b399a28bee3f502be9d1492e25d874eb8e67c2e5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b399a28

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b399a28bee3f502be9d1492e25d874eb8e67c2e5 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b399a28bee3f502be9d1492e25d874eb8e67c2e5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b399a28

Verdict parsed from the review's SHA-scoped output markers for commit b399a28bee3f502be9d1492e25d874eb8e67c2e5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b399a28bee3f502be9d1492e25d874eb8e67c2e5 false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from 0ad8896 to 41dc81b Compare November 18, 2026 00:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch 5 times, most recently from ed490f9 to 1358131 Compare November 18, 2026 07:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from 1358131 to b238b71 Compare November 18, 2026 07:50
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from b238b71 to 41d1d2a Compare November 18, 2026 08:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from 41d1d2a to dcd4f9a Compare November 18, 2026 09:07
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
Base automatically changed from mkt-app/sel-gateway-route to main November 18, 2026 11:36
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from cdef2d3 to 508c4a9 Compare November 18, 2026 17:48
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override opus 508c4a9 deferred: backend.py:870 (an app enabled before its owner-enable marker keeps no gateway origin until its backend restarts) is a known follow-up, not introduced by this fix: the app has less access, not more, and a restart clears it. Tracked in #39160 with repro and acceptance.

@github-actions

Copy link
Copy Markdown
Contributor

AI-review override not recorded: the command shape is not recognised. Use /ai-review override <fable|gpt|design|ux|first-principles|scope|all> <current-sha> <class>[ #N]: <reason, 20 to 500 characters>, where <class> is one of false-positive, not-applicable, pre-existing, evidence-supplied, infra-failure, reissue, deferred, owner-decision. What each class needs: docs/ci/ci-and-reviews.md#risk-release-records.

@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch 3 times, most recently from 581d9ce to d1aa2f6 Compare November 18, 2026 20:12
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch 3 times, most recently from 186ec81 to f856349 Compare November 19, 2026 10:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from f856349 to ea9efa5 Compare November 19, 2026 10:52
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from ea9efa5 to 8c030b8 Compare November 19, 2026 11:13
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch 2 times, most recently from 8bf494a to 894d554 Compare November 19, 2026 12:26
…rmalized operation

Spawned app backends hold no SEL key, so their audit rows reach the log through the
gateway's _sel route and a client that only holds the per-app secret. These tests pin:

- the route and the client write the row as app:<name> with the operation kept under
  the app's namespace;
- the operation is normalized and the row is kept: 'Publish Doc' is stored as
  'Publish_Doc', a newline or control character becomes '_', a 300-character operation
  is clipped to 64 and an empty one is stored as 'unnamed';
- a backend has no SEL key, and the builtin backends' own operation names are stored as
  written;
- a critical write the gateway did not take refuses the action;
- an app enabled without its owner-enable marker has a backend with no gateway origin,
  so enabling it again, which creates the marker, stops and starts its backend; an enable
  that finds the marker already present starts the backend without stopping it;
- a gateway started with an app backend's variables set (a detached restart started
  from a backend inherits them) drops KIROCREW_SPAWNED, KIROCREW_APP_NAME,
  KIROCREW_GATEWAY_ORIGIN and KIROCREW_PROXY_SECRET, so its sel() writes its own rows;
  any other command keeps them;
- the route client belongs to the backend's own process: the backend gets it and
  claims it with its process id, and a child process that inherits the same variables
  (an agent session, a review worker) finds another id, gets no client and writes its
  own security rows.
…nnot read the SEL key

A spawned app backend now holds only its per-app secret. Its audit rows go to the
gateway's POST /api/apps/{name}/_sel route through apps/sel_client.py, and the gateway
signs and appends them as app:<name>. The backend's sel() is that client, the SEL key
is masked from the backend, and the built-in security callers log through sel().

The operation is normalized by one function, sel_client.normalize_operation, that the
client and the route both call: every character outside [A-Za-z0-9_.-] becomes '_',
the result is clipped to 64 characters, and an empty one is stored as 'unnamed'. The row
is kept, so no newline or control character reaches a text view of the log. Credentials
are redacted before the operation is normalized, including a tool name.

The client posts the row as UTF-8 rather than ASCII escapes, so three 500-character
fields of astral characters stay under the route's 16 KiB body cap, and a lone surrogate
in a field is replaced instead of raising.

A critical write waits (bounded, as the exit drain does) for the rows queued before it, so
it cannot land ahead of an earlier row.

An app enabled before its owner-enable marker existed was spawned without the gateway
origin. When an enable creates the marker, the enable route now stops the app's backend
before starting it, so the new backend gets the origin; an enable that finds the marker
already present does not restart it.

A gateway restarted from inside an app backend (a detached restart passes no environment)
inherits the backend's KIROCREW_SPAWNED, KIROCREW_APP_NAME, KIROCREW_GATEWAY_ORIGIN and
KIROCREW_PROXY_SECRET, which would make its own sel() the route client. The gateway
command drops those four variables at entry; every other command keeps them.

Any other process the backend starts inherits them too (an agent session, a review
worker), so the route client is pinned to the backend's own process: the first process
holding the variables writes its id to KIROCREW_SEL_CLIENT_PID, and from_env returns the
client only to the process whose id matches. A child reads a different id and keeps the
real writer.

The docs describe the route, the client and the normalized operation.
@chenmingwei23
chenmingwei23 force-pushed the mkt-app/sel-gateway-client branch from 894d554 to b399a28 Compare November 19, 2026 12:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant