Skip to content

fix: count repeated app token refusals and answer them with retry-after - #39347

Open
Pearcekieser wants to merge 1 commit into
mainfrom
pearcekieser/app-token-denial-backoff
Open

Pearcekieser wants to merge 1 commit into
mainfrom
pearcekieser/app-token-denial-backoff

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An app whose token exchange keeps failing stops flooding the security event log, and the gateway tells it when to try again.

When the gateway refuses an app's secret at POST /api/apps/<name>/token, it answers 403 and writes two rows. The handler writes an app_token_exchange row with outcome denied, and the request audit writes a POST row with outcome error. Nothing tells the app to slow down, so an app with a stale secret retries as often as it polls. On one host, a single app made 15,715 refused exchanges between 2026-11-11 14:51Z and 2026-11-12 10:48Z, all with the error invalid secret. In that time the log rotated a 33.5 MB file every 2.5 to 5 hours.

Why it matters

The security event log keeps a fixed number of rotated files. When one app's refusals fill them, every other event leaves the retained files within hours. An owner who reads the log after an incident then finds only that app's refusals. The app also gets no signal, so it keeps failing at full rate.

Not a goal

What changed (motivation → approach → change)

The log fills because each identical refusal writes its own rows. The gateway now writes the first refusal of a streak as before and counts the rest. A streak is one app name, one reason (missing header or invalid secret) and one forwarded value. Each 300 s window that counted refusals writes one summary row. It has the same operation, outcome, source and error, resources set to repeated <n> times, and metadata.repeated, first_at and last_at. The log thus still counts every attempt. A grant writes the app's pending summaries before its own row, and gateway shutdown writes the rest, so a crash loses at most one window of counts. A window with no refusal ends the streak. The table holds at most 1,024 streaks, and each key keeps at most 128 characters of the app name.

A counted refusal writes no request row either. refuse marks its request (origin.mark_audit_counted), and the request audit of both the dashboard and the headless chain (server_runtime/middleware_chain.py) skips a marked request. The first refusal and a grant keep their request rows, and an exception is still recorded.

A counted refusal gets a 429 with code auth_throttled and Retry-After: 60. The webhook and messaging routes already give repeated failed auth this status and code. The body keeps the 403's error and hint, so the #30941 wording does not change. The first refusal stays a 403 and its body carries the code missing_secret or invalid_secret.

The in-tree clients back off. The Python App Kit client (kirocrew_client.client._exchange_app_token) and the Dev Fleet pointer broker hold the next exchange after a refusal for its Retry-After. The hold is at least 1 s and doubles to 30 s over refusals in a row, and it is never more than 300 s. Inside the hold they fail without a request. The Python client keys the hold on gateway, app and a hash of the secret, so it tries a corrected secret at once.

SecurityEventLog.log_api_access in src/kiro_crew/sel.py takes a metadata dict for the summary row and does not let it set forwarded. The two refusals in core.py now return a code, so error-code-baseline.json counts two fewer responses without one. A test-floor fixture in test/conftest_resets.py empties the streak table before each test. The dashboard-token-auth, sel and dev-fleet specs and the App Kit API reference describe the behavior.

App repo: KiroCrewCaptain/kirocrew-app-dev-fleet#9

A question for maintainers: do summary rows meet the audit requirements for refused exchanges? Each attempt after the first in a window becomes a count and two timestamps, not a row with its own timestamp and hash link. If the log must keep one row per attempt, the Retry-After answer and the client backoff work without the counting, and I will remove the counting from this PR.

Backwards compatibility

Compatible: every request the gateway refused before is still refused with the same error and hint, and a correct secret is granted as before. After the first refusal of a streak, the status is 429, not 403. A client outside this repo sees one more non-2xx refusal. A client that reads Retry-After waits, and one that does not works as it does today.

Reader: src/kiro_crew/dashboard/handlers/core_security.py:api_sel_events -- crew page -- a summary is an ordinary api_access row, and its count is in resources and metadata; test_app_token_denials.py::test_each_window_writes_one_summary_with_its_count
Reader: src/kiro_crew/dashboard/handlers/core_security.py:api_sel_verify -- crew page -- a summary row goes through SecurityEventLog.log, so it is in the HMAC chain like any row
Reader: src/kiro_crew/cli_commands_security.py:_security -- chat -- kirocrew security prints the summary row with its repeated <n> times resources

Tests

Proof: not provable -- other: prove.py replaces addopts, so the client package's async fixtures do not load and its baseline run is red

Against the base's production code with this head's tests, the changed tests fail. With only middleware_chain.py put back to the previous head, the chain test fails on both chains. test/test_dev_fleet_pointer_broker_backoff.py fails 11 of 11, test/test_sel_bindings.py fails its log_api_access signature pin, test/test_app_token_denials.py cannot import the module, and the client contract test fails 9 and errors 4.

  • test/test_app_token_denials.py drives api_app_token on a ManualClock. It runs the handler through each chain's real request audit and pins one request row for the first refusal, none for the counted ones and one for the grant. It also pins the first 403 row, the counted 429 with Retry-After, one summary per window, the window timer, the quiet window that ends a streak, a grant inside a streak, per-key streaks, the shutdown write, the table bound, the name cap, and a summary write that fails.
  • test/test_dev_fleet_pointer_broker_backoff.py runs the broker against a loopback aiohttp server and pins the hold, the backoff, the 300 s cap and the reset on a grant.
  • packages/kirocrew-client-py/tests/test_gateway_contract.py pins the client hold, its sharing across clients, the skip for a new secret, and no exchange inside the hold.
  • test/test_sel_bindings.py pins the new metadata parameter.

Determinism: the four gateway test files passed 20 runs at -n0 and 10 runs at -n 4 beside the other app token and broker tests. All 187 of their test ids passed in one shuffled order with seed 20261118. The client contract test passed 20 runs and one shuffled order of its 78 ids.

Related backend tests ran in all 7 batches of run_scoped_tests.py on the first head. Each red test there is also red on the base d0321624ef14 on this host, or is a Dev Fleet git-worktree test whose red set differs run to run on both sides. For the request audit change, the audit, chain composition, start-up and provenance tests passed, 445 in all. The flake-pattern ratchet, the process-cache floor ratchet, semgrep 1.78.0 on the diff and pre_push_gate.py pass.

Manual verification

N/A -- the real-machine run below drives the handler over HTTP.

Real-machine verification

Machine: the author's Linux x86_64 host, 128 vCPU.
Before (main dcc3121): 1,000 refused POST /api/apps/demo-app/token requests through the dashboard's real middleware chain, served on a loopback port with an empty data home. The chain had the deny-audit boundary, the Host and CSRF barriers, token auth and the request audit -> 1,000 responses with 403 and no Retry-After, and 2,000 rows for the route. 1,000 were app_token_exchange rows and 1,000 were POST request rows.
After (head e1e579e): the same command -> 1 response with 403 and 999 with 429 and Retry-After: 60, and 3 rows for the route. They are two app_token_exchange rows, the second repeated 999 times with metadata.repeated 999, and one POST row for the first refusal. The previous head 6957ed2 still wrote 1,002 rows, because its request audit wrote a row for each refusal. Every run also wrote one start-up telemetry row that this change does not touch.
Cleanup: own computer, scratch data homes only.

Related Issues

Sibling sweep

Swept: git grep -nE 'headers.get("X-[A-Za-z-]*(Secret|Token)' -- src/kiro_crew/dashboard src/kiro_crew/apps
Sibling: src/kiro_crew/dashboard/handlers/core.py -- changed here
Sibling: src/kiro_crew/dashboard/handlers/relay_viewer.py -- not affected -- its refusal already sends Retry-After
Sibling: src/kiro_crew/dashboard/handlers/core_local_token.py -- not affected -- a loopback-only check of the desktop app's X-Local-Secret, which no app polls
Sibling: src/kiro_crew/dashboard/handlers/updates_channel.py -- not affected -- the same loopback-only X-Local-Secret check
Sibling: src/kiro_crew/dashboard/token_auth.py -- not affected -- on a refused app token both in-tree clients exchange the secret again, so a stale secret repeats at the exchange

Pattern harvest

Rule candidate: review-prompt
Pattern: an audit row written for each refused request on an endpoint that a client retries with no backoff

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Fixes #39346

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners November 19, 2026 01:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e1e579e49db15f000958bbe68540e07b2d125031 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision e1e579e49db15f000958bbe68540e07b2d125031 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 18f4ebc0506f522903f7d5c07381a970acd8f16e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: refusals are still checked and still recorded, the rows are counted the same way the gateway already counts its other repeated refusals, and every part can be reverted.

Notes

  • The open question to maintainers is the one that matters. If the log must keep one row per attempt, take out the counting and keep only the 429 with Retry-After and the client backoff. Those two cut the request rate by themselves, so dropping the counting is a small revert, not a redesign.
  • AUDIT_COUNTED_KEY is now a general "skip this request's audit row" flag that both chains obey. Keep the claim in mark_audit_counted's docstring that only app_token_denials sets it true with a grep-style pin, so a second caller can't quietly remove audit rows.

[DESIGN-REVIEWED] 18f4ebc

Description read

[DESCRIPTION-READ] 128b289ffb64843e3926827a45e6007fb92c096ac6419f8437a814ea0e5c3977

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 18f4ebc0506f522903f7d5c07381a970acd8f16e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 18f4ebc

Verdict parsed from the review's SHA-scoped output markers for commit 18f4ebc0506f522903f7d5c07381a970acd8f16e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 18f4ebc0506f522903f7d5c07381a970acd8f16e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 18f4ebc0506f522903f7d5c07381a970acd8f16e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 18f4ebc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 18f4ebc0506f522903f7d5c07381a970acd8f16e false-positive: <reason, 20 to 500 characters>

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

The kirocrew-client (py3.9) red on this head is the runner image, not this diff: actions/setup-python has no Python 3.9 build for ubuntu-26.04 (run 44437435061). #39351 moves that job to ubuntu-24.04. This PR waits for it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/app-token-denial-backoff branch from 6957ed2 to 18f4ebc Compare November 19, 2026 02:47
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the request audit writes its own POST row for every mutating /api/ request, so counting only the handler row left one row per refusal

  • fixed span=fd646d9bf7c7

fixed in 18f4ebc: refuse now takes the request and marks a counted 429 with origin.mark_audit_counted. Both chains' sel_audit_middleware skip their row for a marked request, and their except arm still records a raised error. test_the_request_audit_writes_no_row_for_a_counted_refusal runs the handler through each chain's real audit layer and fails on both chains with the previous middleware_chain.py. Through the full dashboard chain, 1,000 refusals wrote 2,000 route rows on main, 1,002 on 6957ed2 and 3 on 18f4ebc.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the request audit writes its own POST row for every mutating /api/ request, so counting only the handler row left one row per refusal

  • fixed span=9d5dead640d9

fixed in 18f4ebc: a counted 429 marks its request (origin.mark_audit_counted), and both chains' sel_audit_middleware write no row for a marked request. This is narrower than exempting the route. The first refusal and a grant keep their request rows, the summary rows keep the counts, and an exception is still recorded. test_the_request_audit_writes_no_row_for_a_counted_refusal pins it on both chains. Through the full dashboard chain, 1,000 refusals now write 3 route rows, against 2,000 on main.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
An app whose secret went stale retries the token exchange as often as
it polls, and each refusal wrote its own app_token_exchange row plus
the request audit's POST row. One app could rotate the security event
log every few hours and push every other event out of the retained
segments.

A streak of identical refusals (same app, reason and forwarded
verdict) now writes its first rows and answers 403 as before. Each
later refusal is counted, answered 429 auth_throttled with
Retry-After: 60, and written as one summary row per 300 s window with
the count and the first and last times. The handler marks a counted
refusal's request, and both chains' request audit write no row for a
marked request. A grant or shutdown writes the pending summaries.

The Python App Kit client and the Dev Fleet pointer broker hold the
next exchange after a refusal for its Retry-After, and at least 1 s
doubling to 30 s, capped at 300 s.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Repeated app token exchange refusals flood the security event log

1 participant