Repository navigation
fix: count repeated app token refusals and answer them with retry-after - #39347
Pearcekieser wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: refusals are still checked and still recorded, the rows are counted the same way the gateway already counts its other repeated refusals, and every part can be reverted. Notes
[DESIGN-REVIEWED] 18f4ebc Description read[DESCRIPTION-READ] 128b289ffb64843e3926827a45e6007fb92c096ac6419f8437a814ea0e5c3977 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
The kirocrew-client (py3.9) red on this head is the runner image, not this diff: actions/setup-python has no Python 3.9 build for ubuntu-26.04 (run 44437435061). #39351 moves that job to ubuntu-24.04. This PR waits for it. |
6957ed2 to
18f4ebc
Compare
|
self-added: yes
|
|
self-added: yes
|
An app whose secret went stale retries the token exchange as often as it polls, and each refusal wrote its own app_token_exchange row plus the request audit's POST row. One app could rotate the security event log every few hours and push every other event out of the retained segments. A streak of identical refusals (same app, reason and forwarded verdict) now writes its first rows and answers 403 as before. Each later refusal is counted, answered 429 auth_throttled with Retry-After: 60, and written as one summary row per 300 s window with the count and the first and last times. The handler marks a counted refusal's request, and both chains' request audit write no row for a marked request. A grant or shutdown writes the pending summaries. The Python App Kit client and the Dev Fleet pointer broker hold the next exchange after a refusal for its Retry-After, and at least 1 s doubling to 30 s, capped at 300 s.
18f4ebc to
e1e579e
Compare
Problem / Motivation
Goal: An app whose token exchange keeps failing stops flooding the security event log, and the gateway tells it when to try again.
When the gateway refuses an app's secret at
POST /api/apps/<name>/token, it answers 403 and writes two rows. The handler writes anapp_token_exchangerow with outcomedenied, and the request audit writes aPOSTrow with outcomeerror. Nothing tells the app to slow down, so an app with a stale secret retries as often as it polls. On one host, a single app made 15,715 refused exchanges between 2026-11-11 14:51Z and 2026-11-12 10:48Z, all with the errorinvalid secret. In that time the log rotated a 33.5 MB file every 2.5 to 5 hours.Why it matters
The security event log keeps a fixed number of rotated files. When one app's refusals fill them, every other event leaves the retained files within hours. An owner who reads the log after an incident then finds only that app's refusals. The app also gets no signal, so it keeps failing at full rate.
Not a goal
What changed (motivation → approach → change)
The log fills because each identical refusal writes its own rows. The gateway now writes the first refusal of a streak as before and counts the rest. A streak is one app name, one reason (missing header or invalid secret) and one
forwardedvalue. Each 300 s window that counted refusals writes one summary row. It has the same operation, outcome, source and error,resourcesset torepeated <n> times, andmetadata.repeated,first_atandlast_at. The log thus still counts every attempt. A grant writes the app's pending summaries before its own row, and gateway shutdown writes the rest, so a crash loses at most one window of counts. A window with no refusal ends the streak. The table holds at most 1,024 streaks, and each key keeps at most 128 characters of the app name.A counted refusal writes no request row either.
refusemarks its request (origin.mark_audit_counted), and the request audit of both the dashboard and the headless chain (server_runtime/middleware_chain.py) skips a marked request. The first refusal and a grant keep their request rows, and an exception is still recorded.A counted refusal gets a 429 with code
auth_throttledandRetry-After: 60. The webhook and messaging routes already give repeated failed auth this status and code. The body keeps the 403'serrorandhint, so the #30941 wording does not change. The first refusal stays a 403 and its body carries the codemissing_secretorinvalid_secret.The in-tree clients back off. The Python App Kit client (
kirocrew_client.client._exchange_app_token) and the Dev Fleet pointer broker hold the next exchange after a refusal for itsRetry-After. The hold is at least 1 s and doubles to 30 s over refusals in a row, and it is never more than 300 s. Inside the hold they fail without a request. The Python client keys the hold on gateway, app and a hash of the secret, so it tries a corrected secret at once.SecurityEventLog.log_api_accessinsrc/kiro_crew/sel.pytakes ametadatadict for the summary row and does not let it setforwarded. The two refusals incore.pynow return acode, soerror-code-baseline.jsoncounts two fewer responses without one. A test-floor fixture intest/conftest_resets.pyempties the streak table before each test. Thedashboard-token-auth,selanddev-fleetspecs and the App Kit API reference describe the behavior.App repo: KiroCrewCaptain/kirocrew-app-dev-fleet#9
A question for maintainers: do summary rows meet the audit requirements for refused exchanges? Each attempt after the first in a window becomes a count and two timestamps, not a row with its own timestamp and hash link. If the log must keep one row per attempt, the
Retry-Afteranswer and the client backoff work without the counting, and I will remove the counting from this PR.Backwards compatibility
Compatible: every request the gateway refused before is still refused with the same
errorandhint, and a correct secret is granted as before. After the first refusal of a streak, the status is 429, not 403. A client outside this repo sees one more non-2xx refusal. A client that readsRetry-Afterwaits, and one that does not works as it does today.Reader: src/kiro_crew/dashboard/handlers/core_security.py:api_sel_events -- crew page -- a summary is an ordinary
api_accessrow, and its count is inresourcesandmetadata; test_app_token_denials.py::test_each_window_writes_one_summary_with_its_countReader: src/kiro_crew/dashboard/handlers/core_security.py:api_sel_verify -- crew page -- a summary row goes through
SecurityEventLog.log, so it is in the HMAC chain like any rowReader: src/kiro_crew/cli_commands_security.py:_security -- chat --
kirocrew securityprints the summary row with itsrepeated <n> timesresourcesTests
Proof: not provable -- other: prove.py replaces addopts, so the client package's async fixtures do not load and its baseline run is red
Against the base's production code with this head's tests, the changed tests fail. With only
middleware_chain.pyput back to the previous head, the chain test fails on both chains.test/test_dev_fleet_pointer_broker_backoff.pyfails 11 of 11,test/test_sel_bindings.pyfails itslog_api_accesssignature pin,test/test_app_token_denials.pycannot import the module, and the client contract test fails 9 and errors 4.test/test_app_token_denials.pydrivesapi_app_tokenon aManualClock. It runs the handler through each chain's real request audit and pins one request row for the first refusal, none for the counted ones and one for the grant. It also pins the first 403 row, the counted 429 withRetry-After, one summary per window, the window timer, the quiet window that ends a streak, a grant inside a streak, per-key streaks, the shutdown write, the table bound, the name cap, and a summary write that fails.test/test_dev_fleet_pointer_broker_backoff.pyruns the broker against a loopback aiohttp server and pins the hold, the backoff, the 300 s cap and the reset on a grant.packages/kirocrew-client-py/tests/test_gateway_contract.pypins the client hold, its sharing across clients, the skip for a new secret, and no exchange inside the hold.test/test_sel_bindings.pypins the newmetadataparameter.Determinism: the four gateway test files passed 20 runs at
-n0and 10 runs at-n 4beside the other app token and broker tests. All 187 of their test ids passed in one shuffled order with seed 20261118. The client contract test passed 20 runs and one shuffled order of its 78 ids.Related backend tests ran in all 7 batches of
run_scoped_tests.pyon the first head. Each red test there is also red on the based0321624ef14on this host, or is a Dev Fleet git-worktree test whose red set differs run to run on both sides. For the request audit change, the audit, chain composition, start-up and provenance tests passed, 445 in all. The flake-pattern ratchet, the process-cache floor ratchet, semgrep 1.78.0 on the diff andpre_push_gate.pypass.Manual verification
N/A -- the real-machine run below drives the handler over HTTP.
Real-machine verification
Machine: the author's Linux x86_64 host, 128 vCPU.
Before (main dcc3121): 1,000 refused
POST /api/apps/demo-app/tokenrequests through the dashboard's real middleware chain, served on a loopback port with an empty data home. The chain had the deny-audit boundary, the Host and CSRF barriers, token auth and the request audit -> 1,000 responses with 403 and noRetry-After, and 2,000 rows for the route. 1,000 wereapp_token_exchangerows and 1,000 werePOSTrequest rows.After (head e1e579e): the same command -> 1 response with 403 and 999 with 429 and
Retry-After: 60, and 3 rows for the route. They are twoapp_token_exchangerows, the secondrepeated 999 timeswithmetadata.repeated999, and onePOSTrow for the first refusal. The previous head 6957ed2 still wrote 1,002 rows, because its request audit wrote a row for each refusal. Every run also wrote one start-up telemetry row that this change does not touch.Cleanup: own computer, scratch data homes only.
Related Issues
Sibling sweep
Swept: git grep -nE 'headers.get("X-[A-Za-z-]*(Secret|Token)' -- src/kiro_crew/dashboard src/kiro_crew/apps
Sibling: src/kiro_crew/dashboard/handlers/core.py -- changed here
Sibling: src/kiro_crew/dashboard/handlers/relay_viewer.py -- not affected -- its refusal already sends Retry-After
Sibling: src/kiro_crew/dashboard/handlers/core_local_token.py -- not affected -- a loopback-only check of the desktop app's X-Local-Secret, which no app polls
Sibling: src/kiro_crew/dashboard/handlers/updates_channel.py -- not affected -- the same loopback-only X-Local-Secret check
Sibling: src/kiro_crew/dashboard/token_auth.py -- not affected -- on a refused app token both in-tree clients exchange the secret again, so a stale secret repeats at the exchange
Pattern harvest
Rule candidate: review-prompt
Pattern: an audit row written for each refused request on an endpoint that a client retries with no backoff
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsFixes #39346