Skip to content

fix(ci): wait for the rate-limit reset and give readiness its own pool - #39401

Open
Zedmor wants to merge 2 commits into
mainfrom
fix/ci-rate-limit-wait
Open

Zedmor wants to merge 2 commits into
mainfrom
fix/ci-rate-limit-wait

Conversation

@Zedmor

@Zedmor Zedmor commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A CodeQL or Dependency License Gate job that meets a spent API rate limit waits for the reset instead of going red, and PR Readiness can read from a token pool the review lanes do not spend.

Every workflow here shares one pool of 15,000 REST requests an hour through GITHUB_TOKEN. It runs dry most hours. Issue #39367 measured it over 14 days: the rate limit is behind 255 of 256 red CodeQL jobs and 93 of 95 red Dependency License Gate jobs, about 51% of all red jobs. Those jobs fail instead of waiting, and the author re-runs or pushes again.

Where the two jobs die (job logs, 2026-11-19):

Job Run / job Failing step Log line
CodeQL Analyze (actions) 44441837332 / 135284300952 github/codeql-action/analyze, at Uploading results, after the scan finished ##[error]API rate limit exceeded for installation.
Dependency License Gate 44443057178 / 135288396803 actions/dependency-review-action, twice, then our probe Dependency review is not supported on this repository (both attempts), then could not run ... (gh: API rate limit exceeded for installation

Both failing calls are inside a third-party action. No gh wrapper reaches them. In the license gate the action hides the limit behind the words it uses for a dependency-graph outage.

Why it matters

A red CodeQL or license job says nothing about the diff, but it costs a re-run (a CodeQL Python scan is about 22 minutes) and each re-run spends more of the same pool. PR Readiness reads the lanes it aggregates from that pool too, so a drained pool also delays the verdict.

Not a goal

  • Making CodeQL, the license gate or any lane pass, skip or report success when it could not run.
  • Changing a required check, branch protection, the sweep's reserve, or the sweep or lane frequency.
  • Creating the secret, a token or a GitHub App. That is a maintainer action (steps below).
  • Changing the AI review lanes. See "What is left".

What changed (motivation → approach → change)

Jobs die on a spent pool inside a third-party action → the existing shared helper can only wrap gh → extend that one helper with a wait a step can run before an action, and use it in the two jobs.

1. The shared helper (.github/actions/gh-rate-limit, added for #28477). It now also installs gh-rate-limit-wait [floor]. Same script, same job-wide max-wait budget as the gh wrapper; no second helper.

/rate_limit reading (not counted against the quota) Result
core remaining >= floor prints ready, no wait
below the floor, reset inside the budget one sleep until the reset + 5 s, prints waited
below the floor, reset past the budget exit 1, ::error:: names the rate limit and says it is CI infrastructure, not the code under test
still below the floor after the reset exit 1, same wording
unreadable prints ready with a warning; the guarded action then runs and decides by itself

It retries nothing and reports nothing for the action it guards. The wrapper's own behaviour is unchanged except that its budget error now also says it is CI infrastructure.

2. CodeQL (codeql.yml). The upload comes after a scan of up to 22 minutes, so a wait before the scan would not help. analyze now runs with upload: failure-only, and the upload is its own upload-sarif step (same pinned commit, same category) right after a wait for a pool under 100 requests.

There is one upload per job and no retry. At the pinned commit upload-lib.ts validateUniqueCategory sets a per-category marker before it sends the request and refuses a second upload of that category in the same job, so a retry step could only fail. A failed upload fails the job, as it did when analyze uploaded. The job is green only when the upload succeeded. Timeout 60 → 75 minutes, to hold the 30-minute wait budget.

3. Dependency License Gate (dependency-review.yml). Waits before the first review. The pause before the retry is the wait for the reset when the pool is spent, and the fixed 60 s otherwise. After two failed reviews the verdict step decides in this order:

State Result
the retry read the dependency graph and reported a license it does not accept (invalid-license-changes output) the action's license verdict
the pool is under 100 requests "could not run", names the rate limit; no probe, because a probe that waited for the reset would answer and the job would call a rate-limited retry a license finding
otherwise the existing probe: API answers → license verdict; does not → "could not run"

All three fail the job. Timeout 10 → 40 minutes.

4. PR Readiness and the sweep: a pool for reads (second commit). New repository secret PR_READINESS_READ_TOKEN. A step Select the token pool for reads, identical in both workflows, asks that token for /rate_limit and logs the pool and its remaining count:

Secret Log line Reads use
unset readiness reads: token pool -- shared (GITHUB_TOKEN); PR_READINESS_READ_TOKEN is not set GITHUB_TOKEN, as today
set, >= 50 requests left readiness reads: token pool -- readiness (PR_READINESS_READ_TOKEN), N/M remaining the new token
set, but refused or under 50 left ::warning:: with the reading GITHUB_TOKEN

So an expired token degrades to today's behaviour and cannot stop the required check. The token is read only through ${{ secrets.* }} into a step's env; only counts are printed.

What moves to the new pool: every gh_retry call in pr-readiness.yml (all are reads), and in the sweep the scan, the comment and status reads and the active-dispatch listing. pr-readiness.yml also ends with a pr-readiness: readiness pool rate limit -- N/M remaining line.

What stays on GITHUB_TOKEN, on purpose:

Stays Why
Every write (status, labels, the sweep's dispatch and republish) A write made with any other token starts workflow runs of its own. A label change would re-run every workflow that listens for labeled, the license gate among them.
Every collaborator-permission read: the one behind "changes requested", and both evaluator steps (pr_status.py --disposition-gate / --supersession-gate) whole The answer depends on what the asking token may see. _review_contract.author_write_verdict reads a refused permission lookup as "not a writer" and drops the record, so a token refused that read could publish a green the PR has not earned. The evaluator steps are not given the new token at all.
The sweep's reserve (RATE_RESERVE_PERCENT 20, DISPATCH_COST 25) The dispatch and the recompute's writes draw on the shared pool. Unchanged.

Maintainer steps for the secret

Nothing changes until these are done.

  1. With a bot or maintainer account, create a fine-grained personal access token: repository access "Public repositories (read-only)", no further permissions.
  2. gh secret set PR_READINESS_READ_TOKEN -R kirodotdev/KiroCrew and paste the token at the prompt (Actions secret, repository level).
  3. Open the next PR Readiness run and find readiness reads: token pool -- readiness (PR_READINESS_READ_TOKEN), N/M remaining.
  4. Replace the token before it expires. After expiry every run warns and reads with GITHUB_TOKEN.

What the token must be able to read, over REST and GraphQL, on this public repository: pull requests, reviews, issue comments, labels, commit statuses, check runs, workflow runs and file contents. It needs no write access and no access to collaborator permissions: no read that depends on them is made with it.

Limit to know: such a token has its owner's pool of 5,000 requests an hour, not 15,000. When it runs under 50 the run falls back to the shared pool. A GitHub App installation token would carry 15,000, but it expires hourly and cannot be stored as a secret; it needs a mint step this PR does not add.

How the 7-day count will be taken

Same method as the issue: job conclusion plus the log line. Seven days after merge, for codeql.yml and dependency-review.yml:

  1. gh run list -R kirodotdev/KiroCrew --workflow <file> --status failure --created '>=<merge time>' --limit 1000 --json databaseId
  2. for each run, the jobs with conclusion == "failure" (gh run view <id> --json jobs);
  3. for each such job, gh api repos/kirodotdev/KiroCrew/actions/jobs/<job id>/logs | grep -ci 'rate limit'; a job counts when the result is above 0. Both the old line (API rate limit exceeded) and the new ones (GitHub API rate limit is spent) match.

The two counts are compared with 255 (CodeQL) and 93 (license gate). Those baselines cover 14 days, so the like-for-like 7-day figures are about 128 and 47. The result goes on #39367, with the number of green jobs whose log carries waiting ...s for the reset (the jobs the wait saved).

Known limits and what is left

Item State
A pool that empties between the wait and the action's own request That attempt still fails. CodeQL then fails as before (no second upload is possible in one job); the license gate has its retry.
GraphQL bucket and secondary limits inside the two actions Not waited for: gh-rate-limit-wait reads only the REST (core) bucket.
Retry-After The wrapper sees gh's error text, not headers. A secondary limit (403 or 429) waits a fixed 60 s, as before.
Same-repo AI review lanes (Opus, GPT, Design, UX, First Principles) Not changed. They already install this helper's gh wrapper (EVENT_WRAPPED in test/test_scheduled_gh_rate_limit.py). Calls made inside claude-code-action are not covered.
Fork review lanes Not changed. They keep their own inline waits; #36090 tracks the lane that dies before its check.
Security Scope Review keeping a rate-limit result #32110, not changed.
Readiness evaluators and writes in an hour when the shared pool is at zero Still fail as today: both stay on GITHUB_TOKEN.
A 15,000-request readiness pool Needs a GitHub App and a token mint step.

Backwards compatibility

Compatible: with the secret unset both readiness workflows run as today, and the two gated jobs change only when the pool is under 100 requests or a review attempt failed.
Removes nothing: no check, trigger, permission or verdict is removed; analyze no longer uploads, and the upload-sarif step after it uploads the same file under the same category.

Tests

  • test/test_gh_rate_limit_wait.py (new, 23 tests). Runs the action's script against a stub gh, a fixed clock and a recording sleep; nothing sleeps, nothing reaches the network.
    • spent pool with a reset time → one sleep until the reset, waited; quota left → ready, no sleep; under and at the floor; reset past the budget → fails naming the rate limit and CI infrastructure; still spent after the reset; unreadable pool; one budget shared with the wrapper.
    • wrapper: a permission 403 with the pool spent is not waited for (one call, no sleep); a secondary-limit 429 waits and is retried once.
    • CodeQL: the upload is the last step, right after the wait, at the scan's commit and category; exactly one upload per job; no step carries continue-on-error or an if, so no path ends green without the upload.
    • License gate: the wait precedes the review; the pause is the reset wait or 60 s; two failed reviews with the pool spent are "could not run"; a license the retry reported is a license verdict even with the pool spent.
  • test/test_readiness_read_token.py (new, 23 tests). Runs the selection step and gh_retry against a stub gh that records the token each call carried: secret set → reads use it and the write does not; unset → today's token; refused, 49 left, 0 left → fallback with a warning; the token is never printed; the opt-out read; the evaluator steps are not given the token, and each one, run with the readiness pool selected and the token in its environment, still asks for a collaborator permission with the default token; the step and function are identical in both workflows; no write goes through gh_retry; the sweep's dispatch, republish and reserve read are not routed to the new token; the reserve values are unchanged.
  • test/test_scheduled_gh_rate_limit.py: both jobs join EVENT_WRAPPED (install after checkout, timeout above the wait budget).
  • test/test_dependency_license_gate_outage.py, test/pr_readiness_sweep_helpers.py: follow the added quota read and the added step.

Proof: PROVEN test/test_gh_rate_limit_wait.py::test_a_spent_pool_is_waited_out_once_until_its_reset, test/test_gh_rate_limit_wait.py::test_a_pool_with_quota_left_is_not_waited_for, test/test_gh_rate_limit_wait.py::test_a_pool_under_the_floor_is_waited_out, test/test_gh_rate_limit_wait.py::test_a_pool_at_the_floor_is_not_waited_for, test/test_gh_rate_limit_wait.py::test_a_reset_past_the_budget_fails_and_names_the_rate_limit (head bb14576)

Run locally: the two new files plus the 76 existing test files that reference the touched workflows, 4,442 passed, including test_flake_pattern_ratchet.py and test_gnu_only_shell_usage.py.

Manual verification

Not yet verified on a runner:

  • The split CodeQL upload. A complete Analyze run on this PR shows whether upload-sarif after analyze with upload: failure-only produces the same CodeQL result check. The first run was cancelled during the scan, before the upload.
  • The LICENSE_FINDING expression in the license gate reads the action's invalid-license-changes output with fromJSON. Its shape (forbidden, unresolved) is taken from the action's source at the pinned commit, not from a run.
  • The wait itself needs a spent pool; it will show as waiting ...s for the reset in a job log the first time one meets it.
  • The readiness token path needs the secret. Until then every run logs the shared line.

Seen on a runner: the Dependency License Gate passed on the first push of this PR with the helper installed and the wait step in place (run 44463165421).

No workflow linter (actionlint, zizmor) is wired in this repository or installed locally; the five YAML files were parsed with PyYAML and their step scripts run by the tests above.

Real-machine verification

N/A -- CI workflow only

Related Issues

Part of #39367. Related: #36090, #32110 (not changed here). Earlier fixes this builds on: #28477, #35349.

Sibling sweep

Swept: git grep -nE "uses: (github/codeql-action|actions/dependency-review-action|actions/github-script|peter-evans|actions/labeler|dorny)" -- .github/workflows
Sibling: .github/workflows/codeql.yml -- changed here
Sibling: .github/workflows/dependency-review.yml -- changed here
Sibling: .github/workflows/ci.yml:145 -- not affected -- dorny/paths-filter reads the diff from local git, pinned by test/test_paths_filter_local_git.py
Sibling: .github/workflows/build.yml:390 -- not affected -- dorny/paths-filter reads the diff from local git, same pin
Sibling: .github/workflows/macos-on-demand.yml:131 -- not affected -- dorny/paths-filter reads the diff from local git, same pin
Sibling: .github/workflows/claude-review.yml -- tracked in #39367

Pattern harvest

Rule candidate: review-prompt
Pattern: a job whose third-party action calls the GitHub API with the job token, with no rate-limit wait before it and no fail-closed handling after it

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

AI-assisted, opened by Claude Code

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision bb145760e6a16e87b739d6f6e301e4d671ed1d56 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of bb145760e6a16e87b739d6f6e301e4d671ed1d56 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the CodeQL and license-gate jobs now wait for the rate-limit reset where they used to fail, and the new read pool falls back to GITHUB_TOKEN when it cannot be used.

Notes

  • The Select the token pool for reads step and with_read_token are now copied in two workflows, and only a test keeps the copies identical. Moving them into .github/actions/gh-rate-limit (or a sibling composite action) would leave one place to edit.
  • Every write and every collaborator-permission read stays on GITHUB_TOKEN, but only by convention: a future gh_retry call that depends on what the asking token can see would quietly move to the PAT. Listing those reads by name, rather than relying on READINESS_READ_TOKEN="" at each call site, would keep that rule enforced.
  • The ## Backwards compatibility section sits past the truncation notice, so I could not check it; the patch removes no reader-visible surface I could find.

[DESIGN-REVIEWED] bb14576

Description read

[DESCRIPTION-READ] aca55a303e4597d1ec34a669a92f0d0f547d36c95f330621d01b82b21bd7ab79

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7083cb8596f1784af4fe6e72f8ee874510787c8b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The rate-limit wait for CodeQL and the license gate is backed by #39367's measurements. The new read-token pool for PR Readiness is weaker: it claims a harm nobody measured, and it does nothing until a maintainer creates a secret.

Notes

  • Item 7 — inherited: "a drained pool also delays the verdict" comes with no count. ci: one shared 15,000/h API pool causes about half of red jobs; give readiness its own budget and make jobs wait for the reset #39367's numbers (255 CodeQL, 93 license-gate red jobs) cover only CodeQL and the license gate. The item also does nothing until the secret exists ("Nothing changes until these are done").
    Subtraction: move PR_READINESS_READ_TOKEN out of this PR: both Select the token pool for reads steps, both with_read_token functions, the READINESS_READ_TOKEN env lines, the second rate-limit log line, test/test_readiness_read_token.py and its doc paragraph. Bring it back once readiness delay from a drained pool has a count.

What this change ships

Inventory (8 items) — 7 justified

Intent: A CodeQL or license-gate job that meets a spent shared API rate limit waits for the reset instead of failing, and readiness can read from its own pool. FIX, with one addition.

  1. The shared gh-rate-limit helper adds a gh-rate-limit-wait [floor] command, on the same budget as the wrapper — justified
  2. The wrapper's budget-exhausted error now says it is CI infrastructure — justified
  3. CodeQL uploads results in a separate step: wait, upload, then wait and re-upload only if the pool was spent — justified
  4. CodeQL job timeout goes from 60 to 75 minutes — justified
  5. The license gate waits before its first review, waits for the reset instead of a fixed 60 s before its retry, and reports a spent limit as "could not run" — justified
  6. License-gate job timeout goes from 10 to 40 minutes — justified
  7. New optional secret PR_READINESS_READ_TOKEN: readiness and sweep reads switch to it when it has quota, and fall back to GITHUB_TOKEN otherwise — inherited: harm unmeasured, dormant until the secret is set
  8. Readiness's last step logs a second line for the readiness pool — justified

[FIRST-PRINCIPLES-REVIEWED] 7083cb8

Description read

[DESCRIPTION-READ] 12a49cdf594c6e5b2be735ef1e53b1e97912d1d986fab20ed3e615b2f85892de

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ⚠️ review incomplete

GPT 6.1 did not produce a complete verdict for 7083cb8596f1784af4fe6e72f8ee874510787c8b; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 6.1 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7083cb8596f1784af4fe6e72f8ee874510787c8b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
Akim Akimov added 2 commits November 19, 2026 05:46
CodeQL and the Dependency License Gate make their API calls inside a
third-party action, with the job token that shares one hourly REST pool
with every workflow. When the pool is spent the action fails and the
job is red for a reason that says nothing about the diff.

The shared gh-rate-limit action now also installs gh-rate-limit-wait,
the same bounded wait as a command. CodeQL uploads its results in a
separate step right after a wait, once per job. The license gate waits
before each review attempt, and a license the retry itself reported
stays a license verdict. Neither job can pass without its action
having run.

Part of #39367
PR Readiness and the readiness sweep read the lanes they aggregate
through the pool every lane spends. With the repository secret
PR_READINESS_READ_TOKEN set, both make their reads with that token and
log which pool they used and what is left of it. Unset, refused or
nearly spent, they read with GITHUB_TOKEN as before.

Writes, every collaborator-permission read (the evaluators included)
and the sweep reserve stay on GITHUB_TOKEN: a write made with another
token starts workflow runs of its own, and a token refused the
permission read would drop a block or a disposition record.

Part of #39367
@Zedmor
Zedmor force-pushed the fix/ci-rate-limit-wait branch from 7083cb8 to bb14576 Compare November 19, 2026 05:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: checking Automated validation is still running

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant