Skip to content

ci(runners): send fifteen fleet fallbacks to the larger-runner group - #26809

Merged
bolichen97 merged 1 commit into
mainfrom
ci/fleet-else-group-fallback
Oct 31, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/fleet-else-group-fallback

Conversation

@tim-finnigan

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When CI turns a pull request away from the CodeBuild fleet, its jobs land on our own larger-runner group instead of the runner pool the whole company shares.

Fifteen jobs ask the CodeBuild fleet for a runner first — fourteen pull-request lanes, plus ci.yml's cancel-superseded-run, whose if: makes it a push to main. The fleet admits only a same-repository run, from an allowlisted actor, on a narrow set of events. Every run it turns away goes to ubuntu-latest today, the standard pool whose queue is shared enterprise-wide. #25482, #25645 and #26170 already did this for other jobs; these fifteen are what is left of the class.

Why it matters

CI for these lanes waits in a queue we cannot do anything about, and it happens far more often than "forks only" suggests. The fleet admits the pull_request actions opened and synchronize only, while these workflows also subscribe to reopened, edited, labeled and unlabeled — so every bot label and every description edit on an ordinary same-repository pull request misses the fleet and lands on the shared pool today.

Measured over a complete 6-hour window from the runner each job actually got: 450 jobs would move. On the two workflows carrying most of them, 317 of 357 (88%) were same-repository runs on a non-admitted action, and only 40 (11%) were fork pull requests.

Not a goal

  • main-ratchet-audit.yml's four jobs. They stay hosted, but not for the reason the doc previously gave. The merge-capacity argument does not single them out: the queue polls ci.yml, fast-gate.yml and build.yml as well, and this diff moves their merge-group leg for any unlisted queuing actor, while several ci.yml jobs reach the group unconditionally already. The doc now says that plainly. What actually keeps the ratchet audit out is narrower — its hosted tail is pinned by a dedicated assertion rather than derived, so moving it means deleting test_the_ratchet_audit_route_stays_off_the_larger_runner_group on purpose, which deserves its own change and its own evidence. It also measured zero moving jobs in the same window.
  • Buying a bigger machine. kirocrew-linux-4-core is 4 vCPU and 16 GB, the same as ubuntu-latest. Nothing gets faster; the change is about queue position.
  • Deciding the spend. See Capacity below. The numbers are here; the call is the owner's.

What changed (motivation → approach → change)

A turned-away job now waits in our own queue instead of the company's. Nothing runs faster — kirocrew-linux-4-core is 4 vCPU and 16 GB, the same machine ubuntu-latest gives you. What changes is who is ahead of you in line: the standard pool's queue is shared enterprise-wide, and this group's is ours.

Each of these fifteen jobs picks its runner with one runs-on expression: ask the fleet, otherwise fall back. The fallback was a single hosted image. It is now two branches, and which one a job takes depends only on where the run executes.

the run before after
Fleet accepts it (same-repo, listed actor) CodeBuild CodeBuild, unchanged
Fleet turns it away, running in kirodotdev/KiroCrew ubuntu-latest kirocrew-linux-4-core
Fleet turns it away, running in someone's fork ubuntu-latest ubuntu-latest, unchanged

The third row is why this is not a one-word swap. A personal fork cannot reach an organization runner group at all, so a job there asking for that label would sit queued until GitHub cancelled it, which is worse than the shared pool. The repository test in the fallback is what keeps a fork on the image it can actually reach.

This shape already exists in the repository as _GROUP_THEN_HOSTED_TAIL, used by the five non-pull-request sites that moved earlier. Two canonical constants now reuse it instead of ending in a bare hosted literal, so all fifteen lines stay byte-identical to one definition rather than drifting apart.

docs/ci/ci-and-reviews.md carried the old boundary in writing: it named five fleet-fallback sites and explained that the batch was held to non-pull-request lanes precisely so it would add no pull-request-time occupancy. This diff makes that false, so the doc now names all twenty sites, says plainly what the fifteen cost and what they buy, and keeps the rollback recipe and the main-ratchet-audit.yml exclusion intact.

Capacity. This adds pull-request-time occupancy to a capped group the review lanes already compete for, which is what docs/ci/ci-and-reviews.md previously said the first batch was scoped to avoid. Over the measured window the added concurrency peaked at 34 job-slots, p95 17, median 4, and the moving jobs billed about 878 minutes in those 6 hours.

Two caveats on that number, both load-bearing. The group's live maximum_runners is 200; this page said 100 in three places, so every capacity argument written against it was wrong by a factor of two, and the diff corrects them. And the group's current peak occupancy is not measured here — the number that decides this is current peak plus 34 against the cap. That is not the single API query it sounds like (there is no org-wide jobs endpoint; it means walking jobs for every run of every on-group workflow in a window), so it is called out rather than quietly skipped. These lanes fire on the same pull-request traffic as the lanes already there, so the peaks coincide rather than averaging out.

Rollback is per line and does not disturb the fleet route: drop the middle branch so the tail reads || 'ubuntu-latest' }} again, and replace _GROUP_THEN_HOSTED_TAIL in the constant with that literal rather than deleting it.

Backwards compatibility

Compatible: no run loses a destination. A run that reached the fleet still reaches it, and a run executing outside this repository still gets the same hosted image. Only the upstream non-fleet branch changes, and it changes to a runner that exists.

Removes nothing: the diff adds a branch to an expression and widens two test expectations; no key, path, job, or capability is removed or renamed.

Tests

test_ci_fleet_routing_expression_parity.py pins every one of the fifteen runs-on strings against the two canonical constants, so a line that drifts from the shared definition fails. test_ci_additional_fleet_routes.py evaluates each expression across the actor, event and repository truth table and reads the expected non-fleet value off the policy itself, so it covered the move without edits.

test_ci_windows_canary.py::test_all_actual_routes_require_repository_event_and_admitted_actor had pinned the non-fleet destination for ci.yml's changes and await-fast-gate to the single literal ubuntu-latest. That expectation is now two-valued by repository. The test's own contract — fleet admission requires repository, event and an admitted actor — is untouched, and the assertion is strictly more specific than before: it now distinguishes the upstream destination from the fork one where previously it asserted one value for both.

main-ratchet-audit.yml is asserted untouched: test_the_ratchet_audit_route_stays_off_the_larger_runner_group still passes unmodified.

Manual verification

N/A — routing is evaluated by the truth-table tests above, which is stronger than running one pull request and seeing where it landed.

Related Issues

Closes #26802

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@tim-finnigan
tim-finnigan requested review from a team and KiroCrewCaptain as code owners October 31, 2026 19:35
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b26ad1bedc24664667718eddf68b8125cc90453d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b26ad1bedc24664667718eddf68b8125cc90453d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design problems found. Every change is a one-line revert, the fork tail keeps forks on a runner they can reach, and the routing reuses _GROUP_THEN_HOSTED_TAIL that earlier sites already use.

Notes

  • The group's current peak occupancy is still unmeasured, and the added PR-time concurrency (up to 34 slots) peaks alongside the review lanes already on the group. Before merging, measure the current peak plus 34 against the cap of 200. If merge holds start getting longer, roll back pr-policy/code-review first.

[DESIGN-REVIEWED] b26ad1b

Description read

[DESCRIPTION-READ] 034468baa555979ab77886b16a48bd159eb01d9df514582c175945a808c056ac

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b26ad1bedc24664667718eddf68b8125cc90453d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: this applies the existing _GROUP_THEN_HOSTED_TAIL to the last fifteen fleet-fallback sites. The only fallbacks left on bare hosted are the four main-ratchet-audit.yml sites the description excludes.

What this change ships

Inventory (6 items) — 6 justified

Intent: when the CodeBuild fleet turns away a run in this repository, it waits in the org's own larger-runner queue instead of the enterprise-shared pool. This is an ADDITION: a CI runner move, and not a product-shape change.

  1. Fourteen pull-request lanes that the fleet turns away now land on kirocrew-linux-4-core instead of ubuntu-latest — justified
  2. A rejected cancel-superseded-run on push to main lands on the group too — justified
  3. A fork running these workflows still gets ubuntu-latest — justified
  4. The two canonical test constants now reuse _GROUP_THEN_HOSTED_TAIL instead of a hosted literal — justified
  5. The doc's stated cap goes from 100 to 200, with a pointer to the live value — justified
  6. The doc replaces its "five kept to a non-PR batch" and ratchet-audit reasons with the measured cost, and says what still keeps the ratchet audit hosted — justified

[FIRST-PRINCIPLES-REVIEWED] b26ad1b

Description read

[DESCRIPTION-READ] 929030bfd0d53ef6f350addb134487a4642e52392b7ea392ed6f6aa953607c14

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b26ad1bedc24664667718eddf68b8125cc90453d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b26ad1b

Verdict parsed from the review's SHA-scoped output markers for commit b26ad1bedc24664667718eddf68b8125cc90453d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b26ad1bedc24664667718eddf68b8125cc90453d false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b26ad1bedc24664667718eddf68b8125cc90453d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b26ad1b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b26ad1bedc24664667718eddf68b8125cc90453d false-positive: <reason, 20 to 500 characters>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head b26ad1b.

I read every workflow hunk. All 15 changes edit only the last fallback of a runs-on: expression: a run the CodeBuild fleet turns away now gets kirocrew-linux-4-core when the run is in this repository, and ubuntu-latest otherwise. The fleet branch, every if:, the triggers and the job steps are unchanged. That label is already used by other jobs on main, and this finishes the class started by #25482, #25645 and #26170.

Checked against the workflow safe-read rules: no secret or token use is added, no permissions: change, no new pull_request_target or workflow_run trigger, no download-and-execute, and no checkout plus run of PR code in a credentialed job. Fork pull requests still run on pull_request with a read-only token, so moving them to the larger-runner group changes only where they queue and what they cost, not what they can reach.

0 checks are red, and the Design, First Principles, Opus and GPT lanes pass. Only two backend shards and the macOS smoke job are still running, and auto-merge waits for PR Readiness.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 31, 2026
@bolichen97
bolichen97 merged commit a7d247d into main Oct 31, 2026
96 checks passed
@bolichen97
bolichen97 deleted the ci/fleet-else-group-fallback branch October 31, 2026 20:22
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: route the fifteen PR-time fleet fallbacks to the kirocrew-runners group

2 participants