Repository navigation
ci: build a nightly per-test map on main and a shadow test selector - #39356
KiroCrewCaptain wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Shadow-only tooling with a read-only token. Nothing reads the map and nothing is taken away, so nothing blocks. Notes
[DESIGN-REVIEWED] 9c08f55 Description read[DESCRIPTION-READ] b9fa98f8fecf4c9a5be9363d8cf054ff3f2e03d73b2eed3bbd82ecc627b9e362 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: everything here is shadow-only tooling that can be deleted later, and it is tied to #39336. One note: nothing reads the nightly artifact yet. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: find out whether running only the tests a diff affects would be safe and cheap, before anyone turns it on. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 9c08f55 Description read[DESCRIPTION-READ] 3a41e95dafeb28180610421e24afdf7cb489ff5117e9af325097dd454d207121 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
484f0a9 to
e07b3f0
Compare
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
e07b3f0 to
8b72299
Compare
|
|
8b72299 to
79344c4
Compare
|
79344c4 to
ee704e5
Compare
|
|
ee704e5 to
9c08f55
Compare
|
|
This review found no blocking issue on this head. The PR waits for a maintainer approval before it can merge. What the review checked on 9c08f55:
Done well: the plugin test runs a real child pytest. It checks both sides of the recorder's main limit: a thread started inside a test is recorded, and a module body that ran only at collection is not linked. Optional:
|
Problem / Motivation
Goal: Measure, without changing any test a PR runs, whether running only the backend tests a diff affects would be safe and how much it would save.
Every backend source change runs the whole Linux suite. A static import graph cannot narrow it: lazy imports tie about 64% of
src/into one cycle, so it picks the whole suite for 87 of 89 source PRs. A runtime map (which test ran which file) could narrow it, but nobody has checked whether it would miss a failing test, and no full-suite run onmainexists that a landing cannot cancel.Why it matters
Turning on test selection without evidence risks a red change landing on
main. This PR builds the evidence first: a map rebuilt every night onmain, a selector that only prints what it would run, and a replay over past PRs and red pushes.Not a goal
ci.ymlor any other lane. Nothing reads the map.What changed (motivation → approach → change)
A new nightly workflow records, per test, which repository files it ran, and builds one small map: file -> test files. A new script reads a diff and that map and prints the test files it would run. Anything it cannot map falls back to the full suite. Nothing in CI uses it yet.
.github/workflows/test-map.ymlmain(09:30 UTC) plus manual. Runsci.yml's 8 Linux file shards with the recorder, no coverage. Uploadstest-map.json.gz. Read-only token, never cancelled.scripts/ci_test_file_map.py-pin that workflow. Python 3.12sys.monitoringPY_START: the first start of each code object in a test records its file. Kept in memory, written once at session end. Does nothing below 3.12.scripts/ci_test_map.pybuildmerges the parts.selectprints the test files for a diff, or exits 3 with the reason.scripts/ci_test_map_shadow.pyghreads only..github/workflows/scheduled-failure-watch.ymltest_scheduled_failure_watch.pyrequires.Why not coverage contexts (
--cov-context=test): coverage writes its SQLite data at every context switch, inside the test. Tests that patchsqlite3.connectthen break coverage itself. A local full run gave 133 failed, 49,844 errors and an xdist INTERNALERROR, and the per-test data reached 35 GB.Full-suite fall-backs: an added, deleted, renamed or copied file; any
conftest.py; a non-.pyfile, or a.pyfile outsidesrc/andtest/; a.pyfile no test ran; an empty guard tier. Otherwise it selects the changed test files, the test files the map links to the changed files, and always the guard tier (leaf_test_scope.corpus_gates, the 910 test files that read the test tree as data).Backwards compatibility
Compatible: the diff adds a workflow and scripts nobody calls, plus one name in the failure watcher's list.
Removes nothing: no lane, setting or test changes.
Tests
test/test_ci_test_map.py:crashed-*marker, and the workflow fails that shard instead of publishing a map with a gap.corpus_gates.Backend Tests (3.12, N)jobs, reads the jobs of the run's own attempt, drops a run whose red shard log cannot be read, and counts misses with and without the guard tier (a changed test file in the tier still counts as the map's own pick).scheduleandworkflow_dispatch,contents: read,cancel-in-progress: false, no credential persisted, and is the only workflow that loads the plugin.Determinism: 26 passed at
-n 0and at-n 4, three runs each, and once in a shuffled order (shuf --random-source=<(yes 7)).Proof: not provable -- other: new shadow-only tooling, nothing on main to revert
Manual verification
Shadow report (the evidence this PR exists for)
One map, built locally at
eecf4c11be(full Linux suite, 48 workers), judged every sample.ci.ymlcoverage run--cov-context=testrun took 3,793 s at 43 workerstest/test_kas_policy_hidden_readiness.py, red on 3 different heads (broken main or flake).pyfile (mostly docs), 6 change the file set, 1 touches a file no test ranUnder 10% is not reachable while the guard tier always runs: it alone is 16.7% of node ids. The map alone picked a median of 0.4% on the reduced PRs. The largest win left is mapping docs and data files, which caused 37 of the 44 fall-backs.
Full Linux backend suite run locally twice on this box (48 workers): once with the recorder, once with
ci.yml's coverage flags. Map built, thenci_test_map_shadow.py --merged 60 --red 200run against GitHub. The workflow itself first runs after merge, byworkflow_dispatch.Real-machine verification
Machine: Amazon Linux 2023 dev desktop, x86_64, 128 vCPU / 247 GiB, own computer.
Before (main eecf4c1):
pytest -n 48 -m "not ipv6_required and not real_venv" --cov=...-> 148 failed, 290,336 passed in 2,546 sAfter (this branch):
pytest -n 48 -m "not ipv6_required and not real_venv" --no-cov -p scripts.ci_test_file_map --test-map-dir=...-> 147 failed, 290,312 passed in 1,890 s; 48 parts, map builtCleanup: own computer
Related Issues
Closes #39336
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits