Skip to content

ci: build a nightly per-test map on main and a shadow test selector - #39356

Open
KiroCrewCaptain wants to merge 1 commit into
mainfrom
ci/coverage-map-shadow
Open

KiroCrewCaptain wants to merge 1 commit into
mainfrom
ci/coverage-map-shadow

Conversation

@KiroCrewCaptain

@KiroCrewCaptain KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem / Motivation

Goal: Measure, without changing any test a PR runs, whether running only the backend tests a diff affects would be safe and how much it would save.

Every backend source change runs the whole Linux suite. A static import graph cannot narrow it: lazy imports tie about 64% of src/ into one cycle, so it picks the whole suite for 87 of 89 source PRs. A runtime map (which test ran which file) could narrow it, but nobody has checked whether it would miss a failing test, and no full-suite run on main exists that a landing cannot cancel.

Why it matters

Turning on test selection without evidence risks a red change landing on main. This PR builds the evidence first: a map rebuilt every night on main, a selector that only prints what it would run, and a replay over past PRs and red pushes.

Not a goal

  • Skipping or selecting any test in ci.yml or any other lane. Nothing reads the map.
  • Changing setup.cfg, pyproject.toml or the coverage settings.
  • Mapping non-Python files (docs, baselines, workflows). Those fall back to the full suite.

What changed (motivation → approach → change)

A new nightly workflow records, per test, which repository files it ran, and builds one small map: file -> test files. A new script reads a diff and that map and prints the test files it would run. Anything it cannot map falls back to the full suite. Nothing in CI uses it yet.

Part Job
.github/workflows/test-map.yml Nightly on main (09:30 UTC) plus manual. Runs ci.yml's 8 Linux file shards with the recorder, no coverage. Uploads test-map.json.gz. Read-only token, never cancelled.
scripts/ci_test_file_map.py pytest plugin, loaded only by -p in that workflow. Python 3.12 sys.monitoring PY_START: the first start of each code object in a test records its file. Kept in memory, written once at session end. Does nothing below 3.12.
scripts/ci_test_map.py build merges the parts. select prints the test files for a diff, or exits 3 with the reason.
scripts/ci_test_map_shadow.py Replays the selector over merged PRs and red PR pushes, with gh reads only.
.github/workflows/scheduled-failure-watch.yml One line: watch the new scheduled workflow, as test_scheduled_failure_watch.py requires.

Why not coverage contexts (--cov-context=test): coverage writes its SQLite data at every context switch, inside the test. Tests that patch sqlite3.connect then break coverage itself. A local full run gave 133 failed, 49,844 errors and an xdist INTERNALERROR, and the per-test data reached 35 GB.

Full-suite fall-backs: an added, deleted, renamed or copied file; any conftest.py; a non-.py file, or a .py file outside src/ and test/; a .py file no test ran; an empty guard tier. Otherwise it selects the changed test files, the test files the map links to the changed files, and always the guard tier (leaf_test_scope.corpus_gates, the 910 test files that read the test tree as data).

Backwards compatibility

Compatible: the diff adds a workflow and scripts nobody calls, plus one name in the failure watcher's list.

Removes nothing: no lane, setting or test changes.

Tests

test/test_ci_test_map.py:

  • The plugin, in a real pytest session, links each test file to the files its tests ran, including a thread's, and not to files only collection ran.
  • A worker that dies leaves a crashed-* marker, and the workflow fails that shard instead of publishing a map with a gap.
  • Parts merge into the map; an empty parts directory and a wrong schema are refused.
  • Every fall-back rule gives the full suite; a mapped edit gives its tests plus the guard tier; the guard tier is corpus_gates.
  • The shadow replay reads only failed Linux Backend Tests (3.12, N) jobs, reads the jobs of the run's own attempt, drops a run whose red shard log cannot be read, and counts misses with and without the guard tier (a changed test file in the tier still counts as the map's own pick).
  • The workflow has only schedule and workflow_dispatch, contents: read, cancel-in-progress: false, no credential persisted, and is the only workflow that loads the plugin.

Determinism: 26 passed at -n 0 and at -n 4, three runs each, and once in a shuffled order (shuf --random-source=<(yes 7)).

Proof: not provable -- other: new shadow-only tooling, nothing on main to revert

Manual verification

Shadow report (the evidence this PR exists for)

One map, built locally at eecf4c11be (full Linux suite, 48 workers), judged every sample.

Measure Result
Map size 6.4 MB JSON, 1.7 MB gzipped. 12,779 files, 8,118 test files, 292,295 node ids
Recorder run vs ci.yml coverage run 1,890 s vs 2,546 s, same 48 workers: the recorder run is 26% faster. The failed --cov-context=test run took 3,793 s at 43 workers
Same pass/fail Yes. 147 failed on both (this box's sandbox and path tests); one more on the coverage run read a file renamed during that run and passes alone
Merged PRs (last 60) Selected share: median 100%, mean 81.0%. 16 reduced, median 17.0% (guard tier alone is 16.7%)
Red PR pushes (182 with failing Linux tests, every red shard log read) Every failing test file selected on 179 (98%)
Red pushes the selector reduced 15. Caught 12 (3 from the map alone, 9 by the guard tier). The 3 misses are one file, test/test_kas_policy_hidden_readiness.py, red on 3 different heads (broken main or flake)
Why merged PRs fell back 37 touch a non-.py file (mostly docs), 6 change the file set, 1 touches a file no test ran

Under 10% is not reachable while the guard tier always runs: it alone is 16.7% of node ids. The map alone picked a median of 0.4% on the reduced PRs. The largest win left is mapping docs and data files, which caused 37 of the 44 fall-backs.

Full Linux backend suite run locally twice on this box (48 workers): once with the recorder, once with ci.yml's coverage flags. Map built, then ci_test_map_shadow.py --merged 60 --red 200 run against GitHub. The workflow itself first runs after merge, by workflow_dispatch.

Real-machine verification

Machine: Amazon Linux 2023 dev desktop, x86_64, 128 vCPU / 247 GiB, own computer.
Before (main eecf4c1): pytest -n 48 -m "not ipv6_required and not real_venv" --cov=... -> 148 failed, 290,336 passed in 2,546 s
After (this branch): pytest -n 48 -m "not ipv6_required and not real_venv" --no-cov -p scripts.ci_test_file_map --test-map-dir=... -> 147 failed, 290,312 passed in 1,890 s; 48 parts, map built
Cleanup: own computer

Related Issues

Closes #39336

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@KiroCrewCaptain
KiroCrewCaptain requested a review from a team as a code owner November 19, 2026 02:16
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9c08f551219385e144a959e07628e8a8fce8419b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9c08f551219385e144a959e07628e8a8fce8419b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Shadow-only tooling with a read-only token. Nothing reads the map and nothing is taken away, so nothing blocks.

Notes

  • The map misses module bodies that ran at collection time and anything a child process ran. The shadow report already counts misses, but before the selector gates anything it should report misses of these two kinds separately, so the gap is measured and not assumed.
  • Most fall-backs come from non-.py files (37 of 44 PRs that fell back). Mapping those is the next win, and it belongs in its own follow-up PR.

[DESIGN-REVIEWED] 9c08f55

Description read

[DESCRIPTION-READ] b9fa98f8fecf4c9a5be9363d8cf054ff3f2e03d73b2eed3bbd82ecc627b9e362

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9c08f551219385e144a959e07628e8a8fce8419b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: everything here is shadow-only tooling that can be deleted later, and it is tied to #39336. One note: nothing reads the nightly artifact yet.

Notes

  • Item 1 — oversized: every night the workflow runs the 8-shard full suite to upload test-map, and no workflow or script downloads that artifact (grepped test-map and ci_test_map in .github/: only test-map.yml itself matches). ci_test_map_shadow.py takes a local --map path. The description's evidence came from a map built locally. Subtraction: drop the schedule: trigger and keep workflow_dispatch until a reader exists.

What this change ships

Inventory (7 items) — 6 justified

Intent: find out whether running only the tests a diff affects would be safe and cheap, before anyone turns it on. This is an ADDITION.

  1. Every night (and on demand), main runs the full Linux suite with a recorder and publishes a 30-day test-map artifact — oversized: no reader yet
  2. A pytest plugin, loaded only by that workflow, records which files each test ran — justified
  3. ci_test_map.py build merges the shard parts into one map — justified
  4. ci_test_map.py select prints the test files a diff would run, or falls back to the full suite — justified
  5. ci_test_map_shadow.py replays the selector over past merged PRs and red PR pushes — justified
  6. The scheduled-failure watcher now watches "Test Map" — justified
  7. A shard that loses an xdist worker now fails instead of publishing a map with a gap — justified

[FIRST-PRINCIPLES-REVIEWED] 9c08f55

Description read

[DESCRIPTION-READ] 3a41e95dafeb28180610421e24afdf7cb489ff5117e9af325097dd454d207121

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9c08f551219385e144a959e07628e8a8fce8419b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9c08f55

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9c08f551219385e144a959e07628e8a8fce8419b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9c08f551219385e144a959e07628e8a8fce8419b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9c08f55

Verdict parsed from the review's SHA-scoped output markers for commit 9c08f551219385e144a959e07628e8a8fce8419b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9c08f551219385e144a959e07628e8a8fce8419b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author
  • Incomplete failure logs become successful safety samples

fixed in 8b72299: _red_row returns None when any failed Linux backend shard log cannot be read, so that run is neither counted nor cached, and a later pass retries it.
Test: test_shadow_red_row_reads_only_failed_linux_backend_jobs now asserts None for an unreadable log. The red-push numbers in the body were recomputed from a fresh cache.

@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author
  • Map-only metrics discard independently selected guard tests

fixed in 8b72299: judge runs select a second time with a stand-in tier and drops only the stand-in, so a changed test file that is also in the guard tier stays in the map-only pick.
Test: test_shadow_judge_counts_a_missed_failing_file covers an edited guard-tier file (missed_by_map == []).

@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author
  • Even-sized samples report an incorrect median

fixed in 79344c4: summarize uses statistics.median, which averages the middle pair; the hand-rolled _median is gone.
Test: test_shadow_summary_states_both_invariants asserts a median of 62.5% for shares [0.25, 1.0]. The body's medians were recomputed from the same rows.

@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author
  • Cached failures are reused across different run attempts

fixed in ee704e5: the cache key is run-<id>-<run_attempt>.json, and the jobs are read from actions/runs/<id>/attempts/<attempt>/jobs, so a rerun is a fresh sample.
Test: test_shadow_red_row_reads_only_failed_linux_backend_jobs asserts the attempt-scoped jobs URL.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator Author
  • Worker crashes publish incomplete maps as successful shards

fixed in ee704e5: the plugin's controller-side pytest_testnodedown writes a crashed-<worker> marker when a worker dies, and the workflow step fails the shard when any marker exists, so a map with a lost worker is never published. Ordinary test failures still pass through.
Test: test_a_worker_crash_leaves_a_marker_instead_of_a_silent_gap kills a worker with os._exit under -n 2.

@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author
  • Recorder tests independently cold-start interpreter trees

fixed in 9c08f55: the crash test no longer starts a child pytest. It calls the plugin's pytest_testnodedown hook directly with stub nodes, as the xdist controller calls it. One test in the module starts one child pytest (no -n), and the module carries pytestmark = pytest.mark.xdist_group(name="ci_test_map").

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator Author

This review found no blocking issue on this head. The PR waits for a maintainer approval before it can merge.

What the review checked on 9c08f55:

  • The workflow runs only on schedule and manual dispatch. It has a read-only token, no secrets and persist-credentials: false, and no untrusted input reaches a run step.
  • It runs the same shards, pins, timeout and marker filter as the backend lanes in ci.yml. No existing lane, gate or test changes, and nothing reads the map to skip a test.
  • In the selector, every case it cannot prove falls back to the full suite. Those cases are: added, renamed or deleted files, conftest, non-.py files, unmapped files, and an empty guard tier. A crashed xdist worker fails the shard.
  • The new tests are deterministic and leave no side effects. The child pytest runs in tmp_path with a bounded timeout and no cache provider.

Done well: the plugin test runs a real child pytest. It checks both sides of the recorder's main limit: a thread started inside a test is recorded, and a module body that ran only at collection is not linked.

Optional:

  • .github/workflows/test-map.yml:139: the build step pipes into tee under the default bash -e, with no pipefail. If the builder exits 1 with "the parts recorded no executed file", the step still passes and an empty map is uploaded. Add set -o pipefail, or set shell: bash on the step.
  • .github/workflows/test-map.yml:22: the nightly schedule runs a full Linux suite every day, and nothing reads the artifact yet. A dispatch-only workflow would still meet the Goal. You could add the schedule and the watcher entry in the PR that adds the first reader, or reply to the First Principles note saying why it is needed now.
  • Two load-bearing lines have no test that fails when they are removed. One is restart_events() in scripts/ci_test_file_map.py:140: a fixture where a second test calls a function an earlier test already ran would pin it. The other is the conftest and outside-src/test fall-back rules in scripts/ci_test_map.py:153-156: a map that links those paths, plus an assertion on the reason string, would pin them. Both matter before any lane acts on the selector.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: no measured, safe way to run only the backend tests a diff affects

1 participant