Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1361,6 +1361,25 @@ jobs:
echo "reduced=true" >> "$GITHUB_OUTPUT"
echo "Reduced scope: $(wc -l < "$TARGETS_FILE") cross-surface backend file(s)."

# Experiment (#39368): each xdist worker collects the whole shard and
# rewrites the asserts of every test file, because a fresh runner has no
# cached pyc. This writes those pycs once, one process per core, without
# importing a test module, conftest or kiro_crew, into the cache root the
# rootdir conftest points pytest at. Full suite only: it is the arm that
# takes --file-shard-pins. Same shell as the test step, so the same user
# and home, which is what the cache root is derived from. Best effort:
# the script exits 0 whatever happens, and a file it did not cache is
# rewritten by pytest. It prints its own elapsed seconds.
- name: Prewarm assertion rewrites (shard ${{ matrix.group }}/${{ env.SHARD_COUNT }})
if: ${{ steps.scope.outputs.reduced == 'false' }}
shell: /usr/local/bin/ci-shell {0}
continue-on-error: true
timeout-minutes: 5
run: |
python -m scripts.ci_rewrite_prewarm \
--file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} \
--file-shard-pins scripts/ci_file_shard_pins.txt

- name: Run tests (shard ${{ matrix.group }}/${{ env.SHARD_COUNT }})
shell: /usr/local/bin/ci-shell {0}
# scripts.ci_file_shards assigns each file to one shard before pytest
Expand Down Expand Up @@ -1639,9 +1658,11 @@ jobs:
# full (main pushes 44376034204, 44365593626): 26.6 min wall, 183-191 job-min
# reduced, 1,342 files (run 44426610212): 8.7 min wall, 60 job-min
# leaf, 1 file + 909 gates (run 44427119854): 13.8 min wall, 64 job-min
# No pytest prewarm: a serial `--collect-only` of one shard costs 104-115 s
# and saves 42-75 s of worker collection, so it loses on both OSes, with
# and without --cov (prewarm probe in run 44426610212).
# No `--collect-only` prewarm: a serial one costs 104-115 s per shard and
# saves 42-75 s of worker collection, so it loses on both OSes, with and
# without --cov (prewarm probe in run 44426610212). The step before
# "Run tests" is a different form: it only rewrites asserts, in parallel,
# and imports nothing.
- name: Select test scope
id: scope
shell: bash
Expand Down Expand Up @@ -1713,6 +1734,18 @@ jobs:
echo "reduced=true" >> "$GITHUB_OUTPUT"
echo "Reduced scope: $(wc -l < "$TARGETS_FILE") cross-surface backend file(s)."

# Same experiment and contract as backend-test's prewarm step (#39368),
# with the Windows pin file and the test step's shell.
- name: Prewarm assertion rewrites (Windows shard ${{ matrix.group }}/${{ env.SHARD_COUNT }})
if: ${{ steps.scope.outputs.reduced == 'false' }}
shell: bash
continue-on-error: true
timeout-minutes: 5
run: |
python -m scripts.ci_rewrite_prewarm \
--file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} \
--file-shard-pins scripts/ci_file_shard_pins_windows.txt

- name: Run tests (Windows shard ${{ matrix.group }}/${{ env.SHARD_COUNT }})
# bash (Git Bash on windows-latest), not pwsh: the invocation shares
# the POSIX job's line-continuation and env-interpolation syntax.
Expand Down
16 changes: 16 additions & 0 deletions docs/ci/ci-and-reviews.md
Original file line number Diff line number Diff line change
Expand Up @@ -884,6 +884,22 @@ coverage or a repair of the crashing test. Periodic console records and best-eff
JSONL artifacts preserve evidence; job-level cancellation can prevent uploads.
Healthy selection, per-test timeouts and coverage selectors are unchanged.

On the full suite, the step before `Run tests` in both jobs runs
`scripts/ci_rewrite_prewarm.py` (an experiment, #39368). Every xdist worker
collects the whole shard and, on a fresh runner, rewrites the asserts of each
test file because no cached pyc exists. The step writes those pycs once, one
process per core, with pytest's own rewrite functions, under the cache root the
rootdir conftest sets (`sys.pycache_prefix`), for the test files the shard owns
and every `conftest.py`. It parses and compiles source and imports no test
module, conftest or `kiro_crew`, which is what separates it from the serial
`--collect-only` prewarm that measured slower. It does not choose what a shard
collects: pytest's discovery and the plugin still do. It is best effort: it
exits 0 whatever happens, a file it did not cache is rewritten by pytest, and
the reduced and leaf scopes skip it. Bytecode is never restored from
`actions/cache` or committed. The step prints its file count and elapsed
seconds. `test/test_ci_rewrite_prewarm.py` proves pytest reads the prewarmed pyc
and pins the cache root against the conftest's.

The root conftest's import-time telemetry guard fails the offending module's
collection report on every worker, so pytest/xdist fails the job even when the
shard does not own `test_host_isolation_floor.py`. The process-wide telemetry-off
Expand Down
9 changes: 9 additions & 0 deletions docs/system-specs/common/testing-conventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -6993,6 +6993,15 @@ Linux pins cut the slowest Windows shard by a median 12% of its summed test
seconds over six held-out green runs, the Windows pins by 20%. macOS, release
and the reduced and leaf scopes keep pure hash ownership.

Before the Linux and Windows full-suite test step, `scripts/ci_rewrite_prewarm.py`
writes pytest's assertion-rewrite pyc for the shard's test files (same hash and
pin file) and every `conftest.py`, in parallel and without importing them, under
the `sys.pycache_prefix` the rootdir conftest sets. It changes where a worker's
collection time goes, not which files the shard collects, and never fails the
job. `test/test_ci_rewrite_prewarm.py` owns the proof that pytest reads those
pycs; a change to the cache root in `test/conftest_floor_configure.py` must keep
`ci_rewrite_prewarm.pycache_prefix` equal to it.

macOS assigns whole files like the other platforms, in eight shards. The shard
count is the knob to reach for when the suite grows, not the 60-minute cap,
because that cap is a spend guard as much as a hang guard on a lane billed at ten
Expand Down
Loading
Loading