Mutation testing: diff against a configured branch, cover only the changed files - #6664
Merged
Merged
Conversation
| echo 'files<<EOF' | ||
| echo "$files" | ||
| echo 'EOF' | ||
| } >> "$GITHUB_OUTPUT" |
ondrejmirtes
force-pushed
the
mutation-testing-changed-files-coverage
branch
from
October 3, 2026 15:20
68ed6d3 to
4b2a02f
Compare
ondrejmirtes
marked this pull request as ready for review
October 3, 2026 15:21
Collaborator
|
This pull request has been marked as ready for review. |
…anged files The diff base came from the repository's default branch, so when 2.3.x became the default on 2026-09-02 every pull request against 2.2.x was diffed against 2.3.x and Infection got hundreds of "changed" files. The base is now set in the workflow, per branch, like its push branch. Infection also opens the coverage report of every source file the index lists, whatever the diff, and the report of all of src/ (3.6 GB of XML) outgrew the runner: the job died with "The runner has received a shutdown signal" on most pull requests that change a source file. Coverage is now collected only for the files Infection mutates - the same files it picks with --git-diff-base and its AM filter - and the job is skipped when there are none. Locally, for a change of IntersectionType.php, Infection's peak RSS drops from 12.1 GB to 0.43 GB with the same mutants generated and killed, and the coverage run takes half the time. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JsHiTmsN24NMF7zvEhwbip
ondrejmirtes
force-pushed
the
mutation-testing-changed-files-coverage
branch
from
October 3, 2026 15:26
4b2a02f to
c9e0ccd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mutation Testing has been dying with "The runner has received a shutdown signal" on most pull requests that change a source file. The cause was two problems layered on top of each other.
1. Wrong diff base since 2026-09-02
The "Determine default branch" step took the base from
git remote show origin. When 2.3.x became the default branch (Open 2.3.x, 2026-09-02 ~12:30 UTC), every PR against 2.2.x was diffed against 2.3.x. The last passing 2.2.x run was at 13:06 UTC. From 15:01 UTC they die, and the #5982 run at 15:24 generated 309 mutations.The base is now configured in the workflow itself (
MUTATION_TESTING_BASE_BRANCH), per branch, like the push branch. The "Open X.Y.x" commit flips it.2. Infection reads all of the coverage, whatever the diff
For every file in the coverage
index.xml, Infection loads that file's coverage XML just to learn its source path (SourceFileInfoProvider::provideFileInfo()), even with--git-diff-lines. The XML-coverage report ofsrc/is 3.6 GB (MutatingScope.php.xmlalone is 321 MB, about 1.7 GB as a DOM). It grows with tests × covered lines: the 2.3.x coverage artifact grew 27% in September alone. Infection's memory sits in libxml, outside PHP's accounting: it reports Memory: 0.31GB while the process RSS reaches 12–14 GB. On the 16 GB runner that ends with the VM being shut down. Small 2.3.x PRs still fit until about 2026-09-23. Since then it's a coin toss: theconstbranch passed on 8.3 and died on 8.4 at the same commit.Coverage is now collected only for the files Infection will mutate. A new small job lists them, using the same
--git-diff-base+AMfilter Infection applies. A generatedphpunit-mutation-testing.xmlrestricts<source>to those files. When nothing insrc/orbuild/PHPStan/Buildchanged, the mutation jobs are skipped instead of spending ~10 minutes discovering that.Measured locally (PHP 8.4, pcov) on a change to two
->yes()lines inIntersectionType.php:src/coverageVerified on CI
With a temporary commit touching two
->yes()lines inIntersectionType.php(since dropped), both Mutation Testing jobs passed:src/Type/IntersectionType.php.run 37118192701
Merge-up
On the merge-up into 2.3.x,
MUTATION_TESTING_BASE_BRANCHmust become"2.3.x"there, the same way the push branch differs between the branches.🤖 Generated with Claude Code
https://claude.ai/code/session_01JsHiTmsN24NMF7zvEhwbip