Skip to content

Mutation testing: diff against a configured branch, cover only the changed files - #6664

Merged
ondrejmirtes merged 1 commit into
2.2.xfrom
mutation-testing-changed-files-coverage
Oct 3, 2026
Merged

ondrejmirtes merged 1 commit into
2.2.xfrom
mutation-testing-changed-files-coverage

Conversation

@ondrejmirtes

@ondrejmirtes ondrejmirtes commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Mutation Testing has been dying with "The runner has received a shutdown signal" on most pull requests that change a source file. The cause was two problems layered on top of each other.

1. Wrong diff base since 2026-09-02

The "Determine default branch" step took the base from git remote show origin. When 2.3.x became the default branch (Open 2.3.x, 2026-09-02 ~12:30 UTC), every PR against 2.2.x was diffed against 2.3.x. The last passing 2.2.x run was at 13:06 UTC. From 15:01 UTC they die, and the #5982 run at 15:24 generated 309 mutations.

The base is now configured in the workflow itself (MUTATION_TESTING_BASE_BRANCH), per branch, like the push branch. The "Open X.Y.x" commit flips it.

2. Infection reads all of the coverage, whatever the diff

For every file in the coverage index.xml, Infection loads that file's coverage XML just to learn its source path (SourceFileInfoProvider::provideFileInfo()), even with --git-diff-lines. The XML-coverage report of src/ is 3.6 GB (MutatingScope.php.xml alone is 321 MB, about 1.7 GB as a DOM). It grows with tests × covered lines: the 2.3.x coverage artifact grew 27% in September alone. Infection's memory sits in libxml, outside PHP's accounting: it reports Memory: 0.31GB while the process RSS reaches 12–14 GB. On the 16 GB runner that ends with the VM being shut down. Small 2.3.x PRs still fit until about 2026-09-23. Since then it's a coin toss: the const branch passed on 8.3 and died on 8.4 at the same commit.

Coverage is now collected only for the files Infection will mutate. A new small job lists them, using the same --git-diff-base + AM filter Infection applies. A generated phpunit-mutation-testing.xml restricts <source> to those files. When nothing in src/ or build/PHPStan/Build changed, the mutation jobs are skipped instead of spending ~10 minutes discovering that.

Measured locally (PHP 8.4, pcov) on a change to two ->yes() lines in IntersectionType.php:

full src/ coverage changed files only
coverage XML 3.6 GB 30 MB
coverage run 164 s 82 s
Infection peak RSS 12.1 GB 0.43 GB
Infection result 4 generated, 4 killed 4 generated, 4 killed

Verified on CI

With a temporary commit touching two ->yes() lines in IntersectionType.php (since dropped), both Mutation Testing jobs passed:

  • The changed-files job listed exactly src/Type/IntersectionType.php.
  • Coverage ran for that file only (~4 min).
  • Infection generated 4 mutants and killed all 4 (MSI 100%), using 0.13 GB.

run 37118192701

Merge-up

On the merge-up into 2.3.x, MUTATION_TESTING_BASE_BRANCH must become "2.3.x" there, the same way the push branch differs between the branches.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JsHiTmsN24NMF7zvEhwbip

echo 'files<<EOF'
echo "$files"
echo 'EOF'
} >> "$GITHUB_OUTPUT"
Comment thread .github/workflows/tests.yml Fixed
@ondrejmirtes
ondrejmirtes force-pushed the mutation-testing-changed-files-coverage branch from 68ed6d3 to 4b2a02f Compare October 3, 2026 15:20
@ondrejmirtes
ondrejmirtes marked this pull request as ready for review October 3, 2026 15:21
@phpstan-bot

Copy link
Copy Markdown
Collaborator

This pull request has been marked as ready for review.

…anged files

The diff base came from the repository's default branch, so when 2.3.x
became the default on 2026-09-02 every pull request against 2.2.x was
diffed against 2.3.x and Infection got hundreds of "changed" files. The
base is now set in the workflow, per branch, like its push branch.

Infection also opens the coverage report of every source file the index
lists, whatever the diff, and the report of all of src/ (3.6 GB of XML)
outgrew the runner: the job died with "The runner has received a shutdown
signal" on most pull requests that change a source file. Coverage is now
collected only for the files Infection mutates - the same files it picks
with --git-diff-base and its AM filter - and the job is skipped when there
are none. Locally, for a change of IntersectionType.php, Infection's peak
RSS drops from 12.1 GB to 0.43 GB with the same mutants generated and
killed, and the coverage run takes half the time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JsHiTmsN24NMF7zvEhwbip
@ondrejmirtes
ondrejmirtes force-pushed the mutation-testing-changed-files-coverage branch from 4b2a02f to c9e0ccd Compare October 3, 2026 15:26
@ondrejmirtes
ondrejmirtes merged commit c9e0ccd into 2.2.x Oct 3, 2026
572 of 573 checks passed
@ondrejmirtes
ondrejmirtes deleted the mutation-testing-changed-files-coverage branch October 3, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants