Niels/investigation.flaky test hunter on pr - #8860
Open
nielsVoogt wants to merge 3 commits into
Open
nielsVoogt wants to merge 3 commits into
nielsVoogt wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical workflow and validation-fixture issues prevent reliable flaky-test detection.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds PR-based flaky-test detection for changed or new Portal E2E specs.
Changes:
- Adds changed-spec selection tooling and an npm command.
- Integrates candidate detection and repeated flaky-test runs into CI.
- Adds temporary stable and flaky validation specs.
File summaries
| File | Description |
|---|---|
tools/select-changed-e2e-tests.mjs |
Selects qualifying changed E2E specs. |
tools/package.json |
Registers the selector command. |
e2e/portal/tests/TestingCi/PurposlyStable.spec.ts |
Adds stable CI validation coverage. |
e2e/portal/tests/TestingCi/PurposelyFlaky.spec.ts |
Adds temporary flaky-test validation. |
.github/workflows/test_e2e_portal.yml |
Runs candidate detection and flaky-test jobs. |
Review details
Suppressed comments (1)
.github/workflows/test_e2e_portal.yml:92
- This new job is gated by
path-filter, but that filter does not includetools/select-changed-e2e-tests.mjs. After this PR, a future change to the selector alone will make the E2E workflow skip candidate detection, so regressions in this CI-critical script can merge without the workflow exercising them. Include the selector in the path-filter inputs.
if: ${{ needs.path-filter.outputs.should_skip != 'true' && github.event_name == 'pull_request' }}
- Files reviewed: 5/5 changed files
- Comments generated: 8
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| SPEC_FILE: ${{ matrix.file }} | ||
| REPEAT_COUNT: ${{ github.event.inputs.repeat_count || '30' }} | ||
| run: | | ||
| npm test -- "$SPEC_FILE" --fail-on-flaky-tests --retries=0 --repeat-each "$REPEAT_COUNT" -x --trace retain-on-failure --reporter=html |
| test-shard-resolution-e2e: | ||
| runs-on: ubuntu-slim | ||
| needs: [test-shard-e2e] | ||
| needs: [test-shard-e2e, find-flaky-changed-tests] |
Comment on lines
+3
to
+4
| // TEMPORARY — for validating the `find-flaky-changed-tests` CI job only. | ||
| // Delete this file before merging; do not commit it to `main`. |
Comment on lines
+11
to
+14
| repeat_count: | ||
| description: 'Number of repetitions per test when hunting flaky tests' | ||
| type: string | ||
| default: '30' |
| async function getChangedSpecFiles({ repositoryRoot, base, head }) { | ||
| const stdout = await execFileAsync( | ||
| 'git', | ||
| ['diff', '--no-renames', '--numstat', base, head], |
| @@ -0,0 +1,5 @@ | |||
| import { expect, test } from '@playwright/test'; | |||
| import { appendFileSync, readFileSync } from 'node:fs'; | ||
| import { parseArgs } from 'node:util'; | ||
|
|
||
| const execFileAsync = (command, commandArgs, options) => |
Comment on lines
+142
to
+147
| function toE2eRelativePath(path) { | ||
| return path.slice('e2e/'.length); | ||
| } | ||
|
|
||
| function writeGithubOutput({ candidates }) { | ||
| const files = candidates.map(({ path }) => toE2eRelativePath(path)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AB#44653
Describe your changes
TESTRUNNING - NOT READY YET
Checklist before requesting a code review
Portal preview-deployment
This PR does not have any preview deployments yet.