Add --rerun-warning flag to emit a warning on each scheduled rerun - #375
LouisDeconinck wants to merge 3 commits into
Conversation
Emits a PytestWarning per rerun so CI tooling such as pytest-github-actions-annotate-failures can surface flaky tests as annotations. Opt-in so filterwarnings=error users are unaffected. Closes pytest-dev#318. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
|
||
| report.outcome = "rerun" | ||
| if item.session.config.option.rerun_warning: | ||
| item.warn( |
There was a problem hiding this comment.
— Comment created by Claude
Bug (high): --rerun-warning combined with filterwarnings = error crashes the whole session with INTERNALERROR.
item.warn() is called from pytest_runtest_protocol, not from inside a setup/call/teardown phase. Under -W error or filterwarnings = error, the warning is raised as an exception, and nothing catches it at this level. I reproduced it with --reruns 1 --rerun-warning -W error on a file where one test fails once: the run ends with INTERNALERROR> pytest.PytestWarning: test_x.py::test_a failed on attempt 1 and will be rerun and no tests ran. The remaining tests never run. The report is also left with outcome = "rerun" but is never logged, and any --max-suite-reruns slot has already been used up.
The help text and README say only that the warnings "turn into errors". Users will read that as a failing test, not an aborted session. Fix options: emit the warning in a way that cannot abort the protocol (for example, catch the exception and fail the test), or document the real behaviour clearly. A test covering -W error would also help.
| continue | ||
|
|
||
| report.outcome = "rerun" | ||
| if item.session.config.option.rerun_warning: |
There was a problem hiding this comment.
— Comment created by Claude
Gap (low): worker-crash reruns under xdist emit no warning.
The warning is only emitted on this in-process rerun path. XDistHooks.pytest_handlecrashitem also schedules reruns (sched.mark_test_pending(crashitem) / report.outcome = "rerun") and does not check rerun_warning. So with -n and --rerun-warning, a test that crashes its worker and is rerun gets no annotation. The help text promises a warning "each time a test is scheduled for rerun". Either emit a warning on that path too, or narrow the wording.
2 participants
Summary
--rerun-warningflag that emits aPytestWarningeach time a test is scheduled for rerunfilterwarnings = errorusers are unaffected; the caveat is documented in--helpand the READMECloses #318
Test plan