Record retried failures as flakyFailure elements in JUnit XML - #380
LouisDeconinck wants to merge 2 commits into
Conversation
pytest's junitxml plugin drops reports with outcome 'rerun', so failures from attempts that were retried left no trace in the XML. Stash rerun reports and append flakyFailure/flakyError elements (the Maven Surefire convention) to the test's final testcase so flaky failures are visible to XML consumers. Closes pytest-dev#170. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
I think setup failures can still disappear here. pending is only flushed when a later call report arrives, but if the final attempt also fails in setup there is no call report, so the earlier rerun setup errors never become <flakyError>. Could we flush on the terminal setup/teardown report too and add that case?
pending was only flushed when a call report arrived, so rerun failures from attempts whose final execution failed in setup were dropped from the JUnit XML. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Fixed in 8734762: |
|
Fixed — pending rerun reports are flushed on the terminal setup/teardown report as well, so setup errors on earlier attempts always become |
Summary
rerun, so failures from retried attempts left no trace in--junitxmloutput<flakyFailure>elements (and<flakyError>for setup/teardown failures) — the Maven Surefire /flakyplugin convention — to the test's final<testcase>when its lastcallreport arrives--junitxmlisn't used; works with xdist-forwarded reports since it keys onnodeid/worker_id/item_index/nodeCloses #170
Test plan
flakyFailureelements with message + traceback, no<failure>flakyFailure(s) plus the real<failure>