Skip to content

Record retried failures as flakyFailure elements in JUnit XML - #380

Open
LouisDeconinck wants to merge 2 commits into
pytest-dev:masterfrom
LouisDeconinck:issue-170-junitxml-flaky-failure
Open

LouisDeconinck wants to merge 2 commits into
pytest-dev:masterfrom
LouisDeconinck:issue-170-junitxml-flaky-failure

Conversation

@LouisDeconinck

@LouisDeconinck LouisDeconinck commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • pytest's junitxml plugin ignores reports with outcome rerun, so failures from retried attempts left no trace in --junitxml output
  • A small per-config hook now stashes rerun reports and appends <flakyFailure> elements (and <flakyError> for setup/teardown failures) — the Maven Surefire / flaky plugin convention — to the test's final <testcase> when its last call report arrives
  • No-op when --junitxml isn't used; works with xdist-forwarded reports since it keys on nodeid/worker_id/item_index/node

Closes #170

Test plan

  • Test that eventually passes: final testcase has 2 flakyFailure elements with message + traceback, no <failure>
  • Test that keeps failing: final testcase has flakyFailure(s) plus the real <failure>
  • Full suite passes (252 tests)

pytest's junitxml plugin drops reports with outcome 'rerun', so failures
from attempts that were retried left no trace in the XML. Stash rerun
reports and append flakyFailure/flakyError elements (the Maven Surefire
convention) to the test's final testcase so flaky failures are visible to
XML consumers.

Closes pytest-dev#170.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think setup failures can still disappear here. pending is only flushed when a later call report arrives, but if the final attempt also fails in setup there is no call report, so the earlier rerun setup errors never become <flakyError>. Could we flush on the terminal setup/teardown report too and add that case?

pending was only flushed when a call report arrived, so rerun failures
from attempts whose final execution failed in setup were dropped from
the JUnit XML.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@LouisDeconinck

Copy link
Copy Markdown
Contributor Author

Fixed in 8734762: pending is now flushed on any failed non-rerun setup/teardown report as well, so rerun setup errors become <flakyError> elements even when the final attempt fails in setup. Added a regression test.

@LouisDeconinck

Copy link
Copy Markdown
Contributor Author

Fixed — pending rerun reports are flushed on the terminal setup/teardown report as well, so setup errors on earlier attempts always become <flakyError> elements, plus a regression test (8734762).

sylvesterkaczmarek

This comment was marked as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failure information for prior failing test executions is omitted from XML reports

2 participants