Skip to content

Keep nested pytester sessions isolated from pytest-randomly - #368

Merged
icemac merged 1 commit into
pytest-dev:masterfrom
cpruijsen:fix/issue-218
Sep 11, 2026
Merged

icemac merged 1 commit into
pytest-dev:masterfrom
cpruijsen:fix/issue-218

Conversation

@cpruijsen

@cpruijsen cpruijsen commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Nested testdir.runpytest sessions now pass -p no:randomly. That flag is a no-op when pytest-randomly is not installed.

Fixes #218.

Summary

  • Nested testdir.runpytest sessions still auto-loaded pytest-randomly when that plugin was installed, even if the outer pytest was started with -p no:randomly. pytester drops PYTEST_ADDOPTS, and in-process pytest.main() re-autoloads entry points.
  • test_run_session_teardown_once_after_reruns (and other teardown tests that assume collection order) failed because they assert default class/module/method order.
  • Session and class fixtures already teardown once under shuffle; inner outcomes stayed 8 failed, 2 passed, 5 skipped, 1 error, 18 rerun. This is test isolation, not a product teardown fix.

What I chose and the alternative

Wrap testdir.runpytest so every nested session gets -p no:randomly.

The other options were making the teardown assertions order-independent, or passing the flag only on test_run_session_teardown_once_after_reruns. The reporter's outer -p no:randomly never reaches in-process nested sessions, and several tests need stable collection order (earlier vs later modules); some of those failures are seed-dependent. Disabling the plugin that reorders is the smallest reversible isolation.

Happy to switch to order-independent assertions, or to add pytest-randomly as a tox extra.

Test plan

  • test_run_session_teardown_once_after_reruns fails with pytest-randomly installed and the wrap removed
  • Same test still fails with outer -p no:randomly without the wrap
  • Same test passes with the wrap, with and without outer -p no:randomly
  • Full tests/ suite passes with pytest-randomly installed (several seeds)
  • Full tests/ suite still passes with pytest-randomly uninstalled
  • CI pytest matrix (8.2-9.1, xdist) once the PR is up

pytest-randomly is still auto-loaded in in-process runpytest sessions even when the outer pytest is started with -p no:randomly, which shuffled collection order and failed order-sensitive teardown tests.
@icemac

icemac commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Thank you for this PR. 😃

@icemac
icemac merged commit 87d81b2 into pytest-dev:master Sep 11, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_run_session_teardown_once_after_reruns fails with pytest-randomly

2 participants