Keep nested pytester sessions isolated from pytest-randomly - #368
Merged
Merged
Conversation
pytest-randomly is still auto-loaded in in-process runpytest sessions even when the outer pytest is started with -p no:randomly, which shuffled collection order and failed order-sensitive teardown tests.
icemac
approved these changes
Sep 11, 2026
Contributor
|
Thank you for this PR. 😃 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nested
testdir.runpytestsessions now pass-p no:randomly. That flag is a no-op when pytest-randomly is not installed.Fixes #218.
Summary
testdir.runpytestsessions still auto-loaded pytest-randomly when that plugin was installed, even if the outer pytest was started with-p no:randomly. pytester dropsPYTEST_ADDOPTS, and in-processpytest.main()re-autoloads entry points.test_run_session_teardown_once_after_reruns(and other teardown tests that assume collection order) failed because they assert default class/module/method order.8 failed, 2 passed, 5 skipped, 1 error, 18 rerun. This is test isolation, not a product teardown fix.What I chose and the alternative
Wrap
testdir.runpytestso every nested session gets-p no:randomly.The other options were making the teardown assertions order-independent, or passing the flag only on
test_run_session_teardown_once_after_reruns. The reporter's outer-p no:randomlynever reaches in-process nested sessions, and several tests need stable collection order (earlier vs later modules); some of those failures are seed-dependent. Disabling the plugin that reorders is the smallest reversible isolation.Happy to switch to order-independent assertions, or to add pytest-randomly as a tox extra.
Test plan
test_run_session_teardown_once_after_rerunsfails with pytest-randomly installed and the wrap removed-p no:randomlywithout the wrap-p no:randomlytests/suite passes with pytest-randomly installed (several seeds)tests/suite still passes with pytest-randomly uninstalled