Skip to content

Run tests with ParallelTestRunner - #323

Merged
devmotion merged 5 commits into
masterfrom
dmw/parallel-test-runner
Oct 23, 2026
Merged

devmotion merged 5 commits into
masterfrom
dmw/parallel-test-runner

Conversation

@devmotion

@devmotion devmotion commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Tests have become a bit slow, so this switches test/runtests.jl to ParallelTestRunner.jl, as ChainRules did in JuliaDiff/ChainRules.jl#848. The test files used to run one after another in one process. Now each runs on its own worker process, in parallel.

  • find_tests picks up every file under test/ except the helpers utils.jl and derivatives/elementwise_utils.jl. ParallelTestRunner evaluates each file in its own module, so the per-file module ... end wrappers are removed. The nested IsolatedModuleForTestingScoping in ChainRulesTests.jl is kept. Without the wrapper, four loop variables in TrackedTests.jl triggered soft-scope warnings, so they are now declared local.
  • The slowest files are split so that no single file sets the wall time. The nested Jacobian tests move from api/JacobianTests.jl to api/NestedJacobianTests.jl. The binary rule tests move from derivatives/ElementwiseTests.jl to derivatives/ElementwiseBinaryTests1.jl and ElementwiseBinaryTests2.jl, which each take every other rule. The shared helpers move to derivatives/elementwise_utils.jl.
  • The multithreaded recording test for double free crash with multi-threaded code only when using multiple threads #218 moves from api/GradientTests.jl to test/ThreadsTests.jl, unchanged. ParallelTestRunner's workers are single-threaded, so test_worker gives only this file a dedicated worker with --threads=4. A new @test Threads.nthreads() > 1 makes the test fail if it ever runs on a single-threaded worker, instead of passing without testing anything concurrent.
  • JULIA_NUM_THREADS: 4 is removed from CI, since only that test needed it. A side effect is that a local Pkg.test now runs it multithreaded too.
  • ParallelTestRunner = "2.0.1" is added to test/Project.toml. v2.0.0 has a bug where find_tests doubles the directory of nested tests, e.g. compat/compat/CompatTests.jl.
  • ParallelTestRunner depends on Statistics and loads it on every worker, so StatisticsExt is already loaded when derivatives/LinAlgTests starts. The check that it is not loaded before using Statistics is removed, along with the DOWNGRADE_TEST variable in the Downgrade workflow, which existed only to skip that check.

You can now select tests, e.g. Pkg.test("ReverseDiff"; test_args=["api/GradientTests"]), or list them with test_args=["--list"].

🤖 Generated with Claude Code

Each test file now runs in its own sandbox module on a pool of worker
processes, so the per-file `module` wrappers are dropped. The #218
multithreaded recording test moves to `test/ThreadsTests.jl` and gets a
dedicated 4-thread worker via `test_worker`; all other workers stay
single-threaded, so `JULIA_NUM_THREADS` is no longer set in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devmotion
devmotion force-pushed the dmw/parallel-test-runner branch from bef8cbd to 3e832c1 Compare October 22, 2026 18:39
ParallelTestRunner loads Statistics on its workers, so StatisticsExt is
always loaded before LinAlgTests runs. Drop the check that it isn't, and
the DOWNGRADE_TEST variable that only existed for it.

Require ParallelTestRunner 2.0.1, which fixes the paths of nested tests
returned by find_tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.27%. Comparing base (ff259e8) to head (26179ca).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #323   +/-   ##
=======================================
  Coverage   94.27%   94.27%           
=======================================
  Files          19       19           
  Lines        1695     1695           
=======================================
  Hits         1598     1598           
  Misses         97       97           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

devmotion and others added 2 commits October 23, 2026 10:26
ElementwiseTests and JacobianTests dominated the wall time of the parallel
test run. Move the binary DiffRules tests into ElementwiseBinaryTests and
the nested Jacobian tests into NestedJacobianTests, with the shared
elementwise helpers in derivatives/elementwise_utils.jl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings

ElementwiseBinaryTests took about 960 s on its own worker and set the
wall time of the whole run, while the other workers finished after about
750 s. Each half now takes every other binary rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread test/api/NestedJacobianTests.jl Outdated
Comment thread test/derivatives/elementwise_utils.jl Outdated
Co-authored-by: David Müller-Widmann <devmotion@users.noreply.github.com>
@devmotion
devmotion merged commit 9d49db8 into master Oct 23, 2026
9 checks passed
@devmotion
devmotion deleted the dmw/parallel-test-runner branch October 23, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant