Skip to content

refactor: move *-experiments features out of the published crates into benchmarks/experiments - #277

Merged
unbalancedparentheses merged 7 commits into
mainfrom
refactor/experiments-to-benchmarks
Nov 16, 2026
Merged

unbalancedparentheses merged 7 commits into
mainfrom
refactor/experiments-to-benchmarks

Conversation

@unbalancedparentheses

@unbalancedparentheses unbalancedparentheses commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Post-0.8 cleanup: the *-experiments features leave the published crates and move to benchmarks/experiments/, kept runnable rather than deleted ("we might do it again").

What moved where

  • New standalone crate benchmarks/experiments/ (spawned-experiments, publish = false, its own [workspace], so monoio/pprof stay out of the root lockfile, cargo deny and cargo audit):
    • benches designs, io_designs, workload_research, lifecycle_soak, pinned_pipeline, runtime_isolation and their support files (from concurrency/benches/);
    • tests backend_experiments, full_actor_shards, native_dispatcher, ready_gate_loom, benchmark_startup (from concurrency/tests/), plus the two descriptor-limit tests split out of backend_resource_limits.rs (now descriptor_limit.rs);
    • features backend-experiments, io-experiments (monoio), profiling-experiments (pprof), same names so the sources moved unchanged.
  • backend-experiments.patch + prepare.sh: the scheduler hooks sit on the actor hot path (spawn routing via SPAWNED_EXPERIMENT_SCHEDULER, MessageTurn batching/time slicing in the dispatch loop, SPAWNED_EXPERIMENT_RUNTIME in threads::block_on) plus rt/src/tasks/experimental* and dispatch_gate.rs. They can't live in a published crate, even behind a feature, so they're kept as a patch. prepare.sh adds a scratch worktree and applies it. Applied to this branch, the patch reproduces main's rt/src and concurrency/src byte for byte. The Python drivers now build with --manifest-path benchmarks/experiments/Cargo.toml from a prepared tree.
  • Published crates: spawned-rt loses async-task and crossbeam-deque. spawned-concurrency loses monoio, pprof and every cfg(feature = "backend-experiments") branch. The three feature names stay as empty no-ops, because api-compatibility (cargo-semver-checks feature_missing) treats removing a feature as a major break under the 0.8 patch policy. Drop them at 0.9. The only change to default-build code is folding one let result = …; result (left behind when the cfg'd turn hook was removed) into a tail expression (clippy let_and_return), which compiles to the same code.
  • deny.toml / .cargo/audit.toml: the quick-xml, fxhash and CDDL (inferno) exceptions are gone with their deps, and so are the now-unused 0BSD and BSD-2-Clause allowances.
  • Docs: benchmarks/README.md (new "Experiments" section), docs/TESTING.md, docs/GUIDE.md, ROADMAP.md, the research notes and CHANGELOG.md.

Checks run locally

cargo clippy --workspace --all-targets [--all-features] -D warnings, cargo test -p spawned-concurrency --lib (243 passed), cargo deny --all-features check, cargo fmt --check. In a prepared tree: clippy on the experiments crate (all targets), the benchmark_startup/backend_experiments tests and the turn_regressions unit tests.

Model: no change because this only deletes code compiled under the removed backend-experiments feature; default and every remaining feature build of the modelled lifecycle code is unchanged.

@unbalancedparentheses unbalancedparentheses added the no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...) label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Preparation output breaks the documented command substitution — prepare.sh:28. git worktree add prints HEAD is now at … to stdout, so dir=$(benchmarks/experiments/prepare.sh) captures that alongside the directory and cd "$dir" fails. Redirect Git’s stdout to stderr so stdout contains only the path.

  • [P2] The soak driver rejects every prepared tree — fede_soak.py:29. The required preparation patch modifies tracked files, but the preceding assert not meta['dirty'] aborts before the new build command runs. Validate and record the expected patch separately from unrelated modifications so the documented workflow can execute.

I found no correctness or concurrency regression in the production actor-path removals.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR #277, move backend/IO/profiling experiments out of the published crates

The direction is right. Taking the scheduler hooks off the actor hot path simplifies run_actor_lifecycle, the spawn path and threads::block_on. Keeping the feature names as no-ops for 0.8 patch compatibility is correct. The actor.rs and link_regression_tests.rs removals are an exact reverse of the patch hunks, and the default dispatch loop is back to its pre-experiment shape.

Two scripts are broken in a prepared tree. There is also a bit-rot risk worth addressing before merge.

Bugs

1. --locked fails in a prepared tree (benchmarks/run_reliability.py)

  • The patch adds crossbeam-deque and async-task as optional deps to rt/Cargo.toml.
  • Cargo records optional deps in Cargo.lock whether or not they are enabled. The root lockfile in the scratch tree is therefore stale.
  • Any --locked command against the workspace fails with "lock file needs to be updated". That includes:
    • the new build-baseline step (cargo build --locked --release -p spawned-concurrency --bench baseline);
    • cancellation-model (--locked -p spawned-rt);
    • whatever builds the tests[...] binaries.
  • benchmarks/README.md explicitly says to run this script in a prepared tree, so this path is the documented one.
  • Fix options:
    • have prepare.sh run cargo update -w (or cargo generate-lockfile --offline) after applying the patch;
    • include the Cargo.lock delta in the patch;
    • drop --locked for these steps when run in a prepared tree.

2. benchmarks/fede_soak.py:28 always fails in a prepared tree

  • meta['dirty'] = bool(git status --porcelain) is followed by assert not meta['dirty'].
  • A worktree with backend-experiments.patch applied is dirty by construction, so the soak can never run.
  • revision = git rev-parse HEAD also no longer identifies what was measured.
  • Suggested fix:
    • exclude the patch's known paths from the dirty check, or compare git diff against the patch;
    • record sha256(backend-experiments.patch) in meta.
  • Check fede_research.py for the same pattern.

Risks / suggestions

3. Nothing builds the experiments crate in CI, so it will rot silently

  • The crate is outside the workspace. It only compiles after prepare.sh, because the moved tests reference spawned_rt::tasks::experimental::*.
  • The patch is about 2.9k lines and touches concurrency/src/tasks/actor.rs, the most frequently edited file. Expect it to stop applying within a few PRs.
  • The moved tests and benches also lose cargo clippy --all-targets coverage.
  • Consider a cheap, non-blocking CI job (scheduled, or path-filtered on actor.rs, rt/src/threads, rt/src/tasks/mod.rs) that runs prepare.sh && cargo check --manifest-path benchmarks/experiments/Cargo.toml --all-targets.
  • At minimum, run git apply --check so drift is noticed when it happens rather than when someone needs the experiments.

4. No lockfile for the experiments crate (.gitignore)

  • The research scripts used to run with --locked. Now monoio, pprof, tokio, etc. float on every build, which hurts reproducibility of archived benchmark numbers.
  • Committing benchmarks/experiments/Cargo.lock does not bring monoio or pprof back into the root audit, because cargo-deny and cargo-audit only look at the root workspace.
  • I'd commit it and restore --locked in design_experiments.py, fede_*.py and run_reliability.py.

5. prepare.sh robustness

  • dir=$(benchmarks/experiments/prepare.sh) captures stdout. Redirect git's output (git worktree add … >&2) so nothing git writes to stdout can end up in $dir.
  • If git apply fails, the worktree is left behind and the next run fails with "already exists". Add a cleanup on failure: git worktree remove --force "$dir".
  • The default target/experiments-tree disappears on cargo clean and leaves stale worktree metadata. Either mention git worktree prune, or default to a path outside target/.

6. Semver note for the CHANGELOG

  • The feature names remain, but spawned_rt::tasks::experimental and rt::experimental::dispatcher were pub (though #[doc(hidden)]) under backend-experiments. They are now gone.
  • That is acceptable for hidden research API, but the CHANGELOG line ("feature names stay as no-ops") should say the hidden experimental modules are removed. Anyone who enabled the feature and used them in 0.8.x will break on a patch bump.

7. Unverified compile of moved tests

  • backend_experiments.rs, full_actor_shards.rs, native_dispatcher.rs and benchmark_startup.rs were renamed at 100% similarity.
  • The new crate only declares futures, tokio, monoio, pprof and loom. If any of these files used another spawned-concurrency dev-dependency (e.g. tracing, proptest, parking_lot) or another tests/support/* module besides deadline.rs, they won't compile.
  • Please confirm cargo test --manifest-path benchmarks/experiments/Cargo.toml --no-run passes in a prepared tree, ideally via the CI job in (3).

Nits

  • run_reliability.py: the single-letter l in the generator (E741). line reads better.
  • docs/GUIDE.md ~L322: the reflowed paragraph has one very long line ("…and Tokio stays the default. Their measurements…"). Rewrap it.
  • deny.toml: the removals of 0BSD and BSD-2-Clause look right (adler2 is gone, and zerocopy's OR expression is satisfied by MIT/Apache). Run cargo deny --all-features list -l license once to confirm nothing else relied on them.

Overall this is a good cleanup. Fix (1) and (2) before merge, and I'd strongly recommend (3) and (4) so the moved experiments stay runnable.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The separation of the experimental backends/dependencies out of spawned-rt and spawned-concurrency into benchmarks/experiments/ is clean, and keeping the feature flags as no-ops preserves 0.8 semver compatibility.

Two issues in the benchmark driver workflow when running from a tree prepared by benchmarks/experiments/prepare.sh:

  1. --locked fails in a prepared tree (benchmarks/run_reliability.py:98, 102, benchmarks/experiments/prepare.sh:29)
    benchmarks/README.md notes that run_reliability.py should be run in a prepared tree. However, backend-experiments.patch adds async-task and crossbeam-deque back to rt/Cargo.toml without updating the root Cargo.lock. Running cargo test --locked -p spawned-rt ... or cargo build --locked ... -p spawned-concurrency in run_reliability.py will fail with a lockfile mismatch unless prepare.sh refreshes the lockfile in the scratch worktree (or --locked is dropped there).

  2. Dirty-worktree assertion fails in a prepared tree (benchmarks/fede_soak.py:29)
    prepare.sh applies backend-experiments.patch to a detached worktree without committing. fede_soak.py checks dirty=bool(capture('git','status','--porcelain')) and assert not meta['dirty'] on line 29, which will always fail in a prepared tree (check fede_research.py as well if it has a similar cleanliness check).


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The published hot path cleanup is right: experiment branches are gone from dispatch, spawn, and block_on, and the 0.8 feature names stay as no-ops. The migration around that has a few blockers.

Experiments crate will not build as written

benchmarks/experiments/Cargo.toml declares its own [workspace] and lives inside the repo workspace, and the root Cargo.toml is not updated to exclude it. Cargo rejects that (current package believes it's in a workspace when it's not). The same layout is what prepare.sh checks out under target/experiments-tree, so the documented cargo test --manifest-path benchmarks/experiments/Cargo.toml path fails even after the patch.

Path deps on spawned-rt / spawned-concurrency (workspace = true fields) make this worse from a nested workspace. Drop the nested [workspace], exclude the package from the root workspace, and confirm inheritance still resolves. Add a CI job that runs prepare.sh and builds; nothing else will catch patch rot.

Also missing from that manifest: whatever the moved harness = false benches and tests used as dev-deps (criterion is the obvious one). They previously compiled as spawned-concurrency targets. default enabling backend-experiments means an unprepared tree fails with unresolved experimental imports instead of a clear “run prepare.sh” error.

prepare.sh / patch workflow

  • backend-experiments.patch starts with a prose header before diff --git. git apply often errors on that (corrupt patch / unrecognized input). Start the file at the diff, or strip the preamble before applying.
  • On apply failure the script exits without git worktree remove, so the next run dies on a leftover worktree (prepare.sh ~28–32).
  • Default dir is $root/target/experiments-tree. cargo clean deletes a registered worktree and leaves the main repo’s worktree metadata broken. Put it outside target/.
  • The applied patch is uncommitted, but benchmarks/fede_soak.py still assert not meta['dirty'] while the README now says to run those drivers in a prepared tree. Commit the patch on the detached HEAD (or drop the dirty assert for this workflow).
  • run_reliability.py dropped --locked on the experiments build, and Cargo.lock is gitignored. Research numbers will drift. Commit a lockfile in the scratch tree at least.

The patch only restores backend-experiments hooks, not io-experiments / monoio code in the library. If io_designs still expects in-crate monoio dispatch, it will not build after prepare. Worth a compile check, not just a rename.

Smaller notes

  • rt/src/threads/mod.rs: the removed SPAWNED_EXPERIMENT_RUNTIME block was the likely sole user of OnceLock in that module. Unused import will fail a deny-warnings build.
  • docs/GUIDE.md drops *-experiments from the unstable list, but the flags still exist and silently do nothing until 0.9. Keep one line so --features backend-experiments is not read as supported behavior.
  • Removing the quick-xml / fxhash ignores is correct for the published graph. Call out that benchmarks/experiments is now unscanned (monoio, pprof, inferno).

I would not merge until a prepared tree actually cargo tests and cargo bench --no-runs the experiments crate.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

⚠️ Review failed: Kimi API request failed with status 520:


Automated review by Kimi (Moonshot AI)

@unbalancedparentheses
unbalancedparentheses added this pull request to the merge queue Nov 16, 2026
@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Overall this is a well-scoped cleanup: the published crates lose the heavy optional dependencies, the no-op feature names preserve patch-compatibility, and moving the experiment code into a patch-applied scratch tree keeps it out of the default build path.

A few issues to consider:


1. Experiments crate defaults to Linux-only dependencies on all platforms

benchmarks/experiments/Cargo.toml enables io-experiments and profiling-experiments in the default feature set (line 32), but:

  • monoio at line 23 pulls in iouring, which is Linux-only.
  • pprof at line 26 is already target-gated to Linux, but the feature is not target-gated.

A plain cargo test --manifest-path benchmarks/experiments/Cargo.toml on macOS or Windows will likely fail building monoio/io-uring. If the crate is intentionally Linux-only, that should be documented or enforced; otherwise, target-gate monoio and the io-experiments/profiling-experiments defaults.


2. Moved tests lose root CI coverage unless the prepared-tree workflow is wired in

The new crate declares [workspace] (benchmarks/experiments/Cargo.toml:16), so root cargo test --all-features will not build or run its tests. docs/TESTING.md also drops benchmark_startup from the root cargo test command. The diff does not add CI steps for benchmarks/experiments/prepare.sh or the standalone manifest, so tests like descriptor_limit, native_dispatcher, ready_gate_loom, and benchmark_startup may silently stop running. If CI previously ran these, ensure the prepared-tree commands are added to CI.


3. prepare.sh is fragile when the scratch directory already exists

benchmarks/experiments/prepare.sh (around line 21) runs:

git -C "$root" worktree add --detach "$dir" "$rev"

If $dir already exists from an interrupted or previous run, this fails with a confusing error. Consider removing a stale worktree first (git worktree remove --force) or generating a unique directory/using mktemp -d.


4. Experiment builds lost --locked

The research scripts (benchmarks/design_experiments.py, benchmarks/fede_research.py, benchmarks/run_reliability.py) now build benchmarks/experiments/Cargo.toml without --locked, and benchmarks/experiments/Cargo.lock is gitignored. Research/reproducibility builds will float dependency versions. If the standalone crate is expected to produce comparable benchmark results, commit/generate its lockfile and use --locked.


5. Patch-applied experiment code still panics on bad env values despite documented error handling

Inside backend-experiments.patch, spawn_sharded in rt/src/tasks/experimental.rs contains:

let count: usize = std::env::var("SPAWNED_EXPERIMENT_WORKERS")
    .unwrap_or_else(|_| "2".into())
    .parse()
    .expect("SPAWNED_EXPERIMENT_WORKERS must be an integer");
assert!((1..=256).contains(&count), ...);

This will panic on an invalid environment variable, while the doc comment claims “no caller-thread panic.” It is experiment-only and not part of the published crates, so this is low priority, but the comment is misleading. If invalid env is a supported failure mode, return io::Error instead of panicking.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

Merged via the queue into main with commit d79c041 Nov 16, 2026
44 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-model-change Modelled code changed without a model change; the PR body says why (Model: no change because ...)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant