[unbalancedparentheses]

Implements the accepted OTP exit-signal design (ROADMAP 5e/5f/5g), including the deferred-reply and pipe-job work from #239. This is a breaking 0.8 candidate; keep it draft until the final release gates pass.

The changes define signal versus administrative stop, trappable linked kills, startup rollback, runtime-drop death, late links/monitors and NoProc, and reply discharge at the death point. The caller's expiry decides request timeouts, as documented in semantics.md and the migration guide. No unsafe code was added to the core crates.

Performance and review fixes

Plain sends carry no reply slot. Requests lend one envelope slot across a turn, allocate deferral state only when needed, and use the message's presence as the dispatch marker. Request construction is inlined. No coarse clock is used: the monotonic deadline and runtime-relative timer are captured before enqueue, preserving paused Tokio clock semantics.

The latest implementation passes the required GitHub perf-gate at 5e323ba3. An independent three-repetition callgrind run on fede (Rust 1.93.0, the same #270 driver, sizes 2,000/6,000, main 33d40160) gives:

Benchmark	Base instructions/op	Candidate	Change
async current-thread burst	3,825	3,887	+1.6%
sequential request	3,395	3,347	-1.4%
16 concurrent requests	2,512	2,564	+2.1%
send, one producer	839	845	+0.8%
send, two producers	833	841	+0.9%
threads request	3,284	2,147	-34.6%

The threshold remains +3%; it now also applies to dependency, toolchain and version changes. The final versioned candidate is being measured separately for release.

Independent Claude and focused Codex reviews found and reproduced two correctness issues: held job completion could reopen emit admission, and sharing an absolute Tokio timer deadline with the monotonic deadline broke paused-clock requests. Both are fixed, with reproducing regressions. Job join waiters are deduplicated and awakened outside the lock. Focused follow-up reviews found no actionable regressions in these fixes or the compact envelope.

Verification

• 27 reply-slot and 11 job-core Loom tests pass, including finish/emit/held-release races.
• The receiver-local ReplyTimer.tla model checks 725 bounded states, including delayed timer arms; both lost-update controls fail as expected. Paused Tokio clock behavior is covered by Rust regressions, rather than claimed as a property of this abstract model.
• The new deferred-job completion invariant and its two-emit control pass/fail respectively. CI trace validation on 5e323ba3 passes; model-sync, simulator, sanitizers, API compatibility and native builds pass.
• A single-CPU fault-injection test exposed a test startup race: RLIMIT_NPROC was applied before the actor's runtime existed. 7f609bf9 adds a bounded startup acknowledgement before denying new threads; its reproduction passes on one CPU. Fresh full CI and both client ports are running.

Release checks

A separate commit retains bounded weekly Lifecycle checks and moves the full MCLifecycleDeep and MCLifecycleLiveDeep checks to manual pre-release gates, alongside the soak. Their models, bounds and properties are unchanged. The full safety sizing run completed in 8h02m; deep liveness exhausted memory and remains incomplete. CI success does not certify those manual gates.

Before release: final-candidate CI and performance gate, quiet performance comparison, release-commit soak/model checks and client validation. Earlier client/soak results predate the final optimizations. Versioning and curated release notes are kept separately in draft #271. No tag or publication has occurred.

[unbalancedparentheses]

/claude

Fresh release review requested for the final 0.8 candidate. Please cover the new changes since e8847a3 through ce4ea4d, particularly precise shared request timers, packed receiver bookkeeping and deadline-shortening races, envelope slot ownership/discharge, native-only feature builds, and the accompanying Loom/TLA+ updates. Earlier full lifecycle/deferred-reply reviews are recorded in this PR's history. Keep the performance threshold unchanged at 3%; review the semantics independently of green CI. If the diff supplied by the workflow is truncated, explicitly report that limitation rather than claiming the entire PR was reviewed.

[github-actions]

🤖 Claude Code Review

⚠️ Warning: Diff was truncated from 62722 to 10000 lines. Some changes were not reviewed.

Review: PR #240 (exit signals, death point D, deferred replies, jobs)

The diff is truncated partway through reply/jobs.rs (ScheduledEmit::settle onward). slot.rs, registry.rs and the tasks/threads run loops are not visible, so this review covers what is shown.

Overall the design is careful. The death point is a single boundary, notices are fenced on the receiver side by edge generation, no user code runs under the job and table locks, and payloads are held until every caller is answered. The Loom, TLA+ and trace coverage is strong. The items below are things I would fix or verify before merging.

Correctness and concurrency

1. reply/job_core.rs JobCore::finish: the check and the notify are not atomic.

{ let mut inner = lock(&self.inner); if !inner.open() {..} if inner.scheduled > 0 {..held..} }
match self.notify(enqueue) { .. }   // re-locks

The comment says "only the work schedules". If a JobCtx (or a clone, or something it was moved into) is still alive in a helper task/thread, this interleaving breaks the "terminal message last" guarantee:

1. emit_at → schedule() runs after finish releases the lock.
2. scheduled becomes 1.
3. notify takes the terminal.
4. The scheduled emit later fires, sees !open(), and is silently dropped, even though emit_at returned Ok.

Fix options:

• Do the scheduled == 0 check and the notification in one critical section (inline notify_keeping_held's body).
• Or document and enforce that JobCtx is !Clone and consumed by the work.

2. link.rs deliver_notice_in_section: callbacks run under the receiver's table lock.

table::take_record runs apply under the receiver's table lock. In production apply → deliver_signal → (target.wake)() and send_exit(exit).

This contradicts the table.rs header, which says the callback "takes only the receiver's exit-request lock". Under that lock it actually:

• cancels a CancellationToken, which runs registered callbacks/wakers;
• enters the mailbox admission gate.

Please verify that nothing reachable from wake or send_exit takes a link-table lock. In particular, check whether Registry::seal, which runs inside close under the table lock, can wait on a SendGate that an in-flight send_exit holds. If verified, update the module doc to list these effects.

3. reply/jobs.rs JobSet::join: wakers accumulate.

Every Pending poll pushes cx.waker().clone() with no dedup, unlike poll_acquire, which uses will_wake. A long join with spurious polls grows idle_async without bound. Apply the same will_wake check.

4. error.rs: From<SendError<T>> for ActorError now yields NoProc.

This is correct only if mailbox admission closes strictly after D on every path. The CHANGELOG promises that sends made while exiting are admitted. Please confirm that neither run loop closes or drains the mailbox before cleanup.die(). ActorStartup::drop does this in the right order; I cannot see the run loops.

Also note the public-API side effect: any user converting a spawned_rt mpsc SendError now gets NoProc.

5. child_handle.rs test wrapping should_panic.

child_handle_wait_blocking_panics_on_current_thread_runtime is now wrapped in test_wait::bounded and keeps #[should_panic(expected = "non-multi-thread tokio runtime")]. This only passes if bounded re-raises the original payload (resume_unwind) rather than its own message. Please verify.

6. reply/deadline.rs thread_service: two OnceLocks for one service.

SERVICE and SERVICE_REF hold the same thing. forget only works after some schedule has called SERVICE_REF.set. It works today, but it is fragile. Hoist SERVICE to module scope and use SERVICE.get() in forget.

Behaviour and API

7. actor_cleanup.rs discharged(): empty actor_type.

It logs log_stopped_skipped(self.id, "", "runtime_drop", ..), so the structured event has an empty actor_type. Either:

• capture std::any::type_name::<A>() when constructing ActorCleanup, or
• drop the field for this cause.

8. lifecycle_trace.rs reason(): wildcard arm.

Some(_) => "Kill" silently maps any future ExitReason variant to Kill in Supervision.tla traces. Inside the crate you can match exhaustively, as life_reason does, so a new variant forces a decision.

9. lib.rs: inconsistent macro re-exports.

The root re-exports send_handler but not request_handler/deferred_handler. If this is only for manual impl Handler use, add a comment. Otherwise re-export all three for consistency.

10. monitor.rs MonitorCleanup: sorting is now unconditional.

sort_unstable_by_key previously ran only under cfg(spawned_sim) and now runs on every actor exit with monitors. The cost is small, but keep the cfg if determinism only matters for the simulator.

Benches

• baseline.rs can hang. Deadline::Never, combined with rx.recv() and an untimed block_on, removes the old 10s guard. A lost reply now hangs the bench forever. Matching 0.7's request_raw is fine, but keep a wall-clock guard on the wait, e.g. tokio::time::timeout or recv with a test-side timer.
• Script-generated handler bodies. The bench handlers (let __reply_value: T = {...}; reply.reply(__reply_value);, extra { { ... } } blocks) look like pasted macro expansion. They would read better as reply.reply(expr) or by using #[send_handler] on the manual impls, which the CHANGELOG says now works.
• Formatting. workload_research.rs around line 690: the .send_request(Route {..}, ..) block is not rustfmt-clean, so rustfmt bailed on the long line. Extract a local.

Process

• Changelog. CHANGELOG.md has several overlapping "Breaking (0.8)" bullets, two marked "in progress" with a "Remaining:" list. Consolidate before release.
• TLA+ coverage. Dropping MCLifecycleDeep and MCLifecycleLiveDeep from the scheduled CI moves them to a manual pre-release step. That is reasonable given the more than 8h runtime, but the release checklist in docs/TESTING.md should require recorded evidence (TLC output and commit) so a skipped run is visible.
• Size. At this size (stacked on rank 11), review confidence is limited. Splitting the bench/test migration churn from the runtime changes would help future bisects.

---

Automated review by Claude · claude-opus-5-5 · custom prompt

[unbalancedparentheses]

/claude

Please independently review the two new performance commits after 7f609bf (reply/turn bookkeeping and await-free generated dispatch). Keep this PR draft; no merge or release yet.

Focus on the zero-external-token turn transition and last-token retirement, reply/defer races, macro source compatibility and name hygiene, parameter/destructor ordering, lazy inherent/Handler calls, and preservation of panic/death fences and causal actor context. Direct dispatch is restricted to messages with no drop glue; shared/explicit-lifetime receivers, suspending/deferred handlers and opaque macros/attributes keep their original adapter. Precise clock reads and the +3% instruction gate are unchanged.

Local review found shared-receiver, ignored-parameter and shadowed Option regressions; these are fixed with regressions. The focused tests/Clippy, all 45 reply/turn/job Loom tests, 270 simulated tests and all 94 recorded traces pass. Negative controls/coverage and the final versioned quiet performance campaign are still running or pending. Please report any diff/context truncation explicitly rather than claiming a full review if the workflow cannot supply the whole change.

[github-actions]

🤖 Claude Code Review

⚠️ Warning: Diff was truncated from 63915 to 10000 lines. Some changes were not reviewed.

Review: PR #240 (exit signals, death point, deferred replies and jobs)

Overall

This is a very large PR. It mixes OTP exit-signal semantics, the death-point refactor, deferred replies and jobs, the deadline service, CI changes and a rewrite of every benchmark. It cannot be reviewed thoroughly as one unit. Consider splitting it into at least these PRs:

1. Link and exit-signal core (link.rs, link/table.rs, monitor/fence.rs, actor_cleanup.rs, error.rs)
2. Jobs and the deadline service (reply/*)
3. Benchmark and example migrations
4. CI and workflow changes

The core design is careful:

• Receiver-side fenced delivery uses edge generations.
• DropFailures is held until every caller is answered.
• Every user payload is moved out before being dropped outside locks.
• Loom, TLA+ and trace coverage are thorough.

Specific findings follow.

---

Correctness / concurrency

1. link.rs link_in_section: TOCTOU against "exit began"

if lock_request(&own.request).reason().is_some() {
    return LinkOutcome::Unchanged;
}
fail_point!("link.before_install");
...
let outcome = table::install(own, own_links, peer, peer_links, gen, try_only);

The comment says the check is "checked again under the table lock below". However, table::install only re-checks a.exited, which is past D, not whether exit began. An exit that starts between the check and install still gets a live record.

The record is cleaned up at D, so this is not a leak. The actual guarantee is narrower than documented, though, and exit_signals_sim / link_fence_loom may not cover this window. Do one of:

• Re-read reason() inside with_pair, taking the request lock under the table lock. This follows the table → request order already used by take_record.
• Reword the doc so it does not promise this.

2. Context::try_link from an exiting caller returns success

The test link_from_an_exiting_or_dead_caller_is_inert confirms that register_link(.., try_only = true) returns Unchanged when the caller is exiting or dead. If try_link maps Unchanged to Ok(()), a caller cannot tell "linked" apart from "nothing happened".

The no-op matches OTP's behaviour for a dead caller. Still, document it explicitly on try_link, or return a distinct value.

3. Lock order is only implicit

There are two cross-actor lock orders:

• table.rs take_record: holds the receiver's link-table lock, then takes the receiver's request lock and enqueues to its mailbox.
• fence.rs register: holds the observer's monitor-table lock, then target_dead() takes the target's link-table lock.

Today nothing takes a link-table lock and then a monitor-table lock, so this is sound. However, the global order is only described in two separate module headers.

Suggestions:

• Put one global ordering comment in a single place: monitor table → link table → exit request → mailbox admission.
• Consider a debug-only lock-rank assertion. A future change to send_exit or wake, for example a callback that touches monitors, would deadlock silently.

4. apply_signal: parent_exit_queued stays set when the send fails

In the trap branch with from_parent, the flag is set before send(exit), and the send result is ignored. If admission is already closed, exit_pending() reports true with no envelope behind it.

This is harmless past D. In the window before D it is a spurious positive. Clear the flag when send returns Err.

5. Keyed job map can grow without bound (job_core.rs, JobSetCore::ended)

"The key keeps its finished job until replaced"

JobKey::owned(String) invites per-request keys such as format!("fetch-{id}"). Each unique key then pins an Arc<JobState> for the actor's whole lifetime, because keys is only cleared in cancel_all. That is a memory leak for long-lived actors.

Options:

• Remove the entry when a finished job's terminal message is dispatched and its queue is empty.
• At minimum, document that keys must come from a bounded set.

6. JobSet::join_blocking uses std::time::Instant

The rest of the job and deadline code uses spawned_rt::time::Instant, which is the simulated clock under spawned_sim. A blocking join under sim therefore uses wall time. Confirm this is intentional, or switch it to the runtime clock.

7. actor_cleanup.rs reach_death → replies.fence() may block the drop path

The fence waits for in-flight outgoing replies and signals. On the runtime-drop path, ActorStartup::drop and ActorCleanup::drop run inside Tokio's shutdown or on a worker thread. This is fine as long as every in-flight sender is synchronous and never parks waiting for this actor.

Please add a comment that asserts this invariant. A future async signal path would turn the fence into a deadlock during runtime drop.

---

Performance

8. Cache default_max_native_jobs() (reply/jobs.rs)

std::thread::available_parallelism() reads cgroup and /proc files on Linux, which makes it a few syscalls. Bounds::resolve calls it through unwrap_or_else whenever MAX_NATIVE_JOBS is unset. If that happens on every actor spawn, cache the result in a OnceLock<usize>.

9. Benchmarks no longer compare like with like

baseline.rs deliberately uses Deadline::Never, with the comment "so the comparison with 0.7 measures the same operation". Several other benchmarks migrated differently:

Benchmark	New deadline	Old behaviour
runtime_isolation.rs	DEFAULT_REQUEST_TIMEOUT (5 s)	request_raw, no deadline
workload_research.rs	DEFAULT_REQUEST_TIMEOUT (5 s)	request_raw, no deadline
designs.rs	Duration::from_secs(10)	request_raw, no deadline

This has two effects:

• It adds per-request timer and deadline bookkeeping to the measured path, which skews any comparison with 0.7.
• Under overload experiments, it turns slow replies into unwrap() panics.

Use Deadline::Never in these benchmarks for consistency.

10. baseline.rs threads reply can now hang forever

Self::Threads(rx) => rx.recv().expect("reply timeout/disconnected"),

The old code used recv_timeout(10s). With Deadline::Never, a lost reply hangs the benchmark until the CI timeout. Keep a wall-clock guard here, or use a deadline only on this wait.

11. benchmarks/hotpath.py timeouts produce a raw traceback

timeout=180 and timeout=120 raise an uncaught subprocess.TimeoutExpired. Catch it and call sys.exit(f"... timed out") so the gate output stays readable. Also confirm that the sh() helper forwards the timeout keyword.

---

Smaller items

• actor_cleanup.rs discharged(): log_stopped_skipped(self.id, "", "runtime_drop", ...) logs an empty actor_type. Thread type_name::<A>() through, or omit the field.
• reply/job_core.rs finish: on the Refused and early-return paths, later and enqueue are dropped inside JobState::finish's trace::atomic section. This only happens with the lifecycle-trace feature, but it runs user payload destructors under the global trace mutex, which contradicts the module's stated rule.
• reply/deadline.rs DeadlineHeap::push → compact: compaction calls pending_deadline() while the heap lock is held. That likely takes the slot lock, giving the order heap → slot. Make sure no path holds a slot lock while pushing.
• Benchmarks: hand-written handlers now contain macro-expansion artifacts such as let __reply_value = { ... } and an empty { ... }; reply.reply(()). They read as pasted macro output. Simplify them, or use #[actor] / #[send_handler] where possible.
• verification.yml: MCLifecycleDeep and MCLifecycleLiveDeep are removed from scheduled CI and become manual pre-release checks. Make sure docs/TESTING.md's release checklist is actually enforced. The release.yaml comment alone will not stop someone skipping it.
• CHANGELOG:
  • It still lists several "in progress" breaking entries ("Remaining: the heavy TLC configurations…"). Reconcile these before release.
  • There is a stray blank line after the timer entry.
• link_regression_tests.rs late_link_failure_relabels_...: this behaviour change is significant for users. A linked survivor of a cleanly stopping actor now cascades if a third actor crashes during stopped(). Call it out prominently in MIGRATION.md, not only in the changelog.

---

Looks good

• Exit request model: ExitRequest is a clean single place for reason precedence: kill > untrapped > request, with the hook-skip rules. The unit test reason_precedence_until_death is clear.
• Death-point ordering: in ActorStartup::drop, the order D → discharge → wake-ups → per-item contained drops → payload → publish is correct. It also avoids double-unwind aborts by dropping each item individually.
• DropFailures::hold / release: this removes a whole class of "user destructor runs before the caller is answered" bugs.
• JobSetCore: admission and listing happen in one critical section with closed, so no admitted job can escape cancel_all.
• Error classification: ActorError classification helpers, plus an exhaustive in-crate test table for every variant, are a good pattern for #[non_exhaustive] enums.

---

Automated review by Claude · claude-opus-5-5 · custom prompt