Skip to content

feat(prover-node): tear down checkpoint sub-tree once block proofs are ready (A-1213) - #24982

Open
PhilWindle wants to merge 2 commits into
merge-train/spartan-v5from
phil/a-1213-release-intermediate-rollup-proofs-in-the-orchestrator-once
Open

feat(prover-node): tear down checkpoint sub-tree once block proofs are ready (A-1213)#24982
PhilWindle wants to merge 2 commits into
merge-train/spartan-v5from
phil/a-1213-release-intermediate-rollup-proofs-in-the-orchestrator-once

Conversation

@PhilWindle

@PhilWindle PhilWindle commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

What

Reduces prover-node memory by tearing down each checkpoint's sub-tree orchestrator as soon as its block proofs are captured, instead of retaining it for the whole proof-submission window.

Each CheckpointProver held its CheckpointSubTreeOrchestrator — and every TxProvingState, AVM circuit input, and base/merge/parity proof it carried — alive until the prover was reaped at the end of the submission window, long after the checkpoint's block proofs were produced. Across every proven checkpoint this accumulates and dominates prover-node memory under load.

Now, once the sub-tree's SubTreeResult is captured, the CheckpointProver drops the sub-tree orchestrator. The block-proof outputs survive via the resolved promise; post-completion consumers (the top-tree job, a rebuilt EpochSession, failure upload) read only whenBlockProofsReady() and the prover's own fields (checkpoint, txs, headers, sibling paths), never the sub-tree. Teardown is tracked so whenDone() awaits it and stays idempotent with the cancel/reap path.

Correctness

  • Teardown of the sub-tree runs exactly once: teardownSubTree() captures the orchestrator and nulls the field in a synchronous prefix before any await, so the success-driven teardown and a concurrent cancel/reap can't both call stop().
  • whenDone() awaits the block-proof settlement between runPromise and the teardown check, so it can't report completion in the window where block-level proving has finished enqueueing (runPromise resolved) but the sub-tree proofs — and the teardown they trigger — are still outstanding.

Scope

This reclaims the cross-checkpoint retention of already-proven sub-trees. It does not free the intermediates of checkpoints that are still proving (the in-flight AVM inputs / base-rollup hints / recursive proofs). Freeing those as they are consumed is a separate, more invasive change and is deliberately left as a follow-up.

The broker remains a separate large consumer (A-1215), as does the prover-node's BrokerCircuitProverFacade inline-inputs retention (A-1517).

Testing

  • yarn build, full @aztec/prover-node checkpoint-prover suite.
  • New tests: the sub-tree is released exactly once on completion while the block-proof outputs still resolve; a subsequent reap cancel is a no-op; and whenDone() stays pending until the sub-tree result lands and teardown completes (asserted by calling it before the result resolves).

…ready

Each CheckpointProver kept its sub-tree orchestrator — and every TxProvingState,
AVM circuit input, and base/merge/parity proof it held — alive for the whole
proof-submission window, long after the checkpoint's block proofs were produced.
With a long submission window this accumulates across every proven checkpoint and
dominates prover-node memory under load.

Tear the sub-tree down as soon as its SubTreeResult is captured. The block-proof
outputs survive via the resolved promise; post-completion consumers (the top-tree
job, a rebuilt EpochSession, failure upload) read only whenBlockProofsReady() and
the prover's own fields (checkpoint, txs, headers, sibling paths), never the
sub-tree. Teardown is tracked so whenDone() awaits it and stays idempotent with the
cancel/reap path.
@AztecBot AztecBot added the port-to-next Forward-port this merged PR into next label Jul 25, 2026
@PhilWindle PhilWindle added the ci-full Run all master checks. label Jul 26, 2026
whenDone() awaited runPromise (which resolves once block-level proving is merely
enqueued) then an optional teardownPromise — but teardownPromise is set only when
the sub-tree result lands, which happens after runPromise resolves. So a
whenDone() call in that window returned while proving and teardown were still
outstanding; the existing test masked it by awaiting whenBlockProofsReady()
first. Await blockProofs settlement between runPromise and the teardown check:
on success the result callback resolves blockProofs and synchronously sets
teardownPromise before the await resumes, so teardown is observed; on
failure/cancel blockProofs rejects and teardown is driven by the finally/cancel
paths already awaited. Adds a regression test that calls whenDone() before the
sub-tree result resolves and asserts it stays pending until stop() completes.
@PhilWindle
PhilWindle force-pushed the phil/a-1213-release-intermediate-rollup-proofs-in-the-orchestrator-once branch from b535c8e to e80f746 Compare July 28, 2026 11:07
@PhilWindle PhilWindle changed the title feat(prover-node): release intermediate rollup proofs/inputs once consumed (A-1213) feat(prover-node): tear down checkpoint sub-tree once block proofs are ready (A-1213) Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-full Run all master checks. port-to-next Forward-port this merged PR into next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants