Repository navigation
perf(tm2): save a block with one fsync instead of two - #6264
Conversation
🛠 PR Checks SummaryAll Automated Checks passed. ✅ Manual Checks (for Reviewers):
Read More🤖 This bot helps streamline PR reviews by verifying automated checks and providing guidance for contributors and reviewers. ✅ Automated Checks (for Contributors):No automated checks match this pull request. ☑️ Contributor Actions:
☑️ Reviewer Actions:
📚 Resources:Debug
|
## What During block sync a node waits for the disk five times per block, across three databases: the block store (twice), the ABCI results, the application's commit and the state record. This PR holds all three databases' writes in memory during block sync and writes them at **flush points**, in the order that leaves them consistent after a crash at any instant: 1. the block store, with a record of the window it writes, 2. the state database, 3. the application, each as one synced, all-or-nothing batch; then the window's events fire, in block order. Consensus mode keeps its per-block flushes. Three changes reach it too: the node's stop order (below), a failed write of the block store's height record now panics instead of being ignored, and a commit's batch carries only the last write to each key. A flush point is taken between blocks only: | when | why | |---|---| | every second (`flushPointInterval`), also when no block arrives | bounds what a crash or a stop costs again | | a database holds more than 64 MiB (`flushPointBytes`), the application's bytes included | memory, and each backend's limit for one write | | the application's halt height is reached | the halt height is on disk before the `BeginBlock` that panics | | a block wrote a halt param | the upgraded binary's startup check reads it from disk | | before the switch to consensus | no consensus block lands in a held database | Follow-up to #6316's review: removing single flushes one by one breaks what each protects, so this keeps every flush's guarantee and changes when things are written, not what is protected. The design went through seven rounds of adversarial review before the code, and the ADR records each rule with the code it relies on. ## What a crash leaves Between flush points, nothing of the window is on disk: all three databases are at the last flush point F, and block sync fetches the window again. Inside a flush point F → F': | crash after | block store | state | app | restart | |---|---|---|---|---| | nothing written | F | F | F | nothing to replay | | step 1 | F' | F | F | **new:** the handshake lowers the block store to F+1 (only when the flush point's record says F to F' and the app is at most at F), then replays F+1 on the real app, an existing case | | step 2 | F' | F' | F | replays F+1..F' on the real app without changing the state (existing case) | | step 3 | F' | F' | F' | nothing to replay | Any other shape, such as a partial restore, panics as today. Events stay at most once. ## The pieces - **tm2/pkg/db:** `HoldingDB` wraps the block store's and the state's databases where the node opens them; it passes every call through until block sync holds it. `BatchCollector` gains the holding switch (shared with rootmulti), a byte count, `WriteTo` (one synced batch, ops dropped only once it is written) and a record of iterators refused while holding: an iterator would miss the held writes, so it panics with `ErrIteratorOverHeldWrites`. `set` now keeps an empty value non-nil. - **tm2/pkg/bft/blockchain:** `DeferredWrites` takes the flush points; the reactor starts and stops it, checks for a flush point after each block and on its ticker, and switches to consensus only after a flush point and only if its own `pool.Stop` succeeds, so a stop never starts consensus. `OnStop` waits for the routine, which also returns when the pool stops. - **tm2/pkg/bft/consensus:** the handshake's rewind after a crash in step 1. - **tm2/pkg/bft/store, state:** `LowerHeight`, the flush-point record, the height record's write error checked; `BlockExecutor` holds events until the flush point. - **tm2/pkg/bft/node:** wraps the databases, hands the in-process app to block sync if it implements `abci.DeferredCommitter` (new, 4 methods), and stops the block-sync reactor first, then the p2p switch, then the RPC environment, the event switch and the event store: an event fired after the event switch stopped panics on a closed channel. - **tm2/pkg/sdk, store:** BaseApp implements `DeferredCommitter` behind a new option, `AllowDeferredCommits` (off by default: the IAVL store iterates its database inside a block). While holding, rootmulti keeps its commits in its collector; turning holding on first writes what its stores collected outside a commit (the bptree fast index's rebuild at load), so a restart at the tip can switch to consensus with no block. Info, queries and `Simulate` answer from the version on disk; a fresh node refuses store and custom queries and `Simulate` until its first flush point; an iterator inside a block ends the block even if a recover swallowed its panic. - **gno.land:** turns the option on (its bptree and plain stores never iterate the database inside a block) and asks for a flush point when a delivered block writes `p:halt_height` or `p:halt_min_version`. - **contribs/gnobr:** refuses, before trimming anything, to roll back above its state a directory whose block store is more than one block ahead of it, as a crash during a flush point can leave it: the state's hashes would be patched from a later block. Roll back to the state or below, or start gnoland once first. ## Measurements or estimates Four syncs of gno.land mainnet (gnoland-1) from genesis to height ~25,000, the light-block range after the genesis transactions, on a 16-vCPU DigitalOcean droplet (NVMe, pebbledb) with the same peers, interleaved: master, this PR, master, this PR, from binaries built from master `21f9abd` and from this PR with Go 1.25.9. Rate = blocks per second from the first 10-second `/status` sample at or above height 3,400 to the last one; the minutes each run spends replaying the genesis transactions and filling its block pool are left out. Other jobs share the box, so its load average is shown. | run | build | blocks/s | ms per block | load average | |---|---|---|---|---| | 1 | master | 102 | 9.8 | 8.1 | | 2 | this PR | 296 | 3.4 | 9.9 | | 3 | master | 119 | 8.4 | 2.9 | | 4 | this PR | 310 | 3.2 | 2.8 | So light blocks sync about 2.6-2.9× faster (8.4-9.8 → 3.2-3.4 ms per block), close to the ~2.6 ms per block measured earlier with every flush disabled. A transaction-heavy block costs 10-15 ms of CPU, so the same ~5-6 ms saving is a smaller share there; over the whole chain the ADR estimates about 1.4-1.5× against master. That part is not measured: a full resync takes hours. **A real crash test**, on the data run 4 left: the PR's binary synced on and was killed with `kill -9` mid-sync five times, then stopped cleanly and started once more. - One kill landed between a flush point's first two writes: the block store was at 34,751 and the state at 34,516. The restart logged "Lowering the block store after a crash in a block-sync flush point" (34,751 → 34,517), replayed that block on the real app and synced on. - That node then reached gnoland-1's governance halt at 36,300 and halted, as every node does. The next start found the application, the block store and the state all at 36,300, so the halt height was on disk before the panic, and synced on. - No log showed any other panic, an app hash mismatch or a consensus failure. ## Tests Every rule has a test that fails without it, but for the node's stop order (it needs two nodes and a stop inside a flush point; checked by review). A mutation run broke one rule at a time, 73 mutants in all, and a test failed on each. - `TestHandshakeAfterFlushPointPowerCut` (`tm2/pkg/bft/consensus`): syncs eight blocks with deferred writes, in windows of four, over disks that log every write and batch, and cuts the power at every step of both flush points. At each cut it restarts, through the real handshake, on every combination of what can survive (the writes the last flush covered, all or none of each later batch, the app at any height it last wrote or lower), then syncs on with deferred writes. Each run must end with the uncut run's state, every key of both databases, the app's height and hash, and no events fired before their block was durable. The chain changes its validator set in the block a restart replays after a crash in step 1, changes its consensus params, has an empty block, and starts at height 1 or 5. `TestLowerAfterFlushPointCrash` covers each condition of the rewind. - `tm2/pkg/bft/blockchain`: two nodes over p2p. Writes are held, and on disk with nothing held at the switch; an app that refuses is written per block; a stop during the switch's flush point waits for it and starts no consensus; a stop while behind writes nothing; a window whose only peer leaves is written on the ticker; each trigger; each step's write and record; a failed write at each step panics before the next. - `tm2/pkg/bft/node`: a node whose app defers holds its databases during block sync, and stops while behind. - `tm2/pkg/db`, `tm2/pkg/store/rootmulti`, `tm2/pkg/sdk`, `tm2/pkg/bft/store`, `tm2/pkg/bft/state`: the holding database, the collector's synced batch and its byte count, held commits and the version on disk, a real fast-index rebuild at load written before holding, queries, Info and `Simulate` from disk (after a restart and after the genesis commit too), the fresh-node refusal, the halt height and the required flush point, the refused iterator in a transaction and in the blockers but not one left by `CheckTx`, `Close` writing nothing, an empty block's held results, the interval running from the end of a slow flush point. - `gno.land/pkg/gnoland`: `TestAppDefersCommits` runs real blocks with commits held, deploys a realm at `r/sys/params` and calls it to write the halt height, which asks for a flush point that puts it on disk; `TestAppDefersCommitsAfterRestart` starts and stops deferring on a restarted app with no block between; `TestNodeParamsKeeperRequiresFlushPoint`: deliver mode only, halt params only, after validation. - `contribs/gnobr`: `TestCheckRollbackable`, each shape and target. - On a 16-vCPU box with Go 1.25.9 (CI's toolchain): `make fix` changes nothing; golangci-lint with the repo's config reports 0 issues on `tm2`, `gno.land/pkg/gnoland` and `contribs/gnobr`; `go test ./...` in `tm2` passes, except the privval file-permission tests, which fail there as root on master too; `gno.land/pkg/gnoland/...`, `pkg/sdk/...`, `cmd/...`, `contribs/gnobr` and `gno.land/pkg/integration` pass; the block-sync tests pass five times in a row under `-race`. ## Files **Block sync (tm2/pkg/bft)** - `blockchain/deferred.go` (new): `DeferredWrites`, the flush points and their triggers - `blockchain/reactor.go`: `SetDeferredWrites`, the flush-point checks after each block and on the ticker, the switch only after a flush point and a successful `pool.Stop`, the pool-quit case, `OnStop` waits for the routine - `state/execution.go`: `HoldEvents`, `FireHeldEvents` - `store/store.go`: `DB`, `LowerHeight`, the `FlushPoint` record, the height record's write error checked - `consensus/replay.go`: the rewind after a crash between a flush point's first two writes - `node/node.go`: the holding databases, the deferral wiring, the stop order - `proxy/client.go`: `LocalApp` - `abci/types/application.go`: `DeferredCommitter` **Databases and the application (tm2/pkg/db, store, sdk)** - `db/holding.go` (new): `HoldingDB` - `db/collecting.go`: the holding switch, the byte count, `WriteTo`, `Drain` skipping overridden ops, the refused-iterator record, empty values kept non-nil - `store/types/store.go`, `store/exports.go`: `CommitHolder` - `store/rootmulti/store.go`: held commits, `WriteHeld`, the version on disk, the live query refused while holding - `sdk/deferred.go` (new): BaseApp's `DeferredCommitter`, `RequireFlushPoint`, the version on disk, the refusal checks - `sdk/baseapp.go`: the held header, Info and queries from the version on disk, the checks around the blockers and `DeliverTx` - `sdk/helpers.go`: `Simulate` from the version on disk, refused before the first flush point - `sdk/options.go`: `AllowDeferredCommits` **p2p** - `tm2/pkg/p2p/switch.go`: skips reactors already stopped **gno.land** - `gno.land/pkg/gnoland/app.go`: turns deferral on, wires the hook - `gno.land/pkg/gnoland/node_params.go`: asks for a flush point when a delivered block writes a halt param **Tools** - `contribs/gnobr/main.go`: refuses a directory whose block store is ahead of its state by more than one block **Tests** - new: `tm2/pkg/bft/consensus/replay_flush_point_test.go`, `tm2/pkg/bft/blockchain/deferred_test.go`, `tm2/pkg/bft/node/deferred_test.go`, `tm2/pkg/bft/store/flushpoint_test.go`, `tm2/pkg/bft/state/held_responses_test.go`, `tm2/pkg/db/holding_test.go`, `tm2/pkg/store/rootmulti/holding_test.go`, `tm2/pkg/sdk/deferred_test.go` - extended: `tm2/pkg/db/collecting_test.go`, `gno.land/pkg/gnoland/app_test.go`, `contribs/gnobr/apphash_test.go` **Docs** - `tm2/adr/pr6334_deferred_block_sync_writes.md`: the ADR (renamed after this PR's number in a second commit) Not in the repo: the mutation runner (69 mutants, a Python script with each mutant's pattern and the tests expected to catch it) and the sync-timing script; available on request. ## Related - #6316 (drop the block store's redundant flush) and #6264 (save a block with one fsync) change `SaveBlock` for consensus mode; during block sync this PR holds those writes anyway. They touch `tm2/pkg/bft/store/store.go` too, so whichever lands second needs a small merge. - #6308 and #6315 cut the CPU per block; the gains add up. - CometBFT #5753 verifies each commit once instead of twice during block sync, a separate saving that would apply here. ## AI assistance Opened by an AI coding agent (Claude Code) at the request of, and on behalf of, its human owner @jaekwon, who is responsible for it. The ADR records the context the agent worked from. Opened as a draft; marked ready after CI and a self-review. --------- Co-authored-by: Morgan Bazalgette <morgan@morganbaz.com>
6c8a2db to
a280399
Compare
|
Rebased onto master at Conflict: one, in
Gate (lyra, |
|
[note: ai-assisted] Severity: 2/10, low Nit: the description's +21.5% block-sync speedup was measured on v1.5.0 and predates #6334. On master, block sync holds fsyncs per SaveBlock, master vs this PR
package store
import (
"testing"
"github.com/gnolang/gno/tm2/pkg/bft/types"
tmtime "github.com/gnolang/gno/tm2/pkg/bft/types/time"
dbm "github.com/gnolang/gno/tm2/pkg/db"
"github.com/gnolang/gno/tm2/pkg/db/memdb"
"github.com/gnolang/gno/tm2/pkg/log"
)
// revCountDB counts writes that reach the disk with a sync.
type revCountDB struct {
dbm.DB
syncs int
}
func (db *revCountDB) SetSync(k, v []byte) error { db.syncs++; return db.DB.SetSync(k, v) }
func (db *revCountDB) DeleteSync(k []byte) error { db.syncs++; return db.DB.DeleteSync(k) }
func (db *revCountDB) NewBatch() dbm.Batch { return &revCountBatch{db.DB.NewBatch(), db} }
func (db *revCountDB) NewBatchWithSize(n int) dbm.Batch {
return &revCountBatch{db.DB.NewBatchWithSize(n), db}
}
type revCountBatch struct {
dbm.Batch
db *revCountDB
}
func (b *revCountBatch) WriteSync() error { b.db.syncs++; return b.Batch.WriteSync() }
// Mirrors node.go's wiring: the block store's DB is a HoldingDB over the disk.
func TestReviewSaveBlockSyncs(t *testing.T) {
state, _, cleanup := makeStateAndBlockStore(log.NewNoopLogger())
defer cleanup()
disk := &revCountDB{DB: memdb.NewMemDB()}
h := dbm.NewHoldingDB(disk)
bs := NewBlockStore(h)
save := func(height int64) {
b := makeBlock(height, state, new(types.Commit))
bs.SaveBlock(b, b.MakePartSet(2), makeTestCommit(height, tmtime.Now()))
}
// Live node (consensus): writes pass through.
save(1)
t.Logf("not held (consensus): syncs for 1 SaveBlock = %d", disk.syncs)
// Block sync since #6334: writes held until a flush point.
disk.syncs = 0
h.Hold(true)
for height := int64(2); height <= 11; height++ {
save(height)
}
t.Logf("held (block sync): syncs for 10 SaveBlocks = %d", disk.syncs)
if err := h.WriteHeld(); err != nil {
t.Fatal(err)
}
h.Hold(false)
t.Logf("held (block sync): syncs after the flush point = %d", disk.syncs)
empty, _ := disk.DB.Has([]byte{})
t.Logf("empty key on disk: %v; reopened height: %d", empty, NewBlockStore(disk.DB).Height())
}Run:
|
BlockStore.SaveBlockflushes to disk twice per block: once insideBlockStoreStateJSON.Save(aSetSync), then again with the trailingbs.db.SetSync(nil, nil)"Flush", which writes an empty key purely to force a second fsync. The first flush already made every earlier write durable (pebble and goleveldb sync one ordered WAL), so the second one buys nothing and costs a full fsync on every height.This PR stages the whole block (meta, parts, both commits, the height descriptor) in one batch and commits it with a single
WriteSync. The save also becomes atomic: a crash can no longer leave a height whose meta is on disk without its parts.BlockStoreStateJSON.Savekeeps its signature and behaviour for other callers.Why it matters
Fast sync applies blocks strictly one at a time, and per block it does five synced writes:
SaveBlock(x2),SaveABCIResponses, the appCommit, andSaveState. A Go execution trace of a node replayinggnoland-1from genesis shows the replay goroutine blocked on those pebble commits for 6.86 s of an 8 s window, split almost evenly across the five call sites, while the process used ~18% of one core. On ordinary hardware, catching up is fsync-latency bound, not CPU bound, so removing one of the five flushes is close to a 20% speedup.Measured
Replaying
gnoland-1from genesis withv1.5.0(same data dir, swapping the binary between rounds, 45 s warmup then 90 s measured, rounds interleaved), pebble backend, Apple Silicon laptop SSD:Disks with slower flushes (SATA, RAID1, no power-loss protection) pay more per fsync, so they gain at least as much. The same path runs once per block in consensus too.
Safety
ApplyBlockruns, exactly as before.Batch.WriteSync. For boltdb, where eachSetwas its own fsynced transaction, this goes from six-plus commits per block to one.saveBlockPartduplicated the one at the top ofSaveBlock, which still runs before anything is staged.Test
TestSaveBlockFlushesOncewraps a memdb and counts synced writes (directSetSync/DeleteSyncand batchWriteSync). It fails onmaster(expected: 1, actual: 2) and passes here.go test ./tm2/pkg/bft/{store,blockchain,state,node,consensus}/...is green.Not in this PR
The remaining four flushes are each load-bearing for crash recovery (the ABCI responses must be durable before the app commits; state after it). Further gains need structural changes, for example pipelining the block-store write of height N+1 with the execution of N, or making the per-block
authparams read (EndBlockerdecodesunrestricted_addrsfrom JSON every block, ~16% of replay CPU at low heights) cheaper.🤖 Generated with Claude Code