Skip to content

perf(tm2): save a block with one fsync instead of two - #6264

Merged
thehowl merged 2 commits into
masterfrom
perf/blockstore-single-fsync
Nov 6, 2026
Merged

thehowl merged 2 commits into
masterfrom
perf/blockstore-single-fsync

Conversation

@moul

@moul moul commented Oct 1, 2026

Copy link
Copy Markdown
Member

BlockStore.SaveBlock flushes to disk twice per block: once inside BlockStoreStateJSON.Save (a SetSync), then again with the trailing bs.db.SetSync(nil, nil) "Flush", which writes an empty key purely to force a second fsync. The first flush already made every earlier write durable (pebble and goleveldb sync one ordered WAL), so the second one buys nothing and costs a full fsync on every height.

This PR stages the whole block (meta, parts, both commits, the height descriptor) in one batch and commits it with a single WriteSync. The save also becomes atomic: a crash can no longer leave a height whose meta is on disk without its parts. BlockStoreStateJSON.Save keeps its signature and behaviour for other callers.

Why it matters

Fast sync applies blocks strictly one at a time, and per block it does five synced writes: SaveBlock (x2), SaveABCIResponses, the app Commit, and SaveState. A Go execution trace of a node replaying gnoland-1 from genesis shows the replay goroutine blocked on those pebble commits for 6.86 s of an 8 s window, split almost evenly across the five call sites, while the process used ~18% of one core. On ordinary hardware, catching up is fsync-latency bound, not CPU bound, so removing one of the five flushes is close to a 20% speedup.

Measured

Replaying gnoland-1 from genesis with v1.5.0 (same data dir, swapping the binary between rounds, 45 s warmup then 90 s measured, rounds interleaved), pebble backend, Apple Silicon laptop SSD:

round heights (base) base blocks/s heights (patched) patched blocks/s
1 48161..52751 51.0 55214..60872 62.9
2 62976..67633 51.7 69897..75438 61.6
3 77814..82184 48.6 84431..89693 58.5
4 91231..96324 56.6 99112..105382 69.7
mean 52.0 63.2 (+21.5%)

Disks with slower flushes (SATA, RAID1, no power-loss protection) pay more per fsync, so they gain at least as much. The same path runs once per block in consensus too.

Safety

  • Ordering against the other stores is unchanged: the block is durable before ApplyBlock runs, exactly as before.
  • Every backend implements Batch.WriteSync. For boltdb, where each Set was its own fsynced transaction, this goes from six-plus commits per block to one.
  • The contiguity check in the removed saveBlockPart duplicated the one at the top of SaveBlock, which still runs before anything is staged.
  • No on-chain or consensus-visible change; only the node's local block store write pattern.

Test

TestSaveBlockFlushesOnce wraps a memdb and counts synced writes (direct SetSync/DeleteSync and batch WriteSync). It fails on master (expected: 1, actual: 2) and passes here. go test ./tm2/pkg/bft/{store,blockchain,state,node,consensus}/... is green.

Not in this PR

The remaining four flushes are each load-bearing for crash recovery (the ABCI responses must be durable before the app commits; state after it). Further gains need structural changes, for example pipelining the block-store write of height N+1 with the execution of N, or making the per-block auth params read (EndBlocker decodes unrestricted_addrs from JSON every block, ~16% of replay CPU at low heights) cheaper.

🤖 Generated with Claude Code

@Gno2D2

Gno2D2 commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

🛠 PR Checks Summary

All Automated Checks passed. ✅

Manual Checks (for Reviewers):
  • IGNORE the bot requirements for this PR (force green CI check)
Read More

🤖 This bot helps streamline PR reviews by verifying automated checks and providing guidance for contributors and reviewers.

✅ Automated Checks (for Contributors):

No automated checks match this pull request.

☑️ Contributor Actions:
  1. Fix any issues flagged by automated checks.
  2. Follow the Contributor Checklist to ensure your PR is ready for review.
    • Add new tests, or document why they are unnecessary.
    • Provide clear examples/screenshots, if necessary.
    • Update documentation, if required.
    • Ensure no breaking changes, or include BREAKING CHANGE notes.
    • Link related issues/PRs, where applicable.
☑️ Reviewer Actions:
  1. Complete manual checks for the PR, including the guidelines and additional checks if applicable.
📚 Resources:
Debug
Manual Checks
**IGNORE** the bot requirements for this PR (force green CI check)

If

🟢 Condition met
└── 🟢 On every pull request

Can be checked by

  • Any user with comment edit permission

thehowl added a commit that referenced this pull request Oct 15, 2026
## What

During block sync a node waits for the disk five times per block, across
three databases: the block store (twice), the ABCI results, the
application's commit and the state record. This PR holds all three
databases' writes in memory during block sync and writes them at **flush
points**, in the order that leaves them consistent after a crash at any
instant:

1. the block store, with a record of the window it writes,
2. the state database,
3. the application,

each as one synced, all-or-nothing batch; then the window's events fire,
in block order. Consensus mode keeps its per-block flushes. Three
changes reach it too: the node's stop order (below), a failed write of
the block store's height record now panics instead of being ignored, and
a commit's batch carries only the last write to each key.

A flush point is taken between blocks only:

| when | why |
|---|---|
| every second (`flushPointInterval`), also when no block arrives |
bounds what a crash or a stop costs again |
| a database holds more than 64 MiB (`flushPointBytes`), the
application's bytes included | memory, and each backend's limit for one
write |
| the application's halt height is reached | the halt height is on disk
before the `BeginBlock` that panics |
| a block wrote a halt param | the upgraded binary's startup check reads
it from disk |
| before the switch to consensus | no consensus block lands in a held
database |

Follow-up to #6316's review: removing single flushes one by one breaks
what each protects, so this keeps every flush's guarantee and changes
when things are written, not what is protected. The design went through
seven rounds of adversarial review before the code, and the ADR records
each rule with the code it relies on.

## What a crash leaves

Between flush points, nothing of the window is on disk: all three
databases are at the last flush point F, and block sync fetches the
window again. Inside a flush point F → F':

| crash after | block store | state | app | restart |
|---|---|---|---|---|
| nothing written | F | F | F | nothing to replay |
| step 1 | F' | F | F | **new:** the handshake lowers the block store to
F+1 (only when the flush point's record says F to F' and the app is at
most at F), then replays F+1 on the real app, an existing case |
| step 2 | F' | F' | F | replays F+1..F' on the real app without
changing the state (existing case) |
| step 3 | F' | F' | F' | nothing to replay |

Any other shape, such as a partial restore, panics as today. Events stay
at most once.

## The pieces

- **tm2/pkg/db:** `HoldingDB` wraps the block store's and the state's
databases where the node opens them; it passes every call through until
block sync holds it. `BatchCollector` gains the holding switch (shared
with rootmulti), a byte count, `WriteTo` (one synced batch, ops dropped
only once it is written) and a record of iterators refused while
holding: an iterator would miss the held writes, so it panics with
`ErrIteratorOverHeldWrites`. `set` now keeps an empty value non-nil.
- **tm2/pkg/bft/blockchain:** `DeferredWrites` takes the flush points;
the reactor starts and stops it, checks for a flush point after each
block and on its ticker, and switches to consensus only after a flush
point and only if its own `pool.Stop` succeeds, so a stop never starts
consensus. `OnStop` waits for the routine, which also returns when the
pool stops.
- **tm2/pkg/bft/consensus:** the handshake's rewind after a crash in
step 1.
- **tm2/pkg/bft/store, state:** `LowerHeight`, the flush-point record,
the height record's write error checked; `BlockExecutor` holds events
until the flush point.
- **tm2/pkg/bft/node:** wraps the databases, hands the in-process app to
block sync if it implements `abci.DeferredCommitter` (new, 4 methods),
and stops the block-sync reactor first, then the p2p switch, then the
RPC environment, the event switch and the event store: an event fired
after the event switch stopped panics on a closed channel.
- **tm2/pkg/sdk, store:** BaseApp implements `DeferredCommitter` behind
a new option, `AllowDeferredCommits` (off by default: the IAVL store
iterates its database inside a block). While holding, rootmulti keeps
its commits in its collector; turning holding on first writes what its
stores collected outside a commit (the bptree fast index's rebuild at
load), so a restart at the tip can switch to consensus with no block.
Info, queries and `Simulate` answer from the version on disk; a fresh
node refuses store and custom queries and `Simulate` until its first
flush point; an iterator inside a block ends the block even if a recover
swallowed its panic.
- **gno.land:** turns the option on (its bptree and plain stores never
iterate the database inside a block) and asks for a flush point when a
delivered block writes `p:halt_height` or `p:halt_min_version`.
- **contribs/gnobr:** refuses, before trimming anything, to roll back
above its state a directory whose block store is more than one block
ahead of it, as a crash during a flush point can leave it: the state's
hashes would be patched from a later block. Roll back to the state or
below, or start gnoland once first.

## Measurements or estimates

Four syncs of gno.land mainnet (gnoland-1) from genesis to height
~25,000, the light-block range after the genesis transactions, on a
16-vCPU DigitalOcean droplet (NVMe, pebbledb) with the same peers,
interleaved: master, this PR, master, this PR, from binaries built from
master `21f9abd` and from this PR with Go 1.25.9. Rate = blocks per
second from the first 10-second `/status` sample at or above height
3,400 to the last one; the minutes each run spends replaying the genesis
transactions and filling its block pool are left out. Other jobs share
the box, so its load average is shown.

| run | build | blocks/s | ms per block | load average |
|---|---|---|---|---|
| 1 | master | 102 | 9.8 | 8.1 |
| 2 | this PR | 296 | 3.4 | 9.9 |
| 3 | master | 119 | 8.4 | 2.9 |
| 4 | this PR | 310 | 3.2 | 2.8 |

So light blocks sync about 2.6-2.9× faster (8.4-9.8 → 3.2-3.4 ms per
block), close to the ~2.6 ms per block measured earlier with every flush
disabled. A transaction-heavy block costs 10-15 ms of CPU, so the same
~5-6 ms saving is a smaller share there; over the whole chain the ADR
estimates about 1.4-1.5× against master. That part is not measured: a
full resync takes hours.

**A real crash test**, on the data run 4 left: the PR's binary synced on
and was killed with `kill -9` mid-sync five times, then stopped cleanly
and started once more.
- One kill landed between a flush point's first two writes: the block
store was at 34,751 and the state at 34,516. The restart logged
"Lowering the block store after a crash in a block-sync flush point"
(34,751 → 34,517), replayed that block on the real app and synced on.
- That node then reached gnoland-1's governance halt at 36,300 and
halted, as every node does. The next start found the application, the
block store and the state all at 36,300, so the halt height was on disk
before the panic, and synced on.
- No log showed any other panic, an app hash mismatch or a consensus
failure.

## Tests

Every rule has a test that fails without it, but for the node's stop
order (it needs two nodes and a stop inside a flush point; checked by
review). A mutation run broke one rule at a time, 73 mutants in all, and
a test failed on each.

- `TestHandshakeAfterFlushPointPowerCut` (`tm2/pkg/bft/consensus`):
syncs eight blocks with deferred writes, in windows of four, over disks
that log every write and batch, and cuts the power at every step of both
flush points. At each cut it restarts, through the real handshake, on
every combination of what can survive (the writes the last flush
covered, all or none of each later batch, the app at any height it last
wrote or lower), then syncs on with deferred writes. Each run must end
with the uncut run's state, every key of both databases, the app's
height and hash, and no events fired before their block was durable. The
chain changes its validator set in the block a restart replays after a
crash in step 1, changes its consensus params, has an empty block, and
starts at height 1 or 5. `TestLowerAfterFlushPointCrash` covers each
condition of the rewind.
- `tm2/pkg/bft/blockchain`: two nodes over p2p. Writes are held, and on
disk with nothing held at the switch; an app that refuses is written per
block; a stop during the switch's flush point waits for it and starts no
consensus; a stop while behind writes nothing; a window whose only peer
leaves is written on the ticker; each trigger; each step's write and
record; a failed write at each step panics before the next.
- `tm2/pkg/bft/node`: a node whose app defers holds its databases during
block sync, and stops while behind.
- `tm2/pkg/db`, `tm2/pkg/store/rootmulti`, `tm2/pkg/sdk`,
`tm2/pkg/bft/store`, `tm2/pkg/bft/state`: the holding database, the
collector's synced batch and its byte count, held commits and the
version on disk, a real fast-index rebuild at load written before
holding, queries, Info and `Simulate` from disk (after a restart and
after the genesis commit too), the fresh-node refusal, the halt height
and the required flush point, the refused iterator in a transaction and
in the blockers but not one left by `CheckTx`, `Close` writing nothing,
an empty block's held results, the interval running from the end of a
slow flush point.
- `gno.land/pkg/gnoland`: `TestAppDefersCommits` runs real blocks with
commits held, deploys a realm at `r/sys/params` and calls it to write
the halt height, which asks for a flush point that puts it on disk;
`TestAppDefersCommitsAfterRestart` starts and stops deferring on a
restarted app with no block between;
`TestNodeParamsKeeperRequiresFlushPoint`: deliver mode only, halt params
only, after validation.
- `contribs/gnobr`: `TestCheckRollbackable`, each shape and target.
- On a 16-vCPU box with Go 1.25.9 (CI's toolchain): `make fix` changes
nothing; golangci-lint with the repo's config reports 0 issues on `tm2`,
`gno.land/pkg/gnoland` and `contribs/gnobr`; `go test ./...` in `tm2`
passes, except the privval file-permission tests, which fail there as
root on master too; `gno.land/pkg/gnoland/...`, `pkg/sdk/...`,
`cmd/...`, `contribs/gnobr` and `gno.land/pkg/integration` pass; the
block-sync tests pass five times in a row under `-race`.

## Files

**Block sync (tm2/pkg/bft)**
- `blockchain/deferred.go` (new): `DeferredWrites`, the flush points and
their triggers
- `blockchain/reactor.go`: `SetDeferredWrites`, the flush-point checks
after each block and on the ticker, the switch only after a flush point
and a successful `pool.Stop`, the pool-quit case, `OnStop` waits for the
routine
- `state/execution.go`: `HoldEvents`, `FireHeldEvents`
- `store/store.go`: `DB`, `LowerHeight`, the `FlushPoint` record, the
height record's write error checked
- `consensus/replay.go`: the rewind after a crash between a flush
point's first two writes
- `node/node.go`: the holding databases, the deferral wiring, the stop
order
- `proxy/client.go`: `LocalApp`
- `abci/types/application.go`: `DeferredCommitter`

**Databases and the application (tm2/pkg/db, store, sdk)**
- `db/holding.go` (new): `HoldingDB`
- `db/collecting.go`: the holding switch, the byte count, `WriteTo`,
`Drain` skipping overridden ops, the refused-iterator record, empty
values kept non-nil
- `store/types/store.go`, `store/exports.go`: `CommitHolder`
- `store/rootmulti/store.go`: held commits, `WriteHeld`, the version on
disk, the live query refused while holding
- `sdk/deferred.go` (new): BaseApp's `DeferredCommitter`,
`RequireFlushPoint`, the version on disk, the refusal checks
- `sdk/baseapp.go`: the held header, Info and queries from the version
on disk, the checks around the blockers and `DeliverTx`
- `sdk/helpers.go`: `Simulate` from the version on disk, refused before
the first flush point
- `sdk/options.go`: `AllowDeferredCommits`

**p2p**
- `tm2/pkg/p2p/switch.go`: skips reactors already stopped

**gno.land**
- `gno.land/pkg/gnoland/app.go`: turns deferral on, wires the hook
- `gno.land/pkg/gnoland/node_params.go`: asks for a flush point when a
delivered block writes a halt param

**Tools**
- `contribs/gnobr/main.go`: refuses a directory whose block store is
ahead of its state by more than one block

**Tests**
- new: `tm2/pkg/bft/consensus/replay_flush_point_test.go`,
`tm2/pkg/bft/blockchain/deferred_test.go`,
`tm2/pkg/bft/node/deferred_test.go`,
`tm2/pkg/bft/store/flushpoint_test.go`,
`tm2/pkg/bft/state/held_responses_test.go`,
`tm2/pkg/db/holding_test.go`, `tm2/pkg/store/rootmulti/holding_test.go`,
`tm2/pkg/sdk/deferred_test.go`
- extended: `tm2/pkg/db/collecting_test.go`,
`gno.land/pkg/gnoland/app_test.go`, `contribs/gnobr/apphash_test.go`

**Docs**
- `tm2/adr/pr6334_deferred_block_sync_writes.md`: the ADR (renamed after
this PR's number in a second commit)

Not in the repo: the mutation runner (69 mutants, a Python script with
each mutant's pattern and the tests expected to catch it) and the
sync-timing script; available on request.

## Related

- #6316 (drop the block store's redundant flush) and #6264 (save a block
with one fsync) change `SaveBlock` for consensus mode; during block sync
this PR holds those writes anyway. They touch
`tm2/pkg/bft/store/store.go` too, so whichever lands second needs a
small merge.
- #6308 and #6315 cut the CPU per block; the gains add up.
- CometBFT #5753 verifies each commit once instead of twice during block
sync, a separate saving that would apply here.

## AI assistance

Opened by an AI coding agent (Claude Code) at the request of, and on
behalf of, its human owner @jaekwon, who is responsible for it. The ADR
records the context the agent worked from. Opened as a draft; marked
ready after CI and a self-review.

---------

Co-authored-by: Morgan Bazalgette <morgan@morganbaz.com>
@moul
moul force-pushed the perf/blockstore-single-fsync branch from 6c8a2db to a280399 Compare November 4, 2026 19:50
@moul

moul commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Rebased onto master at db74a1124 → head a280399c6.

Conflict: one, in tm2/pkg/bft/store/store.go BlockStoreStateJSON.Save, against #6334 (deferred block-sync writes), which made Save panic when its SetSync fails.

Gate (lyra, nice -n 10, GOTOOLCHAIN=auto):

$ cd tm2 && go build ./...                      # OK
$ go vet ./pkg/bft/store/                       # OK
$ go test -run TestSaveBlockFlushesOnce ./pkg/bft/store/
--- PASS: TestSaveBlockFlushesOnce
# same test against master's store.go (branch test file, master impl):
--- FAIL: TestSaveBlockFlushesOnce
        	expected: 1
        	actual  : 2
$ go test -count=1 ./pkg/bft/{store,blockchain,state,node,consensus}/... ./pkg/db/
ok  tm2/pkg/bft/store        0.043s
ok  tm2/pkg/bft/blockchain   8.578s
ok  tm2/pkg/bft/state        0.490s   (+ eventstore/... ok)
ok  tm2/pkg/bft/node        10.475s
ok  tm2/pkg/bft/consensus   60.684s   (includes TestHandshakeAfterFlushPointPowerCut)
ok  tm2/pkg/db               0.174s

@thehowl

thehowl commented Nov 6, 2026

Copy link
Copy Markdown
Member

[note: ai-assisted]

Severity: 2/10, low

Nit: the description's +21.5% block-sync speedup was measured on v1.5.0 and predates #6334. On master, block sync holds SaveBlock's writes until a flush point, so a block costs 0 fsyncs there with or without this PR. The gain left is the atomic save, plus one fsync per block on a live node, which #6316's resync timed at 2.21 → 1.23 ms per SaveBlock.

fsyncs per SaveBlock, master vs this PR

tm2/pkg/bft/store/zz_review_test.go, with the block store wired as in node.go, a HoldingDB over the disk:

package store

import (
	"testing"

	"github.com/gnolang/gno/tm2/pkg/bft/types"
	tmtime "github.com/gnolang/gno/tm2/pkg/bft/types/time"
	dbm "github.com/gnolang/gno/tm2/pkg/db"
	"github.com/gnolang/gno/tm2/pkg/db/memdb"
	"github.com/gnolang/gno/tm2/pkg/log"
)

// revCountDB counts writes that reach the disk with a sync.
type revCountDB struct {
	dbm.DB
	syncs int
}

func (db *revCountDB) SetSync(k, v []byte) error { db.syncs++; return db.DB.SetSync(k, v) }
func (db *revCountDB) DeleteSync(k []byte) error { db.syncs++; return db.DB.DeleteSync(k) }
func (db *revCountDB) NewBatch() dbm.Batch       { return &revCountBatch{db.DB.NewBatch(), db} }
func (db *revCountDB) NewBatchWithSize(n int) dbm.Batch {
	return &revCountBatch{db.DB.NewBatchWithSize(n), db}
}

type revCountBatch struct {
	dbm.Batch
	db *revCountDB
}

func (b *revCountBatch) WriteSync() error { b.db.syncs++; return b.Batch.WriteSync() }

// Mirrors node.go's wiring: the block store's DB is a HoldingDB over the disk.
func TestReviewSaveBlockSyncs(t *testing.T) {
	state, _, cleanup := makeStateAndBlockStore(log.NewNoopLogger())
	defer cleanup()

	disk := &revCountDB{DB: memdb.NewMemDB()}
	h := dbm.NewHoldingDB(disk)
	bs := NewBlockStore(h)

	save := func(height int64) {
		b := makeBlock(height, state, new(types.Commit))
		bs.SaveBlock(b, b.MakePartSet(2), makeTestCommit(height, tmtime.Now()))
	}

	// Live node (consensus): writes pass through.
	save(1)
	t.Logf("not held (consensus): syncs for 1 SaveBlock = %d", disk.syncs)

	// Block sync since #6334: writes held until a flush point.
	disk.syncs = 0
	h.Hold(true)
	for height := int64(2); height <= 11; height++ {
		save(height)
	}
	t.Logf("held (block sync): syncs for 10 SaveBlocks = %d", disk.syncs)
	if err := h.WriteHeld(); err != nil {
		t.Fatal(err)
	}
	h.Hold(false)
	t.Logf("held (block sync): syncs after the flush point = %d", disk.syncs)

	empty, _ := disk.DB.Has([]byte{})
	t.Logf("empty key on disk: %v; reopened height: %d", empty, NewBlockStore(disk.DB).Height())
}

Run: go test ./tm2/pkg/bft/store/ -run 'TestReviewSaveBlockSyncs$' -v -count=1

master db74a11249 this PR a280399c65
live node, 1 SaveBlock 2 1
block sync, 10 held SaveBlocks 0 0
block sync, at the flush point 1 1

@thehowl
thehowl merged commit 1c4ddcb into master Nov 6, 2026
91 checks passed
@thehowl
thehowl deleted the perf/blockstore-single-fsync branch November 6, 2026 11:54
@github-project-automation github-project-automation Bot moved this from 📥 Inbox to ✅ Done in 😎 Manfred's Board Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

📦 🌐 tendermint v2 Issues or PRs tm2 related

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

3 participants