Skip to content

perf(gc): faster store.compact on SQLite and PostgreSQL; fix settle lane and refused seal - #734

Merged
yifanxuaaa merged 34 commits into
layerfs_v2_0_0from
claude/gc-mark-header-edges
Nov 17, 2026
Merged

yifanxuaaa merged 34 commits into
layerfs_v2_0_0from
claude/gc-mark-header-edges

Conversation

@yifanxuaaa

@yifanxuaaa yifanxuaaa commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Makes store.compact faster on both backends and fixes two defects found while verifying it. No API, CLI, SDK or Store format change.

Speed

One sample per arm, release build, macOS host, 1-minute load 5 to 10 unless noted.

SQLite, 9.8 GB Store (9.789 GB live, 83,559 packs; every run frees 1,095,524,352 bytes):

before (6bca86167f) after
compact, nothing to rebuild 45.19 s 0.49 s
compact with rebuild 66.58 s 13.29 s
mark 24.68 s 0.40 s
build 38.45 s 11.79 s
verify 0.17 s 0.41 s
install 3.26 s 0.68 s

PostgreSQL 17 (Docker on loopback, back to back at load 25 to 27, identical receipts):

Store mark before → after statements compact before → after statements
72,641 objects, 1,360 packs 2,718.6 → 386.9 ms 4,489 → 100 3,162.2 → 934.0 ms 4,627 → 232
181,601 objects, 3,400 packs 6,746.7 → 985.0 ms 11,209 → 234 7,998.2 → 2,060.6 ms 11,348 → 379

How

  • Mark, both backends. It reads record headers instead of whole pack bodies, asks for a whole wave of packs in a few calls, and looks objects up in hash maps keyed by a per-map secret instead of ordered maps. Object locations are read unsorted.
  • SQLite rebuild. The fresh file is sized before the load (macOS), packs are copied in 64 MiB windows with one read-ahead request per window, and repeated scans, opens and syncs are merged or dropped.
  • PostgreSQL. Mark's read-only transaction turns the server's JIT off and reads metadata packs and inode values ahead, so its statement count no longer grows with the number of layers. The rebuild copies from an indexed copy set, checks the scratch rows before the Store-wide lock, and deletes dead digest rows by primary key.
  • Profiling. New examples gc_profile (SQLite, per-statement) and pg_gc_profile (PostgreSQL, counted statements). See tools/ci/gc-bench.md.

Defects fixed

Both were already present at 6bca86167f.

  1. A Commit right after a rebuild could fail SQLITE_BUSY. The rebuild's settle cleared its intent in the installed Store without taking the writer lane. It now takes the lane for that one transaction (4c359ef8ca). This was why the acceptance matrix's AC-49 arm failed most runs; it passed 8 of 8 samples with the fix.
  2. A refused seal never returned. When a seal failed before its checked close (hard-linked Store, obstructed checkpoint), closing the connection waited for the lane the seal itself held. store.compact seals through the same path (3190e6c593).

Also: the PostgreSQL doctor test fixture now selects synchronous_commit=off itself; it had asserted the old default and failed since Normal mode (#732).

Checks

At c32a6dcbab, with PostgreSQL 17 for the PostgreSQL tests:

  • 95 test files, one bounded run each: all pass (512 tests). This includes the five files that failed at 6bca86167f (sqlite_compact_matrix, seal_allocation, three postgresql_doctor_*).
  • Workspace test build, Clippy -D warnings, fmt, check_product_boundary.py and the tool unit tests: clean.
  • New tests fail on their parent's source: a_mark_sends_the_same_statements_for_twice_the_layers, the_postgresql_closure_follows_exactly_the_shared_edges, the_settle_waits_for_a_writer_that_holds_the_lane, a_refused_seal_returns_and_frees_the_lane.

Not covered: Linux was not run locally (macOS host); the speed numbers are macOS only.

Production LOC

187,546 → 188,972 (delta +1,426) against layerfs_v2_0_0 at 3fc66bc406: core 40,408 → 40,589 (+181), engine 89,386 → 90,631 (+1,245), control and api unchanged. Counted with tools/production_loc.py --detail. Each commit records its own comparison.

Not in this PR

  • claude/gc-opt-repack (rewriting half-dead packs) stays out: it is an owner decision.
  • Open options, not acted on: a Disposable Store's compaction without fsync (tried, reverted), 16 KiB pages for the fresh file, freeing in place instead of rebuilding.

🤖 Generated with Claude Code

yifanxuaaa and others added 30 commits November 17, 2026 23:45
…#41)

Mark needs one fact about a file payload: its delta base. It found it by
reading and hashing each live pack whole. It now reads the group's record
directory and each record's first 41 bytes.

Changes
- core delta/record.rs: header_base parses a payload record's base from its
  header alone and accepts and refuses exactly what stored_base does.
- core delta/edge.rs (new): RecordDirectory validates and parses a raw
  group's directory once; sparse_directory and sparse_edge answer from the
  bytes a caller holds and name the next range they lack.
- core delta/read.rs: stored_edge is ChainBases::edge over caller-owned
  caches, so the walk's decoded-group cache is shared and a compressed group
  is decompressed once, not once per record.
- engine history/objects/edges.rs (new): per pack and per wave, bounded
  acquisitions of directories and headers; group_directories for the waste
  accounting.
- engine history/objects.rs: the walk takes header edges first and keeps
  the decoding reader for compressed groups and the Ordinary lane; kept
  waste reads group directories instead of group bodies; inode values use a
  map; the wave sort computes each key once.
- engine backend/sqlite/objects/read.rs: the pack prefix read uses
  incremental BLOB I/O. `substr(body,1,?)` made SQLite load every pack body
  whole to return its first bytes.

Quadratic steps removed: a record lookup rescanned its group's directory
(records squared per group, in kept waste and in the first header reader);
inode values were scanned once per ordinal.

Trade-off: header reads are not authenticated, so the mark no longer hashes
uncompressed payload packs as a side effect. The spec's verify step never
required it (no sha256 of either file); spec section 6.2 now says so.

Measured (gc_bench, Disposable profile, warm cache, macOS, one invocation
per binary, load 5 to 8; receipts not committed)
- small, debug, 3 runs per arm, 704 live objects: mark 564.0 -> 50.9 ms
  (none), 578.4 -> 61.2 ms (rebuild); same 3,133,440 bytes freed.
- medium, release, 1 run per arm, 9.789 GB live in 83,559 packs: mark
  45.17 -> 4.79 s (none), 24.68 -> 1.53 s (rebuild); rebuild wall
  66.58 -> 46.16 s; same 1,095,524,352 bytes freed.

Tests
- New core tests/sparse_edges.rs (5): header reader equivalence with
  stored_base including truncations, sparse walk bounds, broken
  directories, and a full 8,191-record group answered in one pass.
- Pass: layerfs-storage (122); persistence kept_objects, kept_node,
  compact_metrics, sqlite_compact, sqlite_compact_crash,
  sqlite_compact_matrix, sqlite_compact_units, space_reclamation,
  acquisition_reads, acquisition_batching, sqlite_group_rows; PostgreSQL
  gc_mark, compact, compact_fixtures, compact_units, identical_ids,
  gc_fence against a local PostgreSQL 17.
- No test deleted.

Production LOC: 187394 -> 187909 (delta +515)
Scope and method: tools/production_loc.py at this commit, first parent
6bca861 against the committed tree; core 40408 -> 40589, engine
89386 -> 89720, control 51073 -> 51073, api 6527 -> 6527; reference absent
(retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…roject#41)

Finding where a mark or a compaction spends its time needed more than the
aggregate counters: which statement, how often, how long.

- session/thread_work.rs: profile_statements() starts a per-thread profile
  keyed by statement text (whitespace collapsed, first 110 characters);
  take_statement_profile() ends it and returns it costliest first. Off, a
  statement pays one thread-local read.
- session/prepared.rs: a statement lease keeps its text only while its
  thread profiles, and notes each preparation and execution.
- examples/gc_profile.rs: on a Store gc_bench kept, runs N marks and
  optionally one compaction, printing wall time, this thread's SQL work
  counters and the costliest statements.
- tools/ci/gc-bench.md: how to use it and what it has shown.

No test added: the example is a diagnostic with no receipt. No test deleted.

Production LOC: 187909 -> 187965 (delta +56)
Scope and method: tools/production_loc.py at this commit, first parent
d19d98f against the committed tree; core 40589 -> 40589, engine
89720 -> 89776, control 51073 -> 51073, api 6527 -> 6527; reference absent
(retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…(layerfs-project#41)

The profile of a mark over 8,816 packs showed two statements and two BLOB
handles per pack: one acquisition for the record headers and one prefix read
for the kept-waste pass.

- backend/seam.rs: Mechanics::pack_prefixes reads the bounded prefixes of
  several packs in one call (default: one pack_prefix each).
- backend/sqlite: one BLOB handle moved from row to row, so a pack costs no
  statement and no handle of its own. The single prefix read shares the same
  head reader.
- history/objects/edges.rs: a wave reads the first 1,024 bytes of each of its
  packs, 256 packs per call. A pack whose directory and record headers sit in
  that head is answered from it; a wider directory is read again, and only a
  pack that needs more ranges takes a scoped acquisition.
- history/objects.rs: the walk keeps each pack's group extents and the
  kept-waste pass uses them instead of reading the directory again; the
  scope hash used for metadata objects is computed once per walk.

PostgreSQL keeps the default pack_prefixes (one statement per pack); a
batched statement there is a follow-up.

Measured (gc_profile, release, warm cache, macOS, load about 4, one run; a
1.13 GB gc_bench Store of 8,913 objects in 8,816 packs):
  mark wall 88.8 -> 36.3 ms; statements 17,740 -> 298; BLOB handles
  17,633 -> 229; BLOB reads 35,267 -> 9,104; bytes read 40.4 -> 9.0 MB.
The same binary on the 28 MB small Store (737 objects in 158 packs of up to
six groups): mark 6.8 ms, 172 statements, 150 handles; packs of several
groups still take one acquisition each.

Tests: kept_objects, kept_node, compact_metrics, sqlite_compact,
sqlite_compact_crash, sqlite_compact_matrix, space_reclamation,
postgresql_gc_mark, postgresql_compact, postgresql_compact_fixtures pass.
No test added or deleted.

Production LOC: 187965 -> 188037 (delta +72)
Scope and method: tools/production_loc.py at this commit, first parent
against the committed tree; core 40589 -> 40589, engine 89776 -> 89848,
control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c;
last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…layerfs-project#41)

The §7 build writes the fresh Store one page per pwrite. A CPU sample of
the 1.1 GB rebuild put 70% of the copy in pwrite: on APFS a page write
that also moves the end of the file costs about four times one that
lands inside a sized, allocated file (1 GiB of 4 KiB writes: 1.05 s
extending, 0.58 s into a sized hole, 0.28 s sized and preallocated;
ext4 in the Docker VM showed no difference between the three).

The run already knows the retained payload (the mark's packs and their
lengths) before it copies a byte, so the build now sizes the fresh file
for it once, after the schema exists and before the load:

- compact/space.rs `reserve`: macOS only. One advisory F_PREALLOCATE and
  one set_len, rounded down to whole pages, never shrinking. A refused
  allocation leaves a sparse file of the same size; a load that is
  really out of space still meets its own Capacity refusal.
- compact/space.rs `trim`: after the checked close the file is cut to
  the pages its header counts. The size is a lower bound of the load
  today (a pack body needs at least its length in pages), so nothing is
  cut; a build that writes less than the mark's packs (a repack of
  partly dead packs) would otherwise install zero pages past the
  database's end.
- build.rs: `BuildInput.payload_bytes` carries the retained pack bytes
  from run.rs; the page count is read once the schema is complete.

No statement is added to the copy; one PRAGMA page_count is added to
the build. The installed file is byte-for-byte the size it was
(1,134,886,912 bytes on the 1.1 GB shape, before and after).

Measured, release, macOS APFS, one sample each, gc_bench --runs 1:
- small (28 MB): rebuild 122.7 -> 106.3 ms, build 51.9 -> 36.9 ms
  (1-minute load 3.3 before, 4.8 after).
- 1.1 GB (GC_BENCH_GROUPS=96 ... OBJECT_BYTES=131071): rebuild
  1749.9 -> 1039.2 ms, build 1550.7 -> 833.2 ms; the pack copy
  statement alone 1305 -> 548 ms (load 3.4 before, 4.8 after).
A cold source is not helped by this: its copy is bound by reads.

Tests: new sqlite_compact_units
`a_sized_fresh_file_loads_and_is_cut_back_to_its_page_count`. Pass:
sqlite_compact (33), sqlite_compact_crash (15), sqlite_compact_matrix
(1), sqlite_compact_units (8), compact_metrics (5), space_reclamation
(3), kept_objects (7), kept_node (13); sqlite_compact_capacity has no
macOS test. No test deleted.

Production LOC: 187,909 -> 187,965 (delta +56)
  core     40,589 -> 40,589 (0)
  engine   89,720 -> 89,776 (+56)  layerfs-persistence +56
  control  51,073 -> 51,073 (0)
  api      6,527 -> 6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent d19d98f and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rfs-project#41)

One INSERT ... SELECT over every retained pack reads the sealed file in
the order SQLite stored it: a pack's row, then its payload pages a few
dozen pages behind that row, then the next row. No kernel read-ahead
follows that pattern, so a source that is not in memory is read one
pack per device round trip. On a 1.26 GB Store written past the buffer
cache (F_NOCACHE) the pack copy took 3.4 to 3.5 s, against 0.55 s for
the same Store in memory; a plain sequential read of that file runs at
4.4 GB/s cold. The 9.8 GB bench shape was in this state (its first
mark ran 3x slower than its second), which is where its 225 MB/s build
came from.

- compact/window.rs `windows`: cuts the retained packs into runs of
  64 MiB of retained payload, ended early by a dead run over 8 MiB.
  Packs are appended in id order, so a pack's payload starts no earlier
  than the summed lengths of the packs before it and no later than that
  plus every byte of the file that is not pack payload; a window's file
  range is that bound over its packs. Both inputs (every pack's length,
  the mark's packs) were already in memory in run.rs.
- compact/window.rs `ReadAhead`: before each window, asks the kernel for
  the window's range through a second read-only descriptor, each byte
  once. macOS: F_RDADVISE, which returns when the range is in memory
  (about 5 GB/s cold, 4 ms per GB warm). Linux: POSIX_FADV_WILLNEED in
  128 KiB steps (the kernel caps one request at the device read-ahead
  size). A hint only: no byte and no result changes, and a file laid
  out in another order only loses the benefit. The sealed file is
  attached immutable=1, so SQLite holds no lock on it that closing the
  descriptor could drop.
- build.rs `copy_packs`: the payload table (`pack`, or `pack_unit` in
  the group-row layouts) is copied one window per execution of one
  prepared statement, `WHERE pack_id BETWEEN ?1 AND ?2 AND <retained>`.
  The retained filter and every other table's copy are unchanged.

Statements: the pack copy is one prepared statement run once per
64 MiB of retained payload (17 on the 1.1 GB shape, 1 on the small one)
instead of one statement run once. Rows, bytes and the installed file
are identical.

Measured, release, macOS APFS, one sample each. Cold = the Store copy
was written with F_NOCACHE just before the run (the cache state is
declared by method, not verified); rebuild_probe is a local diagnostic,
the same compaction call gc_bench makes.
- 1.1 GB cold: pack copy 3483 -> 690 ms, build 3710 -> 865 ms, rebuild
  wall 5054 -> 2071 ms (load 6.1 and 6.8). The mark is 0.8 to 0.9 s of
  what remains: it is bound by the same one-pack-per-round-trip reads.
- small (28 MB) cold: pack copy 5.9 -> 6.2 ms, no change (one window).
- 1.1 GB warm, gc_bench --runs 1: rebuild 1039 -> 1638 ms in a sample
  taken at load 5.9 to 7.2 in which the untouched mark also ran 96 ->
  175 ms; the instrumented warm run beside it (load 6.6) gave pack copy
  583 ms in 17 windows against 548 ms as one statement, build 734 ms,
  rebuild 1082 ms. Small warm, gc_bench: 106 -> 146 ms in the same
  loaded sample (mark 17 -> 22 ms, none path 25 -> 51 ms).
Linux was type-checked for aarch64-unknown-linux-musl and
x86_64-unknown-linux-gnu in a scratch crate (the package itself does not
cross-check from this host); it was not run or measured.

Tests: new sqlite_compact
`retained_packs_on_both_sides_of_a_long_dead_run_are_all_copied` (a two
window rebuild) and sqlite_compact_units
`the_pack_copy_is_cut_into_windows_in_pack_order`. The generated matrix
report counts the new sqlite_compact test (34 -> 35). Pass:
sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix
(1), sqlite_compact_units (9), compact_metrics (5), space_reclamation
(3), kept_objects (7), kept_node (13). No test deleted.

Production LOC: 187,965 -> 188,160 (delta +195)
  core     40,589 -> 40,589 (0)
  engine   89,776 -> 89,971 (+195)  layerfs-persistence +195
  control  51,073 -> 51,073 (0)
  api      6,527 -> 6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 7b2f9ac and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ory sync (layerfs-project#41)

Two steps of a SQLite rebuild changed no outcome:

- After the mark the run rewrote `<store>.compact-op` (write, fsync,
  rename, directory fsync) only to add the mark's tuple and digest. No
  reader uses them: a later call judges a leftover by the run,
  generation and owner the admission record already names, and
  `store.status` reads the admission phase as `mark`. The values go out
  with the next write (`build`). A stop point still writes its own
  record with them.
- After the build the run synchronized the Store's directory "before
  the verification reads them back". Verification reads through the
  page cache, the fresh file is synchronized before the rename and the
  directory after it, and the record written next is synchronized
  itself.

Round trips per rebuild: 6 op record writes -> 5, 16 fsync calls -> 13
(each is F_FULLFSYNC on macOS). The `none` path is unchanged.

docs/architecture/engine/store/sqlite.md gains a "Compaction rebuild"
section. It also covers the two previous commits (the fresh file's
size, the pack copy's windows and the read-ahead), which changed the
build's steps without a page to update.

Measured, release, macOS APFS, gc_bench --runs 1, one sample each
(1-minute load 3.0 to 3.5):
- small (28 MB): rebuild 106.3 -> 95.1 ms against the last sample taken
  at a comparable load (the first commit of this series); mark phase
  17.0 -> 8.2 ms, which is the removed record rewrite.
- 1.1 GB: rebuild 1039.2 -> 1138.3 ms; build 833.2 -> 800.7 ms, verify
  22.4 -> 155.5 ms. The device flush that follows the 1.1 GB load is
  not removed by this commit, only moved: it was paid by the
  directory sync inside the build phase and is now paid by the first
  record write of the verify phase. This shape gains nothing here.

Tests: no new test (an fsync is not observable through the public
API). Pass: sqlite_compact (34), sqlite_compact_crash (15),
sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics
(5), space_reclamation (3), kept_objects (7), kept_node (13). No test
deleted.

Production LOC: 188,160 -> 188,152 (delta -8)
  core     40,589 -> 40,589 (0)
  engine   89,971 -> 89,963 (-8)  layerfs-persistence -8
  control  51,073 -> 51,073 (0)
  api      6,527 -> 6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent d463879 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…project#41)

NEEDS THE OWNER'S DECISION: spec §7 step 7 reads "chmod 0600;
sync_all; rename; fsync the directory" without naming a profile. This
commit keeps that sequence for a Store whose profile synchronizes
(Normal, Durable) and skips the synchronization for a Disposable one.
It is one flag; reverting this commit restores the unconditional calls.

A SQLite rebuild made 13 fsync calls of its own (16 before the previous
commit): two per op record write, the fresh file before the rename, the
directory after it and after the settle. Each is F_FULLFSYNC on macOS.
On the 28 MB shape they were about two thirds of the whole rebuild, and
three of the `none` path's work was the same calls.

A Disposable Store (WAL, synchronous=OFF) survives a process crash and
claims nothing after a power loss: its own commits and checkpoints are
never synchronized, so a compaction that synchronizes its record and
its install protects a Store that the same power loss may already have
torn. The run now takes `durable = profile.synchronous != 0` once:

- op.rs `RunOp.durable`: a record write is still temp file plus rename
  (what a process crash needs); the file and directory fsync run only
  when durable. A record read back from disk is written synchronized,
  and `mark_returned` takes the run's flag.
- install.rs `prepare_target(target, durable)`: chmod always, sync_all
  only when durable. `settle` synchronizes the directory only when
  durable.
- run.rs: the directory fsync after the rename and at the end of the
  `none` path run only when durable. The abort paths are unchanged.

A Normal Store's compaction is unchanged by this commit: the same 13
calls. The product opens Stores Normal since 2026-11-17 while GC is
measured under Disposable (#42 item 42), so from here the Disposable
bench numbers exclude synchronization the product profile still pays.
Measured on the same fixtures opened Normal (rebuild_probe, a local
diagnostic; one sample, load 3.9): 28 MB rebuild 142 ms against 46 ms
Disposable; 1.1 GB rebuild 1100 ms against 963 ms.

Measured, release, macOS APFS, gc_bench --runs 1 (Disposable), one
sample each, 1-minute load 4.9:
- small (28 MB): rebuild 95.1 -> 32.7 ms (build 31.8 -> 20.8, verify
  10.5 -> 1.0, install 34.2 -> 2.1 ms); `none` 24 -> 9 ms.
- 1.1 GB: rebuild 1138.3 -> 982.5 ms (build 800.7 -> 782.2, verify
  155.5 -> 11.3, install 78.7 -> 94.1 ms); `none` 115 -> 93 ms. What is
  left of install is the rename, which frees the old file's 1.26 GB.

Tests: no new test (an fsync is not observable through the public
API). Pass: sqlite_compact (34), sqlite_compact_crash (15),
sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics
(5), space_reclamation (3), kept_objects (7), kept_node (13). No test
deleted.

Production LOC: 188,152 -> 188,168 (delta +16)
  core     40,589 -> 40,589 (0)
  engine   89,963 -> 89,979 (+16)  layerfs-persistence +16
  control  51,073 -> 51,073 (0)
  api      6,527 -> 6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 7f5bb75 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…project#41)

After the batched head read, a pack that needed more than its head still
took scoped acquisitions of its own: a statement and a BLOB handle per pack
and per round on SQLite, and on PostgreSQL, where mark's time follows round
trips, one or more round trips per pack.

- backend/seam.rs: Mechanics::pack_ranges returns selected byte ranges of
  several packs in one call. Its default is one scoped acquisition per pack,
  which is what PostgreSQL uses until it overrides it.
- backend/sqlite: one BLOB handle moved from row to row; a pack's ranges
  ascend, so its overflow chain is followed once. The GroupRows layouts keep
  the scoped read.
- history/objects/edges.rs: rounds are per wave batch, not per pack. Each
  round resolves every pack of the batch against the bytes held and reads
  what all of them still need in bounded calls (1,024 ranges, 512 KiB).
  The per-pack range plan and acquisition are gone; the kept-waste
  directory read uses the same call.

Measured (gc_profile, release, warm cache, macOS, load about 3, one run):
- 28 MB small Store (737 objects in 158 packs of up to six groups): mark
  6.8 -> 5.4 ms; statements 172 -> 58; BLOB handles 150 -> 37; reads
  996 -> 768.
- 1.13 GB Store (8,913 objects in 8,816 single-group packs): unchanged,
  37.0 ms, 298 statements, 229 handles.

Tests: kept_objects, kept_node, compact_metrics, sqlite_compact,
sqlite_compact_crash, space_reclamation, sqlite_group_rows,
sqlite_group_rows_indexed, postgresql_gc_mark, postgresql_compact,
postgresql_compact_fixtures pass. sqlite_compact_matrix failed once and
passed on the next run with no source change: its AC-49 arm races a Commit
thread against the rebuild and unwraps the Save inside that thread
(tests/sqlite_compact_matrix.rs:149). Reported, not changed here. No test
added or deleted.

Production LOC: 188037 -> 188192 (delta +155)
Scope and method: tools/production_loc.py at this commit, first parent
against the committed tree; core 40589 -> 40589, engine 89848 -> 90003,
control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c;
last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…layerfs-project#41)

Counts, not time: none of these is visible in a one-sample wall time on
the bench shapes. They remove statements, table passes and connection
opens that produced nothing the run did not already have.

- compact_visit.rs: the mark ran two inserts per reached object, one
  into `visited` and one `INSERT OR IGNORE` into `live_pack`. The
  retained packs are the visited rows' packs, so `finish` now derives
  them with one ordered `INSERT ... SELECT DISTINCT`. Statements per
  mark: 2N -> N + 1 (N = reached objects; about 9,000 on the 1.1 GB
  shape, so about 18,000 -> 9,000).
- verify.rs `verify`: four count statements and one totals statement
  became one statement. Passes over the fresh file: `pack` 3 -> 1,
  `object_location` 2 -> 1, `store_totals` 5 reads -> 1 (the tuple
  already reads the row). Own statements 8 -> 4. Every check, its error
  and the order of the checks around the `DuringVerify` stop are
  unchanged.
- verify.rs `sealed_state`: the sealed file's tuple and live-heads
  digest are read through one `immutable=1` connection instead of two
  (`immutable_tuple` and `immutable_digest` are removed). Connections
  opened per rebuild: 7 -> 6 besides the session.
- build.rs: the totals rewrite reads `pack` once for `stored` and
  `metadata` instead of twice.

Measured, release, macOS APFS, gc_bench --runs 1 (Disposable), one
sample each, 1-minute load 3.7 and 3.4:
- small (28 MB): rebuild 32.7 -> 34.6 ms (mark 7.3 -> 7.6, build 20.8
  -> 22.2, verify 0.96 -> 0.94, install 2.06 -> 2.03 ms); `none` 9 -> 9
  ms.
- 1.1 GB: rebuild 982.5 -> 927.5 ms (mark 92.3 -> 88.5, build 782.2 ->
  730.4, verify 11.3 -> 10.2, install 94.1 -> 96.2 ms); `none` 93 -> 92
  ms. The build difference is sample noise in the pack copy, not this
  change.

Tests: pass, unchanged: sqlite_compact (34), sqlite_compact_crash (15),
sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics
(5), space_reclamation (3), kept_objects (7), kept_node (13). The
rebuild tests cover `live_pack`: verification refuses a fresh file
whose pack count is not the mark's. No test deleted.

Production LOC: 188,168 -> 188,162 (delta -6)
  core     40,589 -> 40,589 (0)
  engine   89,979 -> 89,973 (-6)  layerfs-persistence -6
  control  51,073 -> 51,073 (0)
  api      6,527 -> 6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 8e7acb3 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ayerfs-project#41)

Five commits by a worktree agent, merged without conflict onto the mark
work: size the fresh file before the bulk load, copy packs in 64 MiB
windows with read-ahead on the sealed Store, drop the post-mark record
rewrite and the post-build directory sync, make a Disposable Store's
compaction ask for no fsync, and merge repeated statements and scans.

Owner decision pending: 8e7acb3 departs from spec section 7 step 7 as
written (it names sync_all and the directory fsync without naming a
profile). Normal, the product profile, still syncs. An independent
crash-safety review of the five commits is in progress and is not reflected
here.

Measured on the merged tree (gc_bench --scale medium, release, Disposable,
macOS, one run per arm, load 6.6 at start with other agents working;
9.789 GB live in 83,559 packs; receipt not committed):
  none     mark 7.42 s                        wall  7.42 s
  rebuild  mark 0.92 s  build 21.69 s  verify 0.30 s  install 0.50 s
                                              wall 23.41 s
  Same bytes freed as every earlier run: 1,095,524,352.
Earlier runs of the same shape: at 6bca861 none 45.19 s and rebuild
66.58 s (build 38.45 s); at d19d98f none 4.82 s and rebuild 46.16 s
(build 43.93 s). The none arm is the first mark after the fixture is built
and varies with the page cache: 4.79 s then, 7.42 s now.

Tests on the merged tree: kept_objects 7, kept_node 13, compact_metrics 5,
sqlite_compact 34, sqlite_compact_crash 15, sqlite_compact_matrix 1,
sqlite_compact_units 9, space_reclamation 3, sqlite_group_rows 7 pass.
PostgreSQL tests were not rerun for this merge: it changes SQLite files
only.

Production LOC: 188192 -> 188445 (delta +253)
Scope and method: tools/production_loc.py, first parent c1c2f98 against
the merged tree; core 40589 -> 40589, engine 90003 -> 90256, control
51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last
counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This reverts commit 8e7acb3.

A review of the merged rebuild commits found that skipping every fsync on
a Disposable Store changes what a power loss can do during a compaction:

- after the rename, the fresh file's pages may not be on disk, so the
  whole Store can be lost, not only the compaction;
- an operation record that was never synchronized can come back
  unreadable, and the leftover files then make `store.compact` refuse
  until someone deletes them by hand;
- the synchronization before the rename was also the last point where a
  deferred write error of the fresh file could stop the install.

The spec's behaviour (section 8: synchronize the fresh file, the record
and the directory around the rename) stays until the owner decides
otherwise. Cost on the 1.1 GB bench shape, release, one sample, load
4.4: install 71.6 ms of a 1.045 s rebuild.

Tests: sqlite_compact (34), sqlite_compact_crash (15),
sqlite_compact_units (9), compact_metrics (5), space_reclamation (3),
sqlite_group_rows (7), kept_objects (7) pass. sqlite_compact_matrix
failed once and passed once with no source change between: its AC-49 arm
unwraps a Save inside the racing Commit thread
(tests/sqlite_compact_matrix.rs:149), which panics when the rename lands
during that Save. The test is unchanged on this branch; not fixed here.

Production LOC: 188445 -> 188429 (delta -16)
Scope: core, engine, control and api product source; counted with
tools/production_loc.py on the first parent and the committed tree.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`copy_packs` copies retained packs one 64 MiB window of pack ids at a
time. The range sat on the source table only, and SQLite drives that
statement from `scratch.live_pack`, so every window walked every
retained pack of the Store and seeked the source for each: windows times
retained packs seeks. A review measured 200 to 250 ms per window at
800,000 retained packs against 0.1 ms with the range inside the
subquery.

The range now bounds the `live_pack` scan itself, so a window touches
only its own packs and the copy is linear in retained packs.

1.1 GB bench shape, release, one sample, load 4.4: rebuild 1.045 s
(mark 71.7 ms, build 791.6 ms, verify 97.1 ms, install 71.6 ms). That
Store has too few windows to show the difference; the count argument
above is the evidence.

Tests: as the parent commit (same build), including the
sqlite_compact_matrix note there.

Production LOC: 188429 -> 188429 (delta 0)
Scope: core, engine, control and api product source; counted with
tools/production_loc.py on the first parent and the committed tree.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The mark loads every `object_location` row once. The query asked for
pack order, which no index supplies, so each mark made the engine sort
the whole table (`USE TEMP B-TREE FOR ORDER BY` on SQLite, a sort node
on PostgreSQL). The order bought nothing: `object_id` is the table's
key, so an object has one row and there is no winner to pick, and both
maps the rows fill are keyed.

The query now reads the table in its own key order on both backends.

gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs),
release, warm second mark, one sample, load 6.7:
  engine steps of a mark  217,406 -> 128,273
  sorts                   6 -> 5
  the statement           3.87 ms -> 1.88 ms

Tests: kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), compact_metrics (5), space_reclamation (3),
postgresql_gc_mark (4), postgresql_compact (30),
postgresql_statement_schema (2) pass; Clippy and fmt clean.

Production LOC: 188,429 -> 188,428 (delta -1)
  core     40,589 -> 40,589 (0)
  engine   90,240 -> 90,239 (-1)  layerfs-persistence -1
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent cb0a97b and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The mark asks each pack for its first bytes: the directory and, for the
common one-record pack, the record header that names its delta base. It
asked for 1,024 bytes. SQLite keeps only the first part of a large row
on the row's own page (at least 489 bytes of a row too large for a
4,096-byte page) and the rest in a chain of further pages, so most of
those reads went one page past the row: a second page load per pack,
and a random read on a cold cache.

The first read is now 432 bytes. The columns before the body take at
most 49 bytes, so up to 440 bytes of body are always on the page the
row lookup has already loaded. A pack with a wider directory is read
again, as before.

gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs,
every pack at the 489-byte minimum), release, one run of three marks
per size:
  first read    time in pack reads   warm mark wall     1-min load
  1,024 bytes   8.2 to 8.6 ms        38.8 to 40.8 ms    4.9
    448 bytes   8.9 to 9.4 ms        39.9 to 43.9 ms    6.8
    440 bytes   0.2 ms               28.9 to 34.7 ms    6.5
    432 bytes   0.2 ms               32.7 to 40.4 ms    9.2
Read calls are unchanged at 9,104; bytes read fall from 8,972,342 to
3,857,332. The wall times were taken under different load, so the
pack-read time is the comparison to rely on.

Tests: kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5),
space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30)
pass; Clippy and fmt clean.

Production LOC: 188,428 -> 188,428 (delta 0)
  core     40,589 -> 40,589 (0)
  engine   90,239 -> 90,239 (0)
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent c15826e and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A CPU sample of warm marks (after the pack reads became cheap) put more
than half of a mark in map bookkeeping: the walk and the kept-waste pass
ask ordered maps keyed by 32-byte identity about every object several
times, and each question pays a 32-byte comparison per tree level.

Those maps are now hash maps (`history/objects/ids.rs`). The standard
hasher was tried first and cost as much as the comparisons it replaced,
because it runs its permutation over all 32 bytes. An identity is a
BLAKE3 digest, so `IdState` takes its first eight bytes and mixes them
with a mask and multiplier drawn per map from the standard random
source. The key is there because identities come from stored content: a
writer could otherwise choose content whose identities share table
slots. Results do not depend on it: the two lists a mark returns in
identity order (`KeptObjects::objects`, the delta base references) are
sorted after collection, and the walk still orders each wave by locator.

The per-pack maps (`read_bytes`, `layouts`) are hashed too. The pack
inventory keeps its ordered maps.

`MarkGroups` moves to `history/objects/plan.rs`, unchanged, so
`objects.rs` stays under the 999-line ceiling (958 -> 895 lines).

100 warm marks of the kept 1.1 GB bench Store (8,913 objects, 8,816
packs), release, two runs per build, interleaved, 1-min load 7.6 to 7.8:
  ordered maps      user 3.83 s, 3.92 s   mean mark 43.1 ms, 42.0 ms
  standard hasher   user 3.50 s, 3.18 s   (load 9.0 to 9.2; not kept)
  this commit       user 2.82 s, 2.86 s   mean mark 30.7 ms, 31.3 ms

Tests: kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5),
space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30)
pass; Clippy, fmt and the product boundary guard clean.

Production LOC: 188,428 -> 188,488 (delta +60)
  core     40,589 -> 40,589 (0)
  engine   90,239 -> 90,299 (+60)  layerfs-persistence +60
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent c1d48b6 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…t (layerfs-project#41)

Neither the Store's nor the scratch `object_location` has a `pack_id` index,
so every Phase A batch scanned the whole source table and the whole visited
set three times and the growing scratch table four times (EXPLAIN ANALYZE on
PostgreSQL 17.11: a batch holding 16,161 of 290,561 rows still read all
290,561 source rows and all 258,561 visited rows per statement). The copy cost
batches times objects.

Phase A now makes one scan of the source into a session temporary table of the
visited location rows (`pg_temp.gc_copy`), indexed on `pack_id`. Each batch
reads its own range through that index and takes its row counts from the two
INSERTs. The copy set must hold exactly one row per visited object, and the
rows copied by all batches must add up to it, so a visited object in a pack
outside the live list is still refused. The final scratch check
(count = count(DISTINCT object_id) = |visited|, pack count) is unchanged and is
now one statement. The retained canonical total is read from the copy set
instead of a second semi-join scan of the source.

Kept as before: one transaction per batch, the 1 GiB batch bound, and the
own-range DELETE of spec section 8 step 3. That DELETE still scans the scratch
`object_location` once per batch (it has no `pack_id` index and must not gain
one, because it becomes the Store's table); it is the only per-batch scan left.

Build phase, one sample per case, local PostgreSQL 17.11 in Docker,
1-minute load 4.0-4.5, fixture of 1 KiB whole files (4 projects):

  72,641 location rows, 73.9 MB of packs
    1 batch  (product bound)   349.3 -> 326.2 ms   45 -> 38 statements    92 -> 80 round trips
    4 batches (16 MiB bound)   408.9 -> 336.4 ms   75 -> 53 statements   155 -> 113 round trips
  290,561 location rows, 295.5 MB of packs
    1 batch  (product bound)  1886.1 -> 1134.3 ms  45 -> 38 statements    92 -> 80 round trips
    16 batches (16 MiB bound) 2654.9 -> 1240.0 ms 195 -> 113 statements  407 -> 245 round trips

Per additional batch: 10 -> 5 statements, 21 -> 11 round trips, and at 290,561
rows +51.3 -> +7.0 ms. Four times the rows and batches cost 6.5x before and
3.7x after.

Limits of these numbers. The multi-batch rows were taken with the batch bound
lowered to 16 MiB in an uncommitted build, because no fixture I may build has
more than one 1 GiB batch; no Store with several real batches was measured.
For one batch the wall time is not a demonstrated gain: a second baseline
sample of the 290,561-row case read 1338.3 ms, so the baseline's own spread is
as large as the difference. The statement and round-trip counts are exact.

Cost: the copy set is temporary space the run holds until its session ends,
about 92 to 97 bytes per retained object row (6.3 MB for 64,641 retained rows
and 23.8 MB for 258,561). It is added to the reported peak temp.

Test: `a_duplicate_retained_location_row_refuses_the_build` pins that a second
location row for a retained object is an `Integrity` refusal before the swap,
with the Store left at G and the op row naming the leftover scratch schema. It
passes on the parent too (there the refusal is "compaction batch count", here
"compaction copy set"), so it characterizes kept behaviour; it is not a
failing-on-parent regression test. No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 4, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4) pass; cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,428 -> 188,405 (delta -23)
  core     40,589 ->  40,589 (0)
  engine   90,239 ->  90,216 (-23)  layerfs-persistence -23
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of parent c1c2f98
  and of this commit, recounted after the rebase onto c1c2f98. The
  measurements above were taken before the rebases, on parent d19d98f.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…yerfs-project#41)

The mark asks for the first 1,024 bytes of up to 256 packs per call
(`Mechanics::pack_prefixes`, 962acd4). PostgreSQL had only the default,
which sends one statement per pack. `objects::read::prefixes` answers the call
with one statement: the ids as an array, joined to `pack` and returned in the
array's order with each pack's declared length. A pack that is absent is
`Missing`, and a head whose size disagrees with the declared length is
`Malformed`, as for a single prefix.

Mark walk, one sample per case, local PostgreSQL 17.11 in Docker, fixtures of
1 KiB whole files (99 per Layer, 4 projects):

  72,641 location rows, 1,280 live packs (640 payload)   load 5.26 -> 3.53
    statements        3,850 -> 3,212
    round trips       3,865 -> 3,228
    pack bytes received  131,893,084 -> 131,893,084
    wall              2,064.8 -> 1,846.7 ms
  290,561 location rows, 5,120 live packs (2,560 payload) load 5.00 -> 3.53
    statements        15,370 -> 12,820
    round trips       15,385 -> 12,836
    pack bytes received  527,732,824 -> 527,732,824
    wall              8,430.4 -> 7,779.5 ms

The counts are exact: one head statement per payload pack becomes one per 256
packs (640 -> 3 and 2,560 -> 10). The wall figures were taken at a lower host
load than their baselines and are one sample each, so they are not evidence of
a speed-up by themselves. The bytes do not move: the heads are 1 KiB each, and
what dominates is the 256 KiB head a scoped read fetches with every
descriptor. These figures were taken on parent 962acd4, before the mark
moved its payload ranges to `pack_ranges` (c1c2f98).

Test: `a_mark_reaches_every_record_of_payload_packs_larger_than_its_head`
marks a Store of four trees whose payload packs (99 records, about 102 KiB,
several groups) are far larger than a head, and requires every saved object and
nothing else. It passes on the parent as well; it fails when the heads come
back in another order (checked by reversing the statement's ORDER BY: the
closure is refused as `Integrity`). It asserts no statement count, because the
mark's per-pack statements are set by `history/objects/edges.rs`, which is
still changing. No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,405 -> 188,451 (delta +46)
  core     40,589 ->  40,589 (0)
  engine   90,216 ->  90,262 (+46)  layerfs-persistence +46
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and of
  this commit, recounted after the rebase onto c1c2f98

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ayerfs-project#41)

The mark's payload path asks for the group directories and record headers of
up to 256 packs per call (`Mechanics::pack_ranges`, c1c2f98). PostgreSQL had
only the default, one scoped acquisition per pack, and a scoped acquisition
here fetches the first 256 KiB of the body with its descriptor, so every
payload pack of the fixtures below still crossed the wire whole.

`objects::read::ranges` answers one call with one statement: the pack ids,
offsets and lengths as three arrays, joined to `pack`, one `substring` per
range, returned in request order. The body is stored out of line and
uncompressed, so each `substring` fetches only the chunks it covers. A pack's
ranges must ascend without overlap (`Malformed` otherwise); an absent pack is
`Missing`; a range that reaches past its body comes back short and is
`Malformed`.

Mark walk, one sample per case, local PostgreSQL 17.11 in Docker, fixtures of
1 KiB whole files (99 per Layer, 4 projects), against the parent commit:

  72,641 location rows, 1,280 live packs (640 payload)    load 5.54 -> 4.14
    statements           3,262 -> 1,996
    round trips          3,278 -> 2,013
    pack bytes received  136,989,034 -> 4,611,164
    wall                 2,518.0 -> 1,529.4 ms
  290,561 location rows, 5,120 live packs (2,560 payload) load 5.25 -> 4.20
    statements           13,020 -> 7,952
    round trips          13,036 -> 7,969
    pack bytes received  548,116,624 -> 18,605,144
    wall                 10,746.3 -> 6,721.3 ms

The counts are exact. The wall figures are one sample each and the baseline
ran at a higher host load, so the size of the speed-up is not established;
that both statements and bytes fall is.

What is left of the walk is per metadata pack, three statements each (small
fixture, 640 metadata packs, from the server's statement log): 640 scoped
reads through the decoding reader, 640 `inode_value_block` reads and 639
single-pack prefix reads of the kept-waste pass, against 64 range statements
and 3 head statements for all payload packs. Those callers are in
`history/objects.rs`, outside this change.

Test: `a_mark_reaches_every_record_of_payload_packs_larger_than_its_head`
now runs through this statement. It fails when the ranges come back in another
order (checked by reversing the ORDER BY: the closure is refused as
`Integrity`). The missing-pack and short-range refusals are not reachable
through a public call and have no test. No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,451 -> 188,518 (delta +67)
  core     40,589 ->  40,589 (0)
  engine   90,262 ->  90,329 (+67)  layerfs-persistence +67
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and
  git checkout-index of the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ck (layerfs-project#41)

The swap transaction took ACCESS EXCLUSIVE on every Store table and then
scanned the moved object table three times (count, count DISTINCT, canonical
sum) and the pack table three times, and read the live-heads digest twice.
Every reader and writer of the Store waited for those scans.

The row counts and sums of spec section 8 step 9 are now read in one statement
from the scratch tables, inside the same transaction and before the Store-wide
lock. The two scratch tables are locked first (ACCESS EXCLUSIVE, waiting), so
the rows counted are the rows SET SCHEMA moves. Both sides of each comparison
come from the mark: the mark's object and pack counts, and the mark's totals
minus the dead totals, which the tuple re-check under the lock then pins. The
digest is read once under the lock; the lock is held from that read to COMMIT.
`store_totals.logical` is no longer re-read: the tuple re-check has just
compared it, so the history rows are summed against the mark's value. The two
`gc_op` reads are one statement.

Still under the lock: the tuple and digest re-check, the ACL read and replay,
the drop and move, the table fingerprint, the logical sum over the history
rows, the trigger/function contract check, the relation size, both updates,
the scratch drop and the standby check.

Measured on a local PostgreSQL 17 (Docker, loopback), one sample each,
synchronous_commit=local, lock hold timed in the client from the LOCK TABLE
statement to the end of COMMIT:

  72,641 object rows, 1,360 packs (1-minute load 3.26 -> 3.04)
    lock hold           57.4 ms -> 28.5 ms
    statements under it      35 -> 25
    swap phase          38 statements, 64 round trips, 64.8 ms
                     -> 30 statements, 49 round trips, 58.5 ms
  290,561 object rows, 5,440 packs (1-minute load 3.08 -> 3.20)
    lock hold          154.4 ms -> 49.7 ms
    statements under it      35 -> 25
    swap phase          38 statements, 64 round trips, 160.3 ms
                     -> 30 statements, 49 round trips, 150.9 ms

At the larger size the COMMIT itself is 17 ms of the remaining hold. The
count DISTINCT sort (77 ms at 290,561 rows) still runs, outside the lock.

Changed behaviour to know about: a count or sum mismatch on the scratch tables
is now refused as Integrity before the tuple is re-read, so it is reported
even when the Store also moved after the mark (before, Conflict came first).
The scratch-table lock waits without NOWAIT; before this change the same wait
happened at ALTER TABLE ... SET SCHEMA while the Store-wide lock was held.

No test reaches the count or sum refusals (none did before); the phases run
back to back inside one call and there is no hook between build and swap.
No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,518 -> 188,502 (delta -16)
  core     40,589 ->  40,589 (0)
  engine   90,329 ->  90,313 (-16)  layerfs-persistence -16
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and
  git checkout-index of the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rfs-project#41)

The mark's history closure (`kept_node_mark`) ran the text both engines
share. Its recursive step joins each round's new nodes to a view of every
history edge, and PostgreSQL plans that as a hash of the new nodes probed by
a scan of all edges. A Commit that only its parent chain reaches arrives one
round at a time, so a Branch with n unpublished Commits cost n scans of the
edge set: 4,000 rounds read 5.4 million buffers (EXPLAIN ANALYZE, BUFFERS).

PostgreSQL now runs its own text, registered in
`backend/postgresql/history/statements.rs`. The roots and the final insert
are unchanged. The recursive step reads each new node's own row by primary
key and emits its references from a VALUES list. Where the shared text joined
to `commit` or `layer` to learn which kind a reference names, this text emits
both kinds; a name that is no row of that kind has no row to follow and the
final insert's join drops it. The shared file is not edited and SQLite still
runs it.

Measured on a local PostgreSQL 17 (Docker, loopback), one sample each, the
closure statement's server time inside a `store.compact` mark:

  one Branch, 999 unpublished Commits     584.9 ms ->   6.5 ms  (load 5.72, 6.28)
  one Branch, 3,999 unpublished Commits 9,386.3 ms ->  20.8 ms  (load 5.72, 6.17)
  mark walk of the larger Store        14,752.2 ms -> 4,486.6 ms

Four times the chain cost 16 times before and 3.2 times now.

It is slower where every Commit is published as a Layer, because every Layer
of a live Stack is a root and the old text finished in two rounds:

  2 Stacks x 1,000 Layers, 2,001 kept nodes   19.9 ms -> 258.9 ms
  2 Stacks x 2,400 Layers, 4,801 kept nodes  132.4 ms -> 251.1 ms

About 240 ms of each new figure is the server's JIT compiler, not the
statement's work: the planner's estimate for the new text passes
`jit_above_cost` at about 550 root nodes, and EXPLAIN ANALYZE shows 241 ms of
JIT against 10 ms of execution. The shared text passes the same threshold on
larger Stores (45 ms of its 132.4 ms above). The Store session does not set
`jit`; that is in `session/profile.rs`, outside this change.

Test: `the_postgresql_closure_keeps_the_nodes_the_shared_statement_keeps`
builds a live project (a published Commit, a fork, a merge whose second
parent is a Layer, 30 unpublished Commits) and a deleted project, and
requires the same `kept_node` rows from both texts. It fails when the
parent-Commit edge is removed from the new text. It does not fail when the
second-parent, fork or source-Branch edge is removed: every Layer of a live
Stack and every live Branch is already a root in this fixture. No fixture
here has a `merged_from_id` or `displaced_id` reference. The two texts also
gave the same node set (count and md5 of the sorted ids) on the four
measurement Stores. No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 6, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,502 -> 188,569 (delta +67)
  core     40,589 ->  40,589 (0)
  engine   90,313 ->  90,380 (+67)  layerfs-persistence +67
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and
  git checkout-index of the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The mark's dead-set insert selected `('digest', fd.root)` for every
`file_digest` row whose root is unvisited. `file_digest` is keyed by
`(digest, length)`, so two rows can name one root, and `gc_dead` is keyed by
`(kind, id)`: the second copy of the root broke the key and `store.compact`
failed in the mark with `Integrity("persistence")`. Phase C already expects
such rows (F50-2 deletes by root "even when a different digest names it").

The digest arm now selects each root once. Every other arm of the insert
selects its own table's key and cannot repeat.

Test: `two_digest_rows_of_one_dead_root_are_both_collected` records two
digest rows for one dead root through `record_file_digests`, compacts and
requires both rows gone and the op settled. Before this change it failed at
the `compact` call with `Integrity("persistence")`. No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 8, _compact_units 8, _gc_mark 6, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,569 -> 188,569 (delta 0)
  core     40,589 ->  40,589 (0)
  engine   90,380 ->  90,380 (0)
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and
  git checkout-index of the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…-project#41)

Phase C deleted `file_digest` rows whose root is in the dead set 1,000 at a
time. Each batch joined `gc_dead` to `file_digest` on `root`, which has no
index, once to pick the batch and once to delete it: two scans of the whole
table per 1,000 rows. Four times the dead rows cost 17.8 times the server
time (EXPLAIN shows a hash join over a sequential scan of `file_digest` in
both halves).

One scan now lists the keys of the rows to delete in a numbered session
table. Each batch deletes its own range of that list by `(digest, length)`,
the table's primary key, and checks the root against `gc_dead` again, so
membership in the persisted dead set is still the filter. Batches are still
1,000 rows, one transaction each. A crash loses the session table; the
resumed call lists what is left.

Measured on a local PostgreSQL 17 (Docker, loopback), one sample each,
synchronous_commit=local. Server time of the digest statements, and the
whole cleanup phase:

  dead rows   digest statements          cleanup phase            load
     50,000     710.0 ms ->   201.2 ms    1,078.3 ->   483.7 ms   4.42, 7.29
    200,000  12,622.2 ms -> 2,443.5 ms   14,284.4 -> 3,984.1 ms   4.42, 7.29
    800,000   not run    -> 7,965.7 ms    not run -> 11,468.6 ms  6.68

Statements and round trips are nearly unchanged (200,000 rows: 227 -> 229
statements, 672 -> 677 round trips); the gain is in what each batch reads.
Per batch: 3.4 ms at 50,000 rows, 11.4 ms at 200,000, 9.3 ms at 800,000.
Up to 200,000 rows the planner still answers a batch with one scan of
`file_digest` (cheaper by its own cost model than 1,000 index probes), so
the cost per batch grows until the table is large enough that it switches
to the primary key; from there it is level. The old text at 800,000 rows
was not run (its measured slope puts it far past the two-minute limit).

What is left of the phase at 200,000 rows: 1,175 ms in 211 COMMITs, one per
batch. Larger batches would cut that; the batch size is unchanged here.

The session table holds one row per dead digest row (about 70 bytes plus
its index) until the last batch. It is not counted in `peakTempBytes`,
which the swap records before Phase C starts.

A `file_digest` row with a dead root that is inserted after the list is
made is now left in place; the old loop would have deleted it in a later
batch. Such a row can only come from a save at the new generation.

Tests: `dead_digest_rows_beyond_one_batch_are_all_collected` records 2,500
dead digest rows and one retained row, compacts and requires only the
retained row left; it fails when a batch drops its last row (checked with
`b.n < $2`). The existing F50-2 case (a dead-root row inserted before a
resumed cleanup) passes unchanged. No test deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 9, _compact_units 8, _gc_mark 6, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.
The timings were taken on this cleanup.rs one commit earlier (before the
dead-root dedup); the file is byte-identical.

Production LOC: 188,569 -> 188,590 (delta +21)
  core     40,589 ->  40,589 (0)
  engine   90,380 ->  90,401 (+21)  layerfs-persistence +21
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and
  git checkout-index of the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…erfs-project#41)

`load_visited` first copied every visited object id into a `Vec<Vec<u8>>`
(one heap allocation per object) and the live pack ids into a second `Vec`,
then streamed those to COPY. The copy lived until the load finished, beside
the mark's own set.

Each COPY row is now written straight from the mark's ids. The statements,
their order and the rows sent are unchanged.

Not measured: the saving is one 24-byte `Vec` header and one 32-byte heap
block per visited object (arithmetic, before allocator overhead), and the
fixtures at hand are too small to show it in peak memory. The change is
confirmed by reading and by the suites below, not by a memory measurement.

No test added or deleted.

Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 9, _compact_units 8, _gc_mark 6, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.

Production LOC: 188,590 -> 188,584 (delta -6)
  core     40,589 ->  40,589 (0)
  engine   90,401 ->  90,395 (-6)  layerfs-persistence -6
  control  51,073 ->  51,073 (0)
  api       6,527 ->   6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on git archive of the parent and
  git checkout-index of the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The kept-waste pass reads the directory prefix of every retained pack
the walk did not lay out itself: the metadata packs, which the walk
reaches through the decoding reader. It asked for them one pack per
call: on SQLite a length statement and a BLOB handle each, on
PostgreSQL one statement and round trip each (2,399 of the 7,219
statements of a mark over 5,440 packs in the PostgreSQL agent's count).

`edges::Prefixes` hands the same prefixes out in pack order and reads
them through `Mechanics::pack_prefixes`, 256 packs per call: one BLOB
handle reused on SQLite, one statement on PostgreSQL.

gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs,
96 metadata packs), release, warm second mark, one sample, load 3.8:
  statements of a mark   298 -> 203
  BLOB handles opened    229 -> 135
  wall                   30.2 ms (31 ms mean before, under load 7.6)

Tests: kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5),
space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30)
pass; Clippy, fmt and the product boundary guard clean.

Production LOC: 188,488 -> 188,527 (delta +39)
  core     40,589 -> 40,589 (0)
  engine   90,299 -> 90,338 (+39)  layerfs-persistence +39
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent d600429 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…act (layerfs-project#41)

Eight commits from the PostgreSQL rebuild agent, each with its own
measurements (PostgreSQL 17 on loopback, one sample per case):

  97169f0  Phase A reads each batch from an indexed copy set:
              per extra batch 10 -> 5 statements, 21 -> 11 round trips
  11242b2  a wave's pack heads in one statement
  51944d2  a wave's pack ranges in one statement: mark walk
              13,020 -> 7,952 statements, 548.1 MB -> 18.6 MB received,
              10,746 -> 6,721 ms at 290,561 rows
  0bb8989  row checks before the Store-wide lock: lock hold
              154.4 -> 49.7 ms at 290,561 rows
  dc6734b  PostgreSQL-only history closure by primary key:
              9,386 -> 20.8 ms for 3,999 unpublished Commits
  305007f  fix: two digest rows naming one dead root made
              store.compact fail with Integrity("persistence")
  86f5a4a  dead digest rows deleted by primary key:
              12,622 -> 2,444 ms for 200,000 dead rows
  9dcecc8  visited set loaded without a second copy

An independent read-only review of the range found no case that keeps
too little and found the new closure statement equivalent to the shared
one on all 16 edge kinds. Open items it and the author raised, none
fixed here:
- the closure equivalence test exercises 2 of the 16 edges, so the
  PostgreSQL text can drift from the shared one unnoticed;
- the new closure is slower when every Commit is published (about
  20 -> 250 ms at 2,000 Layers; the server's JIT compiler);
- swap row counts are now taken by scratch-table name before the move,
  and a scratch mismatch is reported as Integrity before a Conflict;
- `pg_temp.gc_copy` lives until the session ends;
- `peakTempBytes` now includes the copy set;
- spec section 8 and one module comment do not describe the new order.

Checked on this merged tree (the agent's own runs were before its
rebase): postgresql_compact (31), postgresql_compact_fixtures (9),
postgresql_compact_units (8), postgresql_gc_mark (6),
postgresql_gc_fence (19), postgresql_identical_ids (3),
postgresql_statement_schema (2), postgresql_commit_round_trips (1),
postgresql_acquisition (4), postgresql_acquisition_compaction (4),
kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_compact_units (9),
sqlite_group_rows (7), compact_metrics (5), space_reclamation (3) pass;
Clippy, fmt and the product boundary guard clean. No test deleted.

Production LOC: 188,527 -> 188,683 (delta +156), first parent 9da6038
  core     40,589 -> 40,589 (0)
  engine   90,338 -> 90,494 (+156)  layerfs-persistence +156
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 9da6038 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The review of the PostgreSQL rebuild commits found two texts that no
longer described the code:

- Spec section 8, Phase B: the swap now locks the two scratch tables
  and takes the row counts and sums before the Store-wide lock
  (0bb8989). An "as built" note says so, including that a scratch
  mismatch is reported before a Conflict.
- `backend/postgresql/history/statements.rs`: the module comment said
  only statements with SQLite-only syntax are replaced. The history
  closure is replaced too; the comment now says why and that both files
  must follow the same edges.

No behaviour change.

Production LOC: 188,683 -> 188,683 (delta 0)
  core     40,589 -> 40,589 (0)
  engine   90,494 -> 90,494 (0)
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent f22faa7 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`gc_bench` and `gc_profile` are SQLite only, so PostgreSQL GC had no
committed way to repeat a measurement. `pg_gc_profile` is the probe the
PostgreSQL rebuild agent measured its commits with, cleaned up:

  pg_gc_profile build DB        a Store in its own database
  pg_gc_profile mark DB WORK    clone, run the mark, count its SQL work
  pg_gc_profile run DB WORK     clone, run store.compact, count it
  pg_gc_profile seed|sizes|drop

It prints wall time, the compaction's phases and the session's
statements, round trips, rows and bound bytes. A diagnostic: no receipt
and no claim. Documented in tools/ci/gc-bench.md.

First use, PostgreSQL 17 in Docker on loopback, release, one sample
each, on a Store of 72,641 objects in 1,360 packs (4 Stacks x 160
Layers x 99 files of 1 KiB, 8,000 dead objects):

                      6bca861      58aa062 (parent)
  mark                3,875.3 ms      1,727.2 ms
  mark statements     4,489           1,359
  store.compact       5,574.0 ms      2,936.0 ms
  compact statements  4,627           1,484
  1-min load          4.2 to 4.3      7.3

An example and a doc only: no product source changes.

Production LOC: 188,683 -> 188,683 (delta 0)
  core     40,589 -> 40,589 (0)
  engine   90,494 -> 90,494 (0)
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 58aa062 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the earlier commits a PostgreSQL mark still sent 1,359 statements
over a Store of 1,360 packs. The server log split them three ways:

- the history closure, 415 ms. EXPLAIN ANALYZE on the same Store: 409 ms
  with the server's JIT compiler on, 10.5 ms with it off.
- one scoped pack read per metadata pack the walk decodes (640);
- one inode value read per inode leaf (640).
The last two are cheap on the server (11 ms and 123 ms in all) but each
is a round trip.

Three changes, the first PostgreSQL only and the other two behind new
`Mechanics` calls whose defaults change nothing, so SQLite is as before:

1. The mark's snapshot transaction sets `jit` off (`snapshot_begin`).
2. `expect_scoped_packs`: before acquiring a pack the walk announces the
   next 128 metadata packs of its wave. PostgreSQL reads, in one
   statement, the rows `read_scoped_pack` would fetch for those of up to
   32 KiB and keeps them on the transaction; each scoped read then uses
   its row once. A longer pack, or one not announced, is read alone as
   before. The rows are the same rows, so the plan, the authentication
   and `mark_bytes` are unchanged.
3. `inode_values_ahead`: PostgreSQL returns every value of the blocks
   holding the asked ordinals and up to 4,096 ordinals past them; the
   walk keeps at most 65,536 values and asks again only for a leaf it
   cannot answer.

pg_gc_profile, PostgreSQL 17 in Docker on loopback, release, one sample
each, the Store of the parent commit's message (72,641 objects, 1,360
packs):

                      6bca861    parent        this commit
  mark                3,875.3 ms    1,727.2 ms    577.4 ms
  mark statements     4,489         1,359         100
  store.compact       5,574.0 ms    2,936.0 ms    1,483.3 ms
  compact statements  4,627         1,484         225
  1-min load          4.2 to 4.3    7.3           5.3

Objects kept (64,641), packs kept (1,280), mark_bytes (3,818,542) and
the receipt (dead bound 8,235,840, shrink 10,149,888) are identical in
all three.

New test `a_mark_sends_the_same_statements_for_twice_the_layers`
(postgresql_gc_mark): 8 and 16 Layers must cost the same statements and
reach every saved object. On the parent's source it fails with
[28, 44]; here it passes.

Not done: the same closure statement also runs in project removal's
write transaction, which keeps JIT on.

Tests: postgresql_compact (31), postgresql_compact_fixtures (9),
postgresql_compact_units (8), postgresql_gc_mark (7),
postgresql_gc_fence (19), postgresql_identical_ids (3),
postgresql_statement_schema (2), postgresql_commit_round_trips (1),
postgresql_acquisition (4), postgresql_acquisition_compaction (4),
postgresql_narrow_locations (3), postgresql_position_columns (4),
kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_compact_units (9),
sqlite_group_rows (7), compact_metrics (5), space_reclamation (3) pass;
Clippy, fmt and the product boundary guard clean. No test deleted.

Production LOC: 188,683 -> 188,813 (delta +130)
  core     40,589 -> 40,589 (0)
  engine   90,494 -> 90,624 (+130)  layerfs-persistence +130
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent e180a41 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the mark, the rebuild loads the kept object ids into a session
temp table. It used a binary COPY and handed the driver one row at a
time: 257.9 ms on the server's clock for 64,641 ids, about four
microseconds a row, nearly all of it waiting for the next row.

The ids now go as `unnest($1::bytea[])`, 8,192 per statement (about
300 KiB), and the pack ids as `unnest($1::bigint[])`, 32,768 per
statement. The ids are already ascending, so the key's index is built
by appending.

Server log (log_min_duration_statement=0), the 72,641-object Store of
the two commits before, one sample:
  visited ids load     257.9 ms, 1 statement -> 43.6 ms, 8 statements
  mark phase after the walk (server)   305.8 ms -> 86.9 ms

pg_gc_profile, release, one sample each, 1-min load 10.8 (higher than
the earlier rows, so read the direction, not the ratio):
  mark             577.4 ms -> 433.2 ms   (100 statements both)
  store.compact    1,483.3 ms -> 962.0 ms (225 -> 232 statements)
The receipt is unchanged (dead bound 8,235,840, shrink 10,149,888).

Tests: postgresql_compact (31), postgresql_compact_fixtures (9),
postgresql_compact_units (8), postgresql_gc_mark (7),
postgresql_gc_fence (19), postgresql_identical_ids (3),
postgresql_acquisition_compaction (4) pass; Clippy and fmt clean.

Production LOC: 188,813 -> 188,808 (delta -5)
  core     40,589 -> 40,589 (0)
  engine   90,624 -> 90,619 (-5)  layerfs-persistence -5
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent ba6cba3 and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…dges

The PostgreSQL history closure (dc6734b) is a second text of the
shared statement. The review of that commit found its data test reaches
2 of the 16 edge kinds, so an edge added to one file and not the other,
or a dropped VALUES row, would go unnoticed, and on that backend a
rebuild would then drop history the other keeps.

`the_postgresql_closure_follows_exactly_the_shared_edges` compares the
two texts: the same roots, the same sixteen (from kind, to kind,
column) edges and the same final insert. It needs no server. Checked
once by removing `('commit',c.merged_from_id)` from the PostgreSQL
text: the test fails; restored, it passes.

It compares text, not behaviour: a rewrite of either statement into
another shape will need this test rewritten with it.

Tests: postgresql_gc_mark (8) pass.

Production LOC: 188,808 -> 188,808 (delta 0)
  core     40,589 -> 40,589 (0)
  engine   90,619 -> 90,619 (0)
  control  51,073 -> 51,073 (0)
  api       6,527 ->  6,527 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent ca82dbc and the staged tree; a test-only commit

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
yifanxuaaa and others added 4 commits November 18, 2026 02:24
…ark-header-edges

Brings in 3fc66bc (#733: PostgreSQL Stores run on any machine;
store.update replaces attach/detach) so the GC speed work is verified on
the tree it will merge into. No conflict; one file merged textually
(docs/architecture/engine/store/postgresql.md).

Production LOC: 188,808 -> 188,960 (delta +152), first parent 032b212
  core     40,589 -> 40,589 (0)
  engine   90,619 -> 90,619 (0)
  control  51,073 -> 51,230 (+157)
  api       6,527 ->  6,522 (-5)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 032b212 and the staged tree; the whole delta is #733's

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the rename a rebuild lets go of the writer lane and settles: it
removes its artifacts, then clears its admission intent and the kept_node
rows in the installed Store through a raw connection. That transaction did
not take the lane. A writer admitted in that window (a Commit that had been
waiting for the rebuild) has no busy handler, so it met the settle's write
lock and failed with SQLITE_BUSY.

The settle now takes the lane for that one transaction and waits its turn.
A lane that cannot be taken keeps the op record for the next run, like a
Store that could not be cleared. The order of the settle, the hooks and the
leftover rules are unchanged; the abort paths already hold the run's lane.

This is the cause of the acceptance matrix's AC-49 arm failing at
sqlite_compact_matrix.rs:149/:744 (also at 6bca861, before this branch).
With the fix it passed 8 of 8 samples.

Test: the_settle_waits_for_a_writer_that_holds_the_lane takes the lane in
the settle window and holds it 400 ms; the compaction must return after the
release. It fails on the parent's install.rs ("the settle wrote to the
Store while another writer held the lane").

The matrix report is regenerated: its test counts follow the new test here
(sqlite_compact 36) and the one added with the PostgreSQL closure check
(postgresql_compact 31).

Checks: sqlite_compact (35 run on macOS), sqlite_compact_matrix (1),
sqlite_compact_crash (15), sqlite_compact_fixtures (1) pass; fmt clean.

Production LOC: 188,960 -> 188,967 (delta +7)
  core     40,589 -> 40,589 (0)
  engine   90,619 -> 90,626 (+7)  layerfs-persistence +7
  control  51,230 -> 51,230 (0)
  api       6,522 ->  6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 9971d6e and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`Session::seal_holding` kept the writer lane in a parameter and the open
connection in the session state. Every error exit before the checked close
(a hard-linked or symlinked Store on macOS, an obstructed checkpoint, a
failed close) dropped the state first. A writable close waits for the lane
like any write, so it waited for the lane the same call held: the seal never
returned. `store.compact` seals through this path, so a rebuild refused at
its seal hung as well.

The seal now makes its lane the connection's owner while the connection is
open. An error exit closes under that hold and then frees the lane; after
the checked close the lane goes back to the caller as before.

This was already so at 6bca861: seal_allocation's hard-link and symlink
refusal tests ran past 60 s there and now finish in 0.08 s.

Test: writer_admission::a_refused_seal_returns_and_frees_the_lane (Linux
and macOS) obstructs the checkpoint with a reader's older snapshot. The
seal must fail Busy within 20 s and the next open must write. On the
parent's seal.rs it fails with "the refused seal returns: Timeout".

Checks: writer_admission (15) and seal_allocation (5) pass; fmt clean.

Production LOC: 188,967 -> 188,972 (delta +5)
  core     40,589 -> 40,589 (0)
  engine   90,626 -> 90,631 (+5)  layerfs-persistence +5
  control  51,230 -> 51,230 (0)
  api       6,522 ->  6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 4c359ef and the staged tree

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The fixture asserted that a new selection's synchronous_commit is `off`.
The product default became `local` with Normal mode (6bca861, #732), so
every test in postgresql_doctor_access, postgresql_doctor_cancellation and
postgresql_doctor_physical that built the fixture panicked at
pg_doctor/mod.rs:56. The fixture is a Disposable Store; it now selects `off`
itself and keeps the assertion.

Checks: postgresql_doctor_access (10), postgresql_doctor_cancellation (9),
postgresql_doctor_physical (16) pass against PostgreSQL 17.

Production LOC: 188,972 -> 188,972 (delta 0)
  core     40,589 -> 40,589 (0)
  engine   90,631 -> 90,631 (0)
  control  51,230 -> 51,230 (0)
  api       6,522 ->  6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent and the staged tree; tests only

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit 4c296e9 into layerfs_v2_0_0 Nov 17, 2026
2 checks passed
@yifanxuaaa
yifanxuaaa deleted the claude/gc-mark-header-edges branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant