Skip to content

perf(postgresql): cut Store round trips on open, mount, Commit and reads (layerfs-project #61) - #581

Merged
yifanxuaaa merged 21 commits into
layerfs_v2_0_0from
pg-perf
Nov 6, 2026
Merged

yifanxuaaa merged 21 commits into
layerfs_v2_0_0from
pg-perf

Conversation

@yifanxuaaa

Copy link
Copy Markdown
Contributor

Cuts PostgreSQL Store round trips on the open, mount, Commit and read paths, for Ephemeral-AI-Lab/layerfs-project#61 (sub-issues #62–#70). Measured in Ephemeral-AI-Lab/layerfs-project#71.

What changes

Measured

Count test, PostgreSQL round trips before → after:

Scenario Before After
Session open 10 1
Commit, nothing changed 8 2
Commit, 1,000 files 247 72
Commit, 1,000 similar files 12,241 65
Commit, 64 MiB 330 115
Read 64 MiB sequentially 1,616 302
Stat 1,000 committed files 1,034 47

fs-bench-quick (Ephemeral-AI-Lab/layerfs-project#71): 162/162 PASS. The median PostgreSQL − SQLite gap per case:

Machine Before After
Docker +352 ms +133 ms
Native +255 ms +58 ms
Lima +495 ms +133 ms
Apple Container +498 ms +133 ms

A bisect across our builds found no SQLite regression. The report is on layerfs-benchmark branch pg-perf-62.

Checks

  • PostgreSQL and SQLite focused tests, the count tests, and the mount lane, all run per binary in fresh containers.
  • Clippy is clean on the touched packages.
  • Durable mode was not run (disabled by owner).

Not in this PR

Production LOC: 137772 -> 139048 (delta +1276), tools/production_loc.py, base origin/layerfs_v2_0_0 (6d820c756b) to head ddd1bc83cd. Each commit records its own comparison. Reference absent (retired R9c; last counted 65,417 at tree 498dd1917812).

🤖 Generated with Claude Code

yifanxuaaa and others added 21 commits November 6, 2026 07:58
…s (layerfs-project #62)

SqlWork gains three counters, filled by the PostgreSQL session and zero on
SQLite:

- round_trips: every exchange that waits for the server (prepare, statement
  execution, BEGIN/COMMIT/ROLLBACK scripts, schema scripts, COPY's prepare,
  start and finish, SHOW server_version at open, and the lane connection's
  BEGIN ... FOR UPDATE and ROLLBACK). Connection handshakes are excluded.
- prepares: statement-cache misses, the open's SHOW and each COPY's
  unnamed statement.
- connects: the session's connection and its lane connection.

Lane counters live in atomics on the session, since the lane is held and
released without the state lock; diagnostics() adds them in.

The Store service now observes PostgreSQL Workspace reader sessions the same
way as SQLite ones (bootstrap.rs: a small Observed trait over both providers,
one shared registration/retirement helper), so readers are no longer
"not observed". The SDK telemetry map gains the three names; daemon
observation sections 13/14 append them after the phase pairs so existing
columns keep their place, and tools/decode_observations.py names them.

Tests: no test deleted. postgresql_open, postgresql_acquisition,
postgresql_identical_ids and layerstack_removal pass on PostgreSQL 17.11;
sdk_fs_reads_replay and workspace_reader_cache pass on SQLite; clippy clean
for layerfs-persistence, layerfs-store-service and layerfs-daemon with
postgresql features.

Production LOC: 137772 -> 137856 (delta +84)
Scope: tools/production_loc.py (default scopes, combined); parent 6d820c7
vs staged tree; engine 62799 -> 62883, core/connect/api unchanged;
reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions (layerfs-project #67)

- branch_snapshot is one joined statement (Branch, LayerStack, base Layer
  root, head Commit; branch_snapshot.sql). On PostgreSQL it runs alone in the
  new Mode::Statement (no BEGIN/COMMIT; schema-qualified text; a second
  statement is refused before it is sent): 6 -> 1 round trips. SQLite runs
  the same statement in its read transaction: 6 -> 3 statements.
- commit_branch reads that joined row instead of Branch, LayerStack, base
  root and the head Commit twice; the Commit insert is ON CONFLICT DO NOTHING
  (an existing identity is then verified, as before); the parent line and the
  run check are one statement (lineage_parent.sql). Lane-held work for an
  unmoved Commit (lane BEGIN, head read, commit_branch, lane ROLLBACK):
  21 -> 10 round trips on PostgreSQL; 19 -> 10 statements on SQLite.
- reserve: Mechanics::allocate; PostgreSQL takes both blocks in one
  UPDATE ... RETURNING with the bounds in the statement (no row: the cursor
  is read to tell a bound from a missing policy row): 4 -> 3 round trips.
  SQLite keeps cursor + advance.
- publish: one Mode::Publish transaction instead of stage + rows. COPY
  BINARY of the bodies first (its prepared statement now cached), then the
  lane row, then one data-modifying statement for locations, totals, inode
  values, released ordinals and signatures, then COMMIT. The lane row is
  held only for the rows statement and COMMIT. Any failure rolls the whole
  publication back. Save of 100 files 26 -> 16, 8 MiB non-repeating 45 -> 31.
- New count test tests/postgresql_commit_round_trips.rs (both engines).
- Docs: engine/store/postgresql.md, backends.md, history-publication.md.

Not changed: publication wave size (shared core policy, 4 MiB), history
write framing (BEGIN/COMMIT stay for writes).
Tests deleted: none.

Production LOC: 137856 -> 138008 (delta +152)
Scope: tools/production_loc.py over the parent (68c24f5) and the staged
tree (git archive of each); engine/layerfs-persistence 12364 -> 12516.
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (layerfs-project #62)

tests/postgresql_round_trips.rs drives a fresh Store through the Store
service without FUSE and prints one `ROUND_TRIPS <json>` line per scenario:
session open, Store open (writer + control reader), the mount path's
Workspace bind, Commits (up-to-date, net-empty 1,000 files, C01-shaped
1,000 files, 1,000 similar 23-byte files, C04 shape, 100 files, one 64 MiB
non-repeating file, a second one-file Commit) and cold reads (stat 1,000
committed files, sequential 128 KiB reads of the 64 MiB file after a Store
reopen). Each carries connects, round_trips, prepares, statements and
transactions for writer and readers, per-phase SQL for Commits, and on
PostgreSQL the server's pg_stat_statements call deltas plus a SELECT 1
latency calibration. It asserts only that counted work happened.
PostgreSQL needs LAYERFS_TEST_POSTGRES_URL (NOT_RUN without it); the
SQLite test always runs with the postgresql feature.

Per-phase SQL comes from a small hook in the Store service's telemetry:
attribute_sql(probe) makes each `phase` run while the thread records read
the probe before and after, and phase_sql() returns the differences. It is
independent of the timing tree, whose node bound stops recording the late
Commit phases of a 1,000-file Commit.

Tests: no test deleted. The new test passes on PostgreSQL 17.11 and SQLite
in about 5 s; clippy clean for layerfs-store-service and layerfs-daemon with
postgresql features.

Production LOC: 137856 -> 137873 (delta +17)
Scope: tools/production_loc.py (default scopes, combined); parent 68c24f5
vs staged tree; engine 62883 -> 62900, core/connect/api unchanged;
reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…connecting (layerfs-project #63)

An open used to pay, after the handshake, `SHOW server_version` (prepare +
execute), a validation transaction with two information_schema scans (eight
round trips), and later a policy read per handle (BEGIN/SELECT/COMMIT). Now:

- The open's check is one unprepared simple-query string (one round trip)
  that also reads the server version and the frozen policy row; the session
  keeps both, so `PackPersistence::policy` costs nothing afterwards.
- `PostgresCheck` on `PostgresSelection` selects it: `Full` (default; Init,
  Store connect, the host Store service) keeps the exact table set and live
  definition check in one REPEATABLE READ snapshot; `Light` reads only the
  history_meta and policy rows, the recorded source's length and the
  acquisition tables by name (no information_schema scan). Both refuse a
  wrong binding, catalog, incarnation, identity format or schema version
  (Integrity) and an invalid policy (Malformed) as before.
- `PostgresHandles::attach_read_only` opens under the recorded binding and
  returns it, for a reach check that compares it itself (#65).
- One TLS connector per process; sslmode is used exactly as given (docs say
  when to choose disable/require).

Counters (measure agent's SqlWork, merged from pg-perf 68c24f5): an open
is connects 1, round trips 1, prepares 0 (was 1 / 10 / 1 plus 3-4 per
policy read). Wire proxy (postgresql_open_cost, TCP handshake excluded):
read-only session open 13 -> 4 round trips; daemon Store open (writer +
control reader) 2 conns / 34 -> 2 conns / 8; mount reader open 17 -> 4.

Tests: new postgresql_open_check (both checks' refusals, 1 RT per open),
postgresql_open_cost (wire counts). PASS on pgperf-pg: postgresql_open_check,
postgresql_open_cost, postgresql_acquisition, postgresql_identical_ids,
layerstack_removal, store-service postgresql_open; sqlite_publication.
No tests deleted.

Production LOC: 137856 -> 138033 (delta +177); tools/production_loc.py
combined scope, parent tree vs staged tree via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erfs-project #63)

A simple-query script counts one statement, as `Transaction::batch` does,
so the count test's invariant (round trips >= statements) holds.
After #63 with the measure agent's count test (0bdc970, receipts
baseline-0bdc9700b7): session_open PG conn/rt/prep/stmt/tx 1/10/1/3/1 ->
1/1/0/1/1; store_open 2/28/-/8/4 -> 2/2/0/2/2; workspace_reader_open
1/30/-/12/3 -> 1/17/6/9/2; workspace_reader_open.cold 1/48/-/29/3 -> 1/35/7/26/2.

Production LOC: 138050 -> 138049 (delta -1); tools/production_loc.py
combined scope, parent tree vs staged tree via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emon open (layerfs-project #64)

- A PostgreSQL Store opens with shared readers (`Store::with_shared_readers`):
  every Workspace's healthy reader returns to the idle readers at unmount
  (at most IDLE_READERS = 4) and the next mount takes one, so a mount after
  an unmount opens no connection. SQLite keeps its readers as before.
- An idle reader is checked before a mount takes it (`StoreReader::usable`,
  `PostgresStorage::usable`: one empty query, outside the ReadSet lock); a
  quarantined or dead one is dropped and a new reader opens, so no operation
  runs on a broken connection and none is retried. Quarantine rules in
  transaction.rs are unchanged.
- The control reader of a PostgreSQL Store opens at its first use
  (`ReadSet::control`), not at Store open. In the daemon the first Commit
  uses it, so its connection moves from sandbox.create to the first Commit.
- Readers and a Sandbox daemon's writer open with the light check (#63);
  the host Store service and Init keep the full check.
- Lane pre-open is NOT changed: the lane connection still opens inside the
  first Commit (owner decision).

Server connections per daemon: writer + lane (first Commit) + control
(first use) + one per mounted Workspace + up to 4 idle.

Counts (measure count test 0bdc970, PG conn/rt): store_open 2/28 at base
-> 2/2 after #63 -> 1/1; workspace_reader_open 1/30 -> 1/17 (first mount
of a fresh daemon still opens); commit.net_empty_1000 connects 1 -> 2 (lane
+ lazily opened control). Wire proxy (postgresql_open_cost): daemon Store
open 2 conns/34 RTs at base -> 1/4; second mount 1/17 -> 0/1; a mount after
the idle reader's connection was cut opens a new reader (1/4).

Tests: PASS postgresql_open_cost, store-service postgresql_open,
postgresql_round_trips (PG); SQLite workspace_reader_cache, sdk_fixed_costs,
object_demand_errors, daemon store_reader_lifecycle. daemon
mounted_store_read NOT_RUN here (needs fusectl mount in a privileged
container; fails at fusectl.rs:55 before any LayerFS code). Clippy clean
for persistence, store-service, daemon (postgresql features). No tests
deleted.

Production LOC: 138049 -> 138163 (delta +114); tools/production_loc.py
combined scope, parent tree vs staged tree via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Store service (layerfs-project #65)

On PostgreSQL, sandbox.create's host side opened three connections: the
reach check's binding read, then `open::hold`'s full Store service (writer
and control reader, both fully validated) only so `sandbox_store` could
list projects. Every other Sandbox op (mount, exec, commit, unmount, close)
also opened that Store service when the process did not hold it, which a
CLI process never does (bench: 2 host connections per mount and per exec).

- New `store/reader.rs`: the process's one held read session per
  PostgreSQL Store (keyed by Store path and dialled URL), opened with the
  light check under the schema's recorded binding
  (`PostgresHandles::attach_read_only`). A held session is checked with one
  empty query before use; one that does not answer is dropped and a new one
  opens. The open-Store reaper closes it after 60 s unused; `open::close`
  (disconnect/move) forgets it.
- The reach check now reads the binding and finds the project on that
  session (binding check + project lookup + reach in one session).
  `sandbox_store` no longer lists projects; every route to sandbox.create
  runs the reach check first. NoSuchProject keeps its code and message but
  is now reported at the reach check, i.e. before deploy/image/platform
  checks (only differs when both are wrong).
- PostgreSQL Sandbox ops no longer call `open::hold`: mount's Branch id and
  status's head read on the held session (`lookup::project_in/branch_in`,
  same refusals); exec, commit, unmount and close open no connection. The
  caller's Store lease is held for each call; previously it lived with the
  open Store for up to 60 s after the call.

Counts (wire proxy, postgresql_open_cost; TCP handshake excluded): host
side of create 3 connections / 49 round trips -> 1 / 8 (emulation of the
same persistence calls; the api path adds nothing). Mount host side: 2
connections (bench, CLI) -> 1 (or 0 within 60 s in a long-lived process).
Exec/commit/unmount/close host side: 2 connections -> 0.

Tests: new layerfs-api postgres_sandbox_reach (NoSuchProject and another
binding refused at the reach check, nothing started). PASS on pgperf-pg:
postgres_sandbox_reach, postgres_connect, postgres_refusals,
sandbox_machines, store_routes, machine_calls. NOT_RUN here: sandbox_family,
store_open_cache, store_reaper (launcher /tmp is overlayfs: project.init
refuses before any changed code). Clippy clean for layerfs-control,
layerfs-api, layerfs-machine with and without postgresql. No tests deleted.

Production LOC: 138163 -> 138417 (delta +254); tools/production_loc.py
combined scope, parent tree vs staged tree via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the measure agent's per-scenario round-trip count test
(layerfs-store-service tests/postgresql_round_trips.rs) and its telemetry.

Production LOC: 138008 -> 138025 (delta +17)
Scope: first-parent comparison (72ebf96 -> merge tree) with
tools/production_loc.py over git archive of each; the delta is the merged
store-service telemetry. Reference absent (retired R9c; last counted 65,417
at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…reate (layerfs-project #66)

`reach()` for an apple-container machine ran `container inspect` (a
subprocess polled at 20 ms steps) on every PostgreSQL Sandbox call to turn a
loopback URL into the container network's gateway.

- `Lasting` gains `gateway: Option<Ipv4Addr>`; apple-container's
  `bring_up` (create and every start) reads it once and the machine record
  keeps it (`lasting.gateway`, machine/spec.rs). Every start replaces it.
- `reach` uses the recorded gateway with no subprocess. A stale one is
  dialled as recorded and fails explicitly at the connection; nothing looks
  it up again. A record without one (started before this change, or a start
  that could not read it) still reads it from the machine, as before.
- Docker/Lima: not changed. Docker dials `host.docker.internal`, resolved
  per connection by the container's resolver (hosts entry on Linux, Docker
  Desktop's embedded DNS on macOS); with #63-#65 a create makes 2
  connections instead of 5, so it is paid at most twice. Not measured.

Not verified on a live Apple Container machine (none runs here): the
per-create inspect cost and the re-measured create gap are open for the
bench agent. Proven by machine_reach
`apple_container_uses_the_gateway_its_start_recorded` (no `container` call
with a recorded gateway; lookup kept without one).

Tests: PASS layerfs-machine conformance, machine_reach, machine_spec,
machine_providers, machine_platform, machine_adapter, machine_lima,
machine_probe, machine_ssh, disabled_platforms; layerfs-control
typed_records. machine_forward `lima_forwards_through_its_own_ssh_config`
FAILED (assert !runs(pid) right after forward_close: the ssh stand-in's exit
races the check; this change touches no forward code). macOS host build not
run (the macOS cargo home lacks libproc offline). No tests deleted.

Production LOC: 138417 -> 138429 (delta +12); tools/production_loc.py
combined scope, parent tree vs staged tree via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re the lane (layerfs-project #67)

- Save: a whole-file object whose ring proposal is still held by this Save
  (pending or sealed, unpublished) sealed and published that proposal before
  selection, so 1,000 similar 23-byte files cost 1,000 publications and
  1,000 one-object packs, while selection stored every one FULL. When no
  PREFIX record can be narrower than the object's FULL record
  (delta::record::prefix_cannot_win: a PREFIX adds the 32-byte base id and a
  zstd frame of at least PREFIX_FRAME_MIN = 14 bytes), the proposal is now
  dropped: selection answers FULL whatever the base. The FULL record is
  prepared once (select::prepare_full) and handed to selection; selection's
  own ring fallback skips the same futile trials. Records are unchanged
  (all FULL before and after); packs and stored bytes fall.
  Count test commit.similar_files_1000: PG 12,241 -> 226 round trips,
  2,041 -> 39 transactions; SQLite 11,211 -> 3,213 statements,
  1,040 -> 41 transactions. Persistence test, one Save: 1,000 x 23 B
  7,006 -> 11 round trips, 1,000 -> 1 pack, 60,000 -> 28,032 stored bytes;
  200 x 200 B 1,026 -> 10 round trips, 146 -> 1 pack, 13,272 -> 8,632 bytes.
- Commit: the candidate is validated and its install prepared before the
  publication lane (immutable reads only); only an in-lane merge step that
  changes the candidate validates again ("validate.lane"). PG lane-held
  round trips for an unmoved Commit: 22-38 -> about 10.
- Tests: codec_frames no_prefix_frame_is_shorter_than_its_declared_minimum;
  postgresql_commit_round_trips asserts one publication for the similar
  files and reports 200 B similar files.
- Docs: physical-storage/saves.md, engine/commit/publish-and-merge.md,
  engine/store/history-publication.md.
Tests deleted: none.

Production LOC: 138025 -> 138094 (delta +69)
Scope: tools/production_loc.py over the parent (17013c3) and the staged
tree (git archive of each); core/layerfs-storage 9068 -> 9105,
engine/layerfs-store-service 10819 -> 10851. Reference absent (retired R9c;
last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rustfmt +1.85.1 on the files these commits changed; no behaviour change.

Production LOC: 138429 -> 138459 (delta +30, formatting only: wrapped
lines); tools/production_loc.py combined scope, parent tree vs staged tree
via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g Commit (layerfs-project #68)

- Fetch::published: after an acknowledged publication the locations this
  Save won are kept in the locator cache (they are the Store's winners)
  instead of invalidated; lost ones are still invalidated and located again.
  The Save's metadata pack bodies (the published Arcs, at most 2 MiB) and
  pooled inode values (at most 512 KiB) are kept until the next Save.
- Only Storage::reader_after_save reads them: a reader that continues the
  latest Save's operation. Every other reader still acquires every body it
  needs (sqlite_transaction_units' fresh-reader assertion holds unchanged).
  Everything returned is authenticated as before.
- Save::read_objects locates only ids this Save does not hold (pending or
  sealed); the sealed-location fallback no longer locates eagerly.
- Commit: logical.count and bindings share one after-save reader (and its
  bounded caches); the in-lane step does the same.
- Per-directory read-back is gone rather than batched: the bindings and
  logical-count walks now cost 0-6 round trips in every scenario.
Count test, PG round trips (logical.count / bindings / Commit total), from
the previous commit: files_1000 28/33/225 -> 2/4/164; c04 110 dirs + 1,000
files 57/147/455 -> 5/5/146; files_100 17/16/93 -> 6/3/64; random 64 MiB
15/15/243 -> 7/3/220; second_one_file 12/8/66 -> 6/0/52. SQLite statements
c04 3,060 -> 2,751. Resident: bounded at 2 MiB of bodies (shared Arcs the
Save already held) + 512 KiB of values per Storage handle.
- Docs: physical-storage/reads.md, engine/commit/publish-and-merge.md.
Tests deleted: none.

Production LOC: 138094 -> 138263 (delta +169)
Scope: tools/production_loc.py over the parent (3868d1a) and the staged
tree (git archive of each); core/layerfs-storage 9105 -> 9274. Reference
absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-project #69 work-around)

file_roots (one per 64-row construction page) ran as a REPEATABLE READ
transaction: BEGIN, the digest_roots array statement, COMMIT. It is one
statement, so on PostgreSQL it now runs alone in Mode::Statement
(HistoryProvider::read_statement): digest_roots.sql names the schema ({s}),
and array statements replace {s} with the Store's schema. 3 -> 1 round trip
per page; SQLite unchanged (it runs the same body in its read transaction).
The FILE_DIGEST_WINDOW contract constant (64) is unchanged; widening it is
an owner decision (#69).

Count test, PG construct round trips: files_1000 111 -> 79, similar files
107 -> 75, c04 95 -> 59, files_100 12 -> 8 (Commit totals 164 -> 132,
157 -> 125, 146 -> 110, 64 -> 60).
Docs: engine/store/postgresql.md (Statement mode users).
Tests deleted: none.

Production LOC: 138263 -> 138263 (delta 0)
Scope: tools/production_loc.py over the parent (317896b) and the staged
tree (git archive of each); the change replaces lines in place
(SQL text and the provider call). Reference absent (retired R9c; last
counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-project #70)

PostgreSQL pays a round trip per statement, so committed reads now take
fewer, larger statements:

- Port hint `PackPersistence::whole_pack_bytes` (default: the control
  prefix, so SQLite is unchanged). PostgreSQL returns the 256 KiB pack
  limit: GroupPlan takes such packs whole, and the first statement of a
  PostgreSQL acquisition now fetches 256 KiB of body with the descriptor,
  so a pack costs one statement and stays in the kept pack cache.
- A demand reads its uncached whole-taken packs in one `read_packs` call
  when there are at least two (Fetch::whole_packs), within the 2 MiB pack
  cache, reserving the demand's kept packs first.
- PostgreSQL `read_packs` is one statement: the server returns bodies only
  when every id is present and the total is within the read budget.
  Descriptors no longer copy the body column (rows::record_first).
- Sequential base read-ahead: a base read of chunked content that continues
  the client's previous read reads 2 MiB further in the same demand; the
  extra objects stay in the bounded canonical cache. One slot per client,
  off for caches under 8 MiB and for memory-only clients.
- SourceView::lookup remembers sibling file lengths (F-032) like Mutation
  planning already did, so a stat walk is one length demand per 128 files.

Count test (postgresql_round_trips, cold reader, LAYERFS_CONSTRUCTION_WORKERS=1):
  read.sequential_random_64mib  PG 1616 -> 302 round trips, SQLite 1138 -> 505
  read.stat_files_1000          PG 1034 -> 47,  SQLite 1034 -> 47
Commit counts unchanged.

Tests: postgresql_commit_round_trips asserts read_packs is one round trip,
found or missing. No tests deleted.

Production LOC: 138263 -> 138406 (delta +143)
Scope: tools/production_loc.py combined (api, connect, core, engine;
reference absent (retired R9c; last counted 65,417 at tree 498dd19)),
parent tree vs staged tree via git archive. Subtotals: layerfs-storage
9274 -> 9327, layerfs-persistence 12516 -> 12515, layerfs-workspace
8991 -> 9082.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (layerfs-project #62)

The traced-call telemetry test asserted 27 SqlWork fields; #62 added
round_trips, prepares and connects (SQL_FIELDS = 30). Expect 30 and check
the three new fields are present and zero on this SQLite fixture.

Found by running the mount lane (each binary in its own container with
/dev/fuse and SYS_ADMIN, as nightly does) on the merged pg-perf tree:
43/44 binaries passed, this one failed on the width only.

Tests deleted: none.
Production LOC: 138992 -> 138992 (delta 0); tools/production_loc.py over
git archive of HEAD and of the staged tree. Test-only change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ount test (layerfs-project #69)

Each Commit scenario now prints `STORAGE <json>` (new objects, packs,
canonical, stored and metadata bytes, record kinds and the process's VmHWM
so far), and the PostgreSQL run prints the Store's running totals and each
table's pg_total_relation_size/heap/TOAST before its schema is dropped.
Test-only; no tests deleted.

Production LOC: 138992 -> 138992 (delta 0)
Scope: tools/production_loc.py combined (api, connect, core, engine;
reference absent (retired R9c; last counted 65,417 at tree 498dd19)),
parent tree vs staged tree via git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…page to 1,024 (layerfs-project #69)

Owner decision (2026-11-06): widen FILE_DIGEST_WINDOW and the overlay page
PAGE_ROWS from 64 to 1,024.

- History contract FILE_DIGEST_WINDOW and the workspace port's window are
  1,024: construction asks `file_roots` once and records digests once per
  captured inode page instead of per 64 rows.
- Overlay PAGE_ROWS is 1,024. Every paged statement whose page is PAGE_ROWS
  (inode and name capture, source names) takes its LIMIT from page_limit!,
  asserted equal to PAGE_ROWS at compile time.
- Operation-record windows keep 64 under a new RECORD_ROWS: they are the key
  window of core content's construction backing (filesystem/state/store.rs,
  64 keys in one 64 KiB envelope), so get_many, key/entry pages, the
  construction writer's changes per job and the cursor's record windows use
  RECORD_ROWS (record_limit!). Widening them is a core content contract
  change and is not part of this decision.
- get_many keys are a Vec instead of a PAGE_ROWS-sized stack array.
- Digest records keep only the last root per content before an engine
  without an array form inserts them, and `file_roots` answers a repeated
  content once: with one 1,024-row page, 1,000 empty files are no longer
  1,000 SQLite inserts and lookups (same final rows as one at a time).

Count test (LAYERFS_CONSTRUCTION_WORKERS=1), PostgreSQL round trips /
SQLite statements, before -> after:
  files_1000   132 -> 72  / 3139 -> 3079   (construct 79 -> 19)
  similar_1000 125 -> 65  / 3144 -> 3084
  C04          110 -> 73  / 2751 -> 1196   (construct 59 -> 20)
  files_100     60 -> 61  /  267 -> 267
  64 MiB       218 -> 218 /  527 -> 525
Storage: unchanged except C04 +121 stored bytes (one more metadata pack).

Tests adapted, none deleted: fixtures that asserted 64-row windows now size
themselves from PAGE_ROWS / RECORD_ROWS with unchanged assertions
(overlay captured_namespace, indexed_operation_record, ..._cursor;
workspace captured_namespace_cursor, captured_namespace_order,
kernel_directory, namespace_view). The cursor growth test compares two
sizes above one record window (a 1,024-row page is as wide as the cursor's
1,024 point answers).

Production LOC: 138992 -> 139029 (delta +37)
Scope: tools/production_loc.py combined (api, connect, core, engine;
reference absent (retired R9c; last counted 65,417 at tree 498dd19)),
parent tree vs staged tree via git archive. Subtotals: layerfs-persistence
12714 -> 12719, layerfs-workspace 9082 -> 9073.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…L (layerfs-project #61)

Owner decision (2026-11-06): PostgreSQL publication waves up to 16 MiB;
SQLite keeps 4 MiB.

- New port hint PackPersistence::publication_bytes (default the shared
  4 MiB - 1 transaction bound). Storage applies it to the wave and
  transaction byte capacities (StorageCapacities::with_publication_bytes,
  which refuses anything outside 4 MiB - 1 ..= PUBLICATION_BYTES_MAX,
  16 MiB - 1).
- PostgreSQL returns PUBLICATION_BYTES_MAX; persistence validates each
  publication against its own provider's size. SQLite, its unit layout and
  headroom checks are unchanged.
- Rows per publication stay 8,191.

Count test (LAYERFS_CONSTRUCTION_WORKERS=1): commit.random_64mib PostgreSQL
218 -> 115 round trips (construct 160 -> 58); every other scenario and all
SQLite counts unchanged; stored bytes unchanged (64 MiB: 67,302,459 both).
Persistence count test: save 8 MiB 31 -> 16 round trips. Peak RSS of the
count-test process after the 64 MiB Commit 101.5 -> 115.9 MB (VmHWM, a
lifetime peak including the test's own 64 MiB buffer).

No tests deleted.

Production LOC: 139029 -> 139048 (delta +19)
Scope: tools/production_loc.py combined (api, connect, core, engine;
reference absent (retired R9c; last counted 65,417 at tree 498dd19)),
parent tree vs staged tree via git archive. Subtotals: layerfs-storage
9327 -> 9345, layerfs-persistence 12719 -> 12720.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit b4fa173 into layerfs_v2_0_0 Nov 6, 2026
2 of 5 checks passed
@yifanxuaaa
yifanxuaaa deleted the pg-perf branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant