Repository navigation
perf(postgresql): cut Store round trips on open, mount, Commit and reads (layerfs-project #61) - #581
Merged
Merged
Conversation
…s (layerfs-project #62) SqlWork gains three counters, filled by the PostgreSQL session and zero on SQLite: - round_trips: every exchange that waits for the server (prepare, statement execution, BEGIN/COMMIT/ROLLBACK scripts, schema scripts, COPY's prepare, start and finish, SHOW server_version at open, and the lane connection's BEGIN ... FOR UPDATE and ROLLBACK). Connection handshakes are excluded. - prepares: statement-cache misses, the open's SHOW and each COPY's unnamed statement. - connects: the session's connection and its lane connection. Lane counters live in atomics on the session, since the lane is held and released without the state lock; diagnostics() adds them in. The Store service now observes PostgreSQL Workspace reader sessions the same way as SQLite ones (bootstrap.rs: a small Observed trait over both providers, one shared registration/retirement helper), so readers are no longer "not observed". The SDK telemetry map gains the three names; daemon observation sections 13/14 append them after the phase pairs so existing columns keep their place, and tools/decode_observations.py names them. Tests: no test deleted. postgresql_open, postgresql_acquisition, postgresql_identical_ids and layerstack_removal pass on PostgreSQL 17.11; sdk_fs_reads_replay and workspace_reader_cache pass on SQLite; clippy clean for layerfs-persistence, layerfs-store-service and layerfs-daemon with postgresql features. Production LOC: 137772 -> 137856 (delta +84) Scope: tools/production_loc.py (default scopes, combined); parent 6d820c7 vs staged tree; engine 62799 -> 62883, core/connect/api unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions (layerfs-project #67) - branch_snapshot is one joined statement (Branch, LayerStack, base Layer root, head Commit; branch_snapshot.sql). On PostgreSQL it runs alone in the new Mode::Statement (no BEGIN/COMMIT; schema-qualified text; a second statement is refused before it is sent): 6 -> 1 round trips. SQLite runs the same statement in its read transaction: 6 -> 3 statements. - commit_branch reads that joined row instead of Branch, LayerStack, base root and the head Commit twice; the Commit insert is ON CONFLICT DO NOTHING (an existing identity is then verified, as before); the parent line and the run check are one statement (lineage_parent.sql). Lane-held work for an unmoved Commit (lane BEGIN, head read, commit_branch, lane ROLLBACK): 21 -> 10 round trips on PostgreSQL; 19 -> 10 statements on SQLite. - reserve: Mechanics::allocate; PostgreSQL takes both blocks in one UPDATE ... RETURNING with the bounds in the statement (no row: the cursor is read to tell a bound from a missing policy row): 4 -> 3 round trips. SQLite keeps cursor + advance. - publish: one Mode::Publish transaction instead of stage + rows. COPY BINARY of the bodies first (its prepared statement now cached), then the lane row, then one data-modifying statement for locations, totals, inode values, released ordinals and signatures, then COMMIT. The lane row is held only for the rows statement and COMMIT. Any failure rolls the whole publication back. Save of 100 files 26 -> 16, 8 MiB non-repeating 45 -> 31. - New count test tests/postgresql_commit_round_trips.rs (both engines). - Docs: engine/store/postgresql.md, backends.md, history-publication.md. Not changed: publication wave size (shared core policy, 4 MiB), history write framing (BEGIN/COMMIT stay for writes). Tests deleted: none. Production LOC: 137856 -> 138008 (delta +152) Scope: tools/production_loc.py over the parent (68c24f5) and the staged tree (git archive of each); engine/layerfs-persistence 12364 -> 12516. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (layerfs-project #62) tests/postgresql_round_trips.rs drives a fresh Store through the Store service without FUSE and prints one `ROUND_TRIPS <json>` line per scenario: session open, Store open (writer + control reader), the mount path's Workspace bind, Commits (up-to-date, net-empty 1,000 files, C01-shaped 1,000 files, 1,000 similar 23-byte files, C04 shape, 100 files, one 64 MiB non-repeating file, a second one-file Commit) and cold reads (stat 1,000 committed files, sequential 128 KiB reads of the 64 MiB file after a Store reopen). Each carries connects, round_trips, prepares, statements and transactions for writer and readers, per-phase SQL for Commits, and on PostgreSQL the server's pg_stat_statements call deltas plus a SELECT 1 latency calibration. It asserts only that counted work happened. PostgreSQL needs LAYERFS_TEST_POSTGRES_URL (NOT_RUN without it); the SQLite test always runs with the postgresql feature. Per-phase SQL comes from a small hook in the Store service's telemetry: attribute_sql(probe) makes each `phase` run while the thread records read the probe before and after, and phase_sql() returns the differences. It is independent of the timing tree, whose node bound stops recording the late Commit phases of a 1,000-file Commit. Tests: no test deleted. The new test passes on PostgreSQL 17.11 and SQLite in about 5 s; clippy clean for layerfs-store-service and layerfs-daemon with postgresql features. Production LOC: 137856 -> 137873 (delta +17) Scope: tools/production_loc.py (default scopes, combined); parent 68c24f5 vs staged tree; engine 62883 -> 62900, core/connect/api unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…connecting (layerfs-project #63) An open used to pay, after the handshake, `SHOW server_version` (prepare + execute), a validation transaction with two information_schema scans (eight round trips), and later a policy read per handle (BEGIN/SELECT/COMMIT). Now: - The open's check is one unprepared simple-query string (one round trip) that also reads the server version and the frozen policy row; the session keeps both, so `PackPersistence::policy` costs nothing afterwards. - `PostgresCheck` on `PostgresSelection` selects it: `Full` (default; Init, Store connect, the host Store service) keeps the exact table set and live definition check in one REPEATABLE READ snapshot; `Light` reads only the history_meta and policy rows, the recorded source's length and the acquisition tables by name (no information_schema scan). Both refuse a wrong binding, catalog, incarnation, identity format or schema version (Integrity) and an invalid policy (Malformed) as before. - `PostgresHandles::attach_read_only` opens under the recorded binding and returns it, for a reach check that compares it itself (#65). - One TLS connector per process; sslmode is used exactly as given (docs say when to choose disable/require). Counters (measure agent's SqlWork, merged from pg-perf 68c24f5): an open is connects 1, round trips 1, prepares 0 (was 1 / 10 / 1 plus 3-4 per policy read). Wire proxy (postgresql_open_cost, TCP handshake excluded): read-only session open 13 -> 4 round trips; daemon Store open (writer + control reader) 2 conns / 34 -> 2 conns / 8; mount reader open 17 -> 4. Tests: new postgresql_open_check (both checks' refusals, 1 RT per open), postgresql_open_cost (wire counts). PASS on pgperf-pg: postgresql_open_check, postgresql_open_cost, postgresql_acquisition, postgresql_identical_ids, layerstack_removal, store-service postgresql_open; sqlite_publication. No tests deleted. Production LOC: 137856 -> 138033 (delta +177); tools/production_loc.py combined scope, parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erfs-project #63) A simple-query script counts one statement, as `Transaction::batch` does, so the count test's invariant (round trips >= statements) holds. After #63 with the measure agent's count test (0bdc970, receipts baseline-0bdc9700b7): session_open PG conn/rt/prep/stmt/tx 1/10/1/3/1 -> 1/1/0/1/1; store_open 2/28/-/8/4 -> 2/2/0/2/2; workspace_reader_open 1/30/-/12/3 -> 1/17/6/9/2; workspace_reader_open.cold 1/48/-/29/3 -> 1/35/7/26/2. Production LOC: 138050 -> 138049 (delta -1); tools/production_loc.py combined scope, parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emon open (layerfs-project #64) - A PostgreSQL Store opens with shared readers (`Store::with_shared_readers`): every Workspace's healthy reader returns to the idle readers at unmount (at most IDLE_READERS = 4) and the next mount takes one, so a mount after an unmount opens no connection. SQLite keeps its readers as before. - An idle reader is checked before a mount takes it (`StoreReader::usable`, `PostgresStorage::usable`: one empty query, outside the ReadSet lock); a quarantined or dead one is dropped and a new reader opens, so no operation runs on a broken connection and none is retried. Quarantine rules in transaction.rs are unchanged. - The control reader of a PostgreSQL Store opens at its first use (`ReadSet::control`), not at Store open. In the daemon the first Commit uses it, so its connection moves from sandbox.create to the first Commit. - Readers and a Sandbox daemon's writer open with the light check (#63); the host Store service and Init keep the full check. - Lane pre-open is NOT changed: the lane connection still opens inside the first Commit (owner decision). Server connections per daemon: writer + lane (first Commit) + control (first use) + one per mounted Workspace + up to 4 idle. Counts (measure count test 0bdc970, PG conn/rt): store_open 2/28 at base -> 2/2 after #63 -> 1/1; workspace_reader_open 1/30 -> 1/17 (first mount of a fresh daemon still opens); commit.net_empty_1000 connects 1 -> 2 (lane + lazily opened control). Wire proxy (postgresql_open_cost): daemon Store open 2 conns/34 RTs at base -> 1/4; second mount 1/17 -> 0/1; a mount after the idle reader's connection was cut opens a new reader (1/4). Tests: PASS postgresql_open_cost, store-service postgresql_open, postgresql_round_trips (PG); SQLite workspace_reader_cache, sdk_fixed_costs, object_demand_errors, daemon store_reader_lifecycle. daemon mounted_store_read NOT_RUN here (needs fusectl mount in a privileged container; fails at fusectl.rs:55 before any LayerFS code). Clippy clean for persistence, store-service, daemon (postgresql features). No tests deleted. Production LOC: 138049 -> 138163 (delta +114); tools/production_loc.py combined scope, parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Store service (layerfs-project #65) On PostgreSQL, sandbox.create's host side opened three connections: the reach check's binding read, then `open::hold`'s full Store service (writer and control reader, both fully validated) only so `sandbox_store` could list projects. Every other Sandbox op (mount, exec, commit, unmount, close) also opened that Store service when the process did not hold it, which a CLI process never does (bench: 2 host connections per mount and per exec). - New `store/reader.rs`: the process's one held read session per PostgreSQL Store (keyed by Store path and dialled URL), opened with the light check under the schema's recorded binding (`PostgresHandles::attach_read_only`). A held session is checked with one empty query before use; one that does not answer is dropped and a new one opens. The open-Store reaper closes it after 60 s unused; `open::close` (disconnect/move) forgets it. - The reach check now reads the binding and finds the project on that session (binding check + project lookup + reach in one session). `sandbox_store` no longer lists projects; every route to sandbox.create runs the reach check first. NoSuchProject keeps its code and message but is now reported at the reach check, i.e. before deploy/image/platform checks (only differs when both are wrong). - PostgreSQL Sandbox ops no longer call `open::hold`: mount's Branch id and status's head read on the held session (`lookup::project_in/branch_in`, same refusals); exec, commit, unmount and close open no connection. The caller's Store lease is held for each call; previously it lived with the open Store for up to 60 s after the call. Counts (wire proxy, postgresql_open_cost; TCP handshake excluded): host side of create 3 connections / 49 round trips -> 1 / 8 (emulation of the same persistence calls; the api path adds nothing). Mount host side: 2 connections (bench, CLI) -> 1 (or 0 within 60 s in a long-lived process). Exec/commit/unmount/close host side: 2 connections -> 0. Tests: new layerfs-api postgres_sandbox_reach (NoSuchProject and another binding refused at the reach check, nothing started). PASS on pgperf-pg: postgres_sandbox_reach, postgres_connect, postgres_refusals, sandbox_machines, store_routes, machine_calls. NOT_RUN here: sandbox_family, store_open_cache, store_reaper (launcher /tmp is overlayfs: project.init refuses before any changed code). Clippy clean for layerfs-control, layerfs-api, layerfs-machine with and without postgresql. No tests deleted. Production LOC: 138163 -> 138417 (delta +254); tools/production_loc.py combined scope, parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the measure agent's per-scenario round-trip count test (layerfs-store-service tests/postgresql_round_trips.rs) and its telemetry. Production LOC: 138008 -> 138025 (delta +17) Scope: first-parent comparison (72ebf96 -> merge tree) with tools/production_loc.py over git archive of each; the delta is the merged store-service telemetry. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…reate (layerfs-project #66) `reach()` for an apple-container machine ran `container inspect` (a subprocess polled at 20 ms steps) on every PostgreSQL Sandbox call to turn a loopback URL into the container network's gateway. - `Lasting` gains `gateway: Option<Ipv4Addr>`; apple-container's `bring_up` (create and every start) reads it once and the machine record keeps it (`lasting.gateway`, machine/spec.rs). Every start replaces it. - `reach` uses the recorded gateway with no subprocess. A stale one is dialled as recorded and fails explicitly at the connection; nothing looks it up again. A record without one (started before this change, or a start that could not read it) still reads it from the machine, as before. - Docker/Lima: not changed. Docker dials `host.docker.internal`, resolved per connection by the container's resolver (hosts entry on Linux, Docker Desktop's embedded DNS on macOS); with #63-#65 a create makes 2 connections instead of 5, so it is paid at most twice. Not measured. Not verified on a live Apple Container machine (none runs here): the per-create inspect cost and the re-measured create gap are open for the bench agent. Proven by machine_reach `apple_container_uses_the_gateway_its_start_recorded` (no `container` call with a recorded gateway; lookup kept without one). Tests: PASS layerfs-machine conformance, machine_reach, machine_spec, machine_providers, machine_platform, machine_adapter, machine_lima, machine_probe, machine_ssh, disabled_platforms; layerfs-control typed_records. machine_forward `lima_forwards_through_its_own_ssh_config` FAILED (assert !runs(pid) right after forward_close: the ssh stand-in's exit races the check; this change touches no forward code). macOS host build not run (the macOS cargo home lacks libproc offline). No tests deleted. Production LOC: 138417 -> 138429 (delta +12); tools/production_loc.py combined scope, parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re the lane (layerfs-project #67) - Save: a whole-file object whose ring proposal is still held by this Save (pending or sealed, unpublished) sealed and published that proposal before selection, so 1,000 similar 23-byte files cost 1,000 publications and 1,000 one-object packs, while selection stored every one FULL. When no PREFIX record can be narrower than the object's FULL record (delta::record::prefix_cannot_win: a PREFIX adds the 32-byte base id and a zstd frame of at least PREFIX_FRAME_MIN = 14 bytes), the proposal is now dropped: selection answers FULL whatever the base. The FULL record is prepared once (select::prepare_full) and handed to selection; selection's own ring fallback skips the same futile trials. Records are unchanged (all FULL before and after); packs and stored bytes fall. Count test commit.similar_files_1000: PG 12,241 -> 226 round trips, 2,041 -> 39 transactions; SQLite 11,211 -> 3,213 statements, 1,040 -> 41 transactions. Persistence test, one Save: 1,000 x 23 B 7,006 -> 11 round trips, 1,000 -> 1 pack, 60,000 -> 28,032 stored bytes; 200 x 200 B 1,026 -> 10 round trips, 146 -> 1 pack, 13,272 -> 8,632 bytes. - Commit: the candidate is validated and its install prepared before the publication lane (immutable reads only); only an in-lane merge step that changes the candidate validates again ("validate.lane"). PG lane-held round trips for an unmoved Commit: 22-38 -> about 10. - Tests: codec_frames no_prefix_frame_is_shorter_than_its_declared_minimum; postgresql_commit_round_trips asserts one publication for the similar files and reports 200 B similar files. - Docs: physical-storage/saves.md, engine/commit/publish-and-merge.md, engine/store/history-publication.md. Tests deleted: none. Production LOC: 138025 -> 138094 (delta +69) Scope: tools/production_loc.py over the parent (17013c3) and the staged tree (git archive of each); core/layerfs-storage 9068 -> 9105, engine/layerfs-store-service 10819 -> 10851. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rustfmt +1.85.1 on the files these commits changed; no behaviour change. Production LOC: 138429 -> 138459 (delta +30, formatting only: wrapped lines); tools/production_loc.py combined scope, parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g Commit (layerfs-project #68) - Fetch::published: after an acknowledged publication the locations this Save won are kept in the locator cache (they are the Store's winners) instead of invalidated; lost ones are still invalidated and located again. The Save's metadata pack bodies (the published Arcs, at most 2 MiB) and pooled inode values (at most 512 KiB) are kept until the next Save. - Only Storage::reader_after_save reads them: a reader that continues the latest Save's operation. Every other reader still acquires every body it needs (sqlite_transaction_units' fresh-reader assertion holds unchanged). Everything returned is authenticated as before. - Save::read_objects locates only ids this Save does not hold (pending or sealed); the sealed-location fallback no longer locates eagerly. - Commit: logical.count and bindings share one after-save reader (and its bounded caches); the in-lane step does the same. - Per-directory read-back is gone rather than batched: the bindings and logical-count walks now cost 0-6 round trips in every scenario. Count test, PG round trips (logical.count / bindings / Commit total), from the previous commit: files_1000 28/33/225 -> 2/4/164; c04 110 dirs + 1,000 files 57/147/455 -> 5/5/146; files_100 17/16/93 -> 6/3/64; random 64 MiB 15/15/243 -> 7/3/220; second_one_file 12/8/66 -> 6/0/52. SQLite statements c04 3,060 -> 2,751. Resident: bounded at 2 MiB of bodies (shared Arcs the Save already held) + 512 KiB of values per Storage handle. - Docs: physical-storage/reads.md, engine/commit/publish-and-merge.md. Tests deleted: none. Production LOC: 138094 -> 138263 (delta +169) Scope: tools/production_loc.py over the parent (3868d1a) and the staged tree (git archive of each); core/layerfs-storage 9105 -> 9274. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-project #69 work-around) file_roots (one per 64-row construction page) ran as a REPEATABLE READ transaction: BEGIN, the digest_roots array statement, COMMIT. It is one statement, so on PostgreSQL it now runs alone in Mode::Statement (HistoryProvider::read_statement): digest_roots.sql names the schema ({s}), and array statements replace {s} with the Store's schema. 3 -> 1 round trip per page; SQLite unchanged (it runs the same body in its read transaction). The FILE_DIGEST_WINDOW contract constant (64) is unchanged; widening it is an owner decision (#69). Count test, PG construct round trips: files_1000 111 -> 79, similar files 107 -> 75, c04 95 -> 59, files_100 12 -> 8 (Commit totals 164 -> 132, 157 -> 125, 146 -> 110, 64 -> 60). Docs: engine/store/postgresql.md (Statement mode users). Tests deleted: none. Production LOC: 138263 -> 138263 (delta 0) Scope: tools/production_loc.py over the parent (317896b) and the staged tree (git archive of each); the change replaces lines in place (SQL text and the provider call). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-project #70) PostgreSQL pays a round trip per statement, so committed reads now take fewer, larger statements: - Port hint `PackPersistence::whole_pack_bytes` (default: the control prefix, so SQLite is unchanged). PostgreSQL returns the 256 KiB pack limit: GroupPlan takes such packs whole, and the first statement of a PostgreSQL acquisition now fetches 256 KiB of body with the descriptor, so a pack costs one statement and stays in the kept pack cache. - A demand reads its uncached whole-taken packs in one `read_packs` call when there are at least two (Fetch::whole_packs), within the 2 MiB pack cache, reserving the demand's kept packs first. - PostgreSQL `read_packs` is one statement: the server returns bodies only when every id is present and the total is within the read budget. Descriptors no longer copy the body column (rows::record_first). - Sequential base read-ahead: a base read of chunked content that continues the client's previous read reads 2 MiB further in the same demand; the extra objects stay in the bounded canonical cache. One slot per client, off for caches under 8 MiB and for memory-only clients. - SourceView::lookup remembers sibling file lengths (F-032) like Mutation planning already did, so a stat walk is one length demand per 128 files. Count test (postgresql_round_trips, cold reader, LAYERFS_CONSTRUCTION_WORKERS=1): read.sequential_random_64mib PG 1616 -> 302 round trips, SQLite 1138 -> 505 read.stat_files_1000 PG 1034 -> 47, SQLite 1034 -> 47 Commit counts unchanged. Tests: postgresql_commit_round_trips asserts read_packs is one round trip, found or missing. No tests deleted. Production LOC: 138263 -> 138406 (delta +143) Scope: tools/production_loc.py combined (api, connect, core, engine; reference absent (retired R9c; last counted 65,417 at tree 498dd19)), parent tree vs staged tree via git archive. Subtotals: layerfs-storage 9274 -> 9327, layerfs-persistence 12516 -> 12515, layerfs-workspace 8991 -> 9082. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (layerfs-project #62) The traced-call telemetry test asserted 27 SqlWork fields; #62 added round_trips, prepares and connects (SQL_FIELDS = 30). Expect 30 and check the three new fields are present and zero on this SQLite fixture. Found by running the mount lane (each binary in its own container with /dev/fuse and SYS_ADMIN, as nightly does) on the merged pg-perf tree: 43/44 binaries passed, this one failed on the width only. Tests deleted: none. Production LOC: 138992 -> 138992 (delta 0); tools/production_loc.py over git archive of HEAD and of the staged tree. Test-only change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ount test (layerfs-project #69) Each Commit scenario now prints `STORAGE <json>` (new objects, packs, canonical, stored and metadata bytes, record kinds and the process's VmHWM so far), and the PostgreSQL run prints the Store's running totals and each table's pg_total_relation_size/heap/TOAST before its schema is dropped. Test-only; no tests deleted. Production LOC: 138992 -> 138992 (delta 0) Scope: tools/production_loc.py combined (api, connect, core, engine; reference absent (retired R9c; last counted 65,417 at tree 498dd19)), parent tree vs staged tree via git archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…page to 1,024 (layerfs-project #69) Owner decision (2026-11-06): widen FILE_DIGEST_WINDOW and the overlay page PAGE_ROWS from 64 to 1,024. - History contract FILE_DIGEST_WINDOW and the workspace port's window are 1,024: construction asks `file_roots` once and records digests once per captured inode page instead of per 64 rows. - Overlay PAGE_ROWS is 1,024. Every paged statement whose page is PAGE_ROWS (inode and name capture, source names) takes its LIMIT from page_limit!, asserted equal to PAGE_ROWS at compile time. - Operation-record windows keep 64 under a new RECORD_ROWS: they are the key window of core content's construction backing (filesystem/state/store.rs, 64 keys in one 64 KiB envelope), so get_many, key/entry pages, the construction writer's changes per job and the cursor's record windows use RECORD_ROWS (record_limit!). Widening them is a core content contract change and is not part of this decision. - get_many keys are a Vec instead of a PAGE_ROWS-sized stack array. - Digest records keep only the last root per content before an engine without an array form inserts them, and `file_roots` answers a repeated content once: with one 1,024-row page, 1,000 empty files are no longer 1,000 SQLite inserts and lookups (same final rows as one at a time). Count test (LAYERFS_CONSTRUCTION_WORKERS=1), PostgreSQL round trips / SQLite statements, before -> after: files_1000 132 -> 72 / 3139 -> 3079 (construct 79 -> 19) similar_1000 125 -> 65 / 3144 -> 3084 C04 110 -> 73 / 2751 -> 1196 (construct 59 -> 20) files_100 60 -> 61 / 267 -> 267 64 MiB 218 -> 218 / 527 -> 525 Storage: unchanged except C04 +121 stored bytes (one more metadata pack). Tests adapted, none deleted: fixtures that asserted 64-row windows now size themselves from PAGE_ROWS / RECORD_ROWS with unchanged assertions (overlay captured_namespace, indexed_operation_record, ..._cursor; workspace captured_namespace_cursor, captured_namespace_order, kernel_directory, namespace_view). The cursor growth test compares two sizes above one record window (a 1,024-row page is as wide as the cursor's 1,024 point answers). Production LOC: 138992 -> 139029 (delta +37) Scope: tools/production_loc.py combined (api, connect, core, engine; reference absent (retired R9c; last counted 65,417 at tree 498dd19)), parent tree vs staged tree via git archive. Subtotals: layerfs-persistence 12714 -> 12719, layerfs-workspace 9082 -> 9073. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…L (layerfs-project #61) Owner decision (2026-11-06): PostgreSQL publication waves up to 16 MiB; SQLite keeps 4 MiB. - New port hint PackPersistence::publication_bytes (default the shared 4 MiB - 1 transaction bound). Storage applies it to the wave and transaction byte capacities (StorageCapacities::with_publication_bytes, which refuses anything outside 4 MiB - 1 ..= PUBLICATION_BYTES_MAX, 16 MiB - 1). - PostgreSQL returns PUBLICATION_BYTES_MAX; persistence validates each publication against its own provider's size. SQLite, its unit layout and headroom checks are unchanged. - Rows per publication stay 8,191. Count test (LAYERFS_CONSTRUCTION_WORKERS=1): commit.random_64mib PostgreSQL 218 -> 115 round trips (construct 160 -> 58); every other scenario and all SQLite counts unchanged; stored bytes unchanged (64 MiB: 67,302,459 both). Persistence count test: save 8 MiB 31 -> 16 round trips. Peak RSS of the count-test process after the 64 MiB Commit 101.5 -> 115.9 MB (VmHWM, a lifetime peak including the test's own 64 MiB buffer). No tests deleted. Production LOC: 139029 -> 139048 (delta +19) Scope: tools/production_loc.py combined (api, connect, core, engine; reference absent (retired R9c; last counted 65,417 at tree 498dd19)), parent tree vs staged tree via git archive. Subtotals: layerfs-storage 9327 -> 9345, layerfs-persistence 12719 -> 12720. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts PostgreSQL Store round trips on the open, mount, Commit and read paths, for Ephemeral-AI-Lab/layerfs-project#61 (sub-issues #62–#70). Measured in Ephemeral-AI-Lab/layerfs-project#71.
What changes
SqlWorkgainsround_trips,preparesandconnects. A store-service count test measures each scenario.branch_snapshotis one statement with no BEGIN/COMMIT, andreserveis oneUPDATE … RETURNING. Publish is one transaction (COPY → lane row → one CTE), and similar small files share one wave. Validation runs before the lane.FILE_DIGEST_WINDOWand overlayPAGE_ROWSare 1,024. Operation-record windows stay 64 (RECORD_ROWS, a core content contract).read_packsstatement. Sequential reads read 2 MiB ahead, and lookups remember sibling file lengths.Measured
Count test, PostgreSQL round trips before → after:
fs-bench-quick (Ephemeral-AI-Lab/layerfs-project#71): 162/162 PASS. The median PostgreSQL − SQLite gap per case:
A bisect across our builds found no SQLite regression. The report is on layerfs-benchmark branch
pg-perf-62.Checks
Not in this PR
Production LOC: 137772 -> 139048 (delta +1276),
tools/production_loc.py, baseorigin/layerfs_v2_0_0(6d820c756b) to headddd1bc83cd. Each commit records its own comparison. Reference absent (retired R9c; last counted 65,417 at tree 498dd1917812).🤖 Generated with Claude Code