Skip to content

PostgreSQL Init: empty the acquisition tables when the last operation is released (#87) - #582

Merged
yifanxuaaa merged 1 commit into
layerfs_v2_0_0from
opt/87-pg-init-cleanup
Nov 6, 2026
Merged

yifanxuaaa merged 1 commit into
layerfs_v2_0_0from
opt/87-pg-init-cleanup

Conversation

@yifanxuaaa

Copy link
Copy Markdown
Contributor

PostgreSQL Store size after project init no longer depends on autovacuum. The transaction that releases a session's last Init operation now also empties the Init working tables. On fs-storage-quick Q6 PostgreSQL, the Store is the same size immediately and after autovacuum: 28,917,760 B, against 38,158,336 B / 31,358,976 B on base.

Fixes Ephemeral-AI-Lab/layerfs-project#87 (parent Ephemeral-AI-Lab/layerfs-project#84). Base: layerfs_v2_0_0 @ b4fa173e40.

Change

engine/crates/layerfs-persistence/src/backend/postgresql/acquisition/cleanup.rs:

  • RELEASE is now a CTE. It deletes the record and reports whether any other record remains.
  • If none remains, the same transaction runs one batch: SAVEPOINT, SET LOCAL lock_timeout 2000, LOCK … ACCESS EXCLUSIVE, TRUNCATE init_operation, init_entry, init_host_file CONTINUE IDENTITY. The record table is locked first, in the same order units take the tables.
  • A definite refusal (a lock wait or a privilege error) rolls back to the savepoint, and the release commits as before.
  • A lost reply stays Unknown, so the session is quarantined. Nothing is retried.

Why it is safe:

  • Every acquisition write, including begin, holds the store_lane row, so no operation can start until the truncate commits.
  • The emptiness check runs after the lane is held, as a READ COMMITTED statement, so it sees every committed record.
  • CONTINUE IDENTITY means operation ids and epochs are never reused.
  • The truncate is the last statement of its transaction, so the SET LOCAL timeout reaches nothing else.

Not in this PR: the issue's second item, writing file_root at insert. It needs a change to acquisition.sql, the recorded schema, so it is a format change; it moves to #90's schema-v2 work. With the truncate, the UPDATE's dead tuples last only while an Init runs.

Production LOC: 139048 -> 139062 (delta +14).

Tests

  • New tests/postgresql_acquisition_compaction.rs: 4/4 pass, and all 4 fail on the base cleanup.rs. It covers:
    • bytes after Init equal a fresh Store's, per table;
    • a VACUUM afterwards changes nothing;
    • back-to-back Inits leave nothing behind;
    • a live operation in another session keeps its rows;
    • a reader holding a table lock doesn't block the release;
    • 3 sessions × 12 concurrent Inits.
  • Existing suites pass:
    • the PostgreSQL acquisition suite, postgresql_open_check, postgresql_identical_ids, postgresql_commit_round_trips, diagnostics_during_lane;
    • the SQLite acquisition suites;
    • project init_postgresql;
    • store-service postgresql_open and postgresql_open_cost.
  • clippy -D warnings and rustfmt are clean.

fs-storage-quick (campaign q87-c1, Docker, both backends, late probe 75 s, against the pinned base baseline)

Row Metric Base (B) This PR (B) Δ
Q6 PostgreSQL net, immediate 38,158,336 28,917,760 −9,240,576 (−24.2%)
Q6 PostgreSQL net, late probe 31,358,976 28,917,760 −2,441,216 (−7.8%)
  • Other rows: no regressions; every other row and metric is within noise. Payload, canonical and raw bytes match exactly.
  • Final-state verification: passes on every row.
  • DIAGNOSTIC attribution: after the run the init_* tables are 24,576 / 24,576 / 16,384 B with 0 dead tuples, the same as a fresh Store. Base still held 950,272 + 1,474,560 B there.
  • Timing: wall times are INELIGIBLE (Docker was shared with other builds).

No format or codec change.

🤖 Generated with Claude Code

… is released (#87)

Deleted Init working rows stayed in init_entry/init_host_file as dead
tuples until autovacuum, and their B-tree pages were never returned, so a
PostgreSQL Store was several MB larger right after Init than a minute later
and kept the emptied indexes for good.

The transaction that releases the last operation (release, or the dispose
job that removes the last row) now learns from the RELEASE statement
whether any other record remains and, if none does, empties the three
acquisition tables with one TRUNCATE ... CONTINUE IDENTITY inside a
savepoint. It runs under the lane row every acquisition write holds, so no
operation can begin until it commits; locks are taken record first (the
order units read them) with a 2 s lock_timeout, past the default
deadlock_timeout that cancels an autovacuum holding a table. A definite
refusal rolls back to the savepoint and the release commits unchanged; a
lost reply stays uncertain. Nothing is retried.

The file_root UPDATE (writes.rs FILE_ROOTS) is unchanged: removing it
needs an acquisition schema change (reported to the lead).

Tests: postgresql_acquisition_compaction (Store bytes right after Init
equal a never-used Store's and are unchanged by VACUUM; another session's
live operation keeps its rows; a held table keeps the release; three
concurrent sessions x 12 Inits).

Production LOC: 139048 -> 139062 (delta +14). Scope/method: combined
production scope of `python3 tools/production_loc.py --root <tree>` on
`git archive` of the parent (HEAD) and of the staged tree (`git write-tree`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit 9597783 into layerfs_v2_0_0 Nov 6, 2026
2 of 4 checks passed
mengmengmengqiang pushed a commit to tokenaissance/layerfs that referenced this pull request Nov 9, 2026
Brings in Ephemeral-AI-Lab#582-Ephemeral-AI-Lab#593 (SQLite idle WAL checkpoint, ring admission, chain-root rebase, PG Init cleanup). One conflict: api/crates/layerfs-control/src/store/open.rs, both sides only added functions (#64 holds; Ephemeral-AI-Lab#86 idle-checkpoint helpers); kept both.

Production LOC: 139821 -> 140178 (delta +357), first-parent comparison (pg-r2 76a32a9 -> merged tree), counted with tools/production_loc.py; the delta is the upstream changes.

Checked on the merged tree: Clippy -D warnings (persistence, store-service, daemon, control, storage, content, workspace, history, bridge with postgresql/machine) and rustfmt clean; 219 test binaries of storage, content, workspace, persistence and store-service run one by one against PostgreSQL 17: 218 pass (1,020 tests), filesystem_attributes fails 1/13 only because its fixture path is relative to the crate directory (known, same at base).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa deleted the opt/87-pg-init-cleanup branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant