Repository navigation
SQLite: checkpoint and truncate the WAL when a Store goes idle (#86) - #583
Merged
Merged
Conversation
When the last call on an open SQLite Store returns, or a Sandbox on it closes, with no other call running in this process, the open-Store cache runs one PRAGMA wal_checkpoint(TRUNCATE) on the Store's live session. The Store stays open (the live-daemon contract and the 60 s IDLE close are unchanged), so a quiesced Store's -wal is 0 bytes and its file equals the closed bytes. - After a Store call the checkpoint runs on its own thread once the call is answered (off the reply path); after sandbox.close it runs before the close is answered, so `store status` right after sees the truncated log. - One attempt, never retried: an empty log needs no SQL; a held lane on this thread, another process holding the Linux writer lane (tried without waiting), or a reader still in the log (busy=1) leaves it for the idle close. - The checkpoint holds the session's FIFO turn and connection lock for the one statement; the entry's `settling` count keeps the reaper and `close` from dropping the service under it, and `close` waits for it to end. - PostgreSQL Stores are untouched (no Store file). Hook files outside #86's owned set (approved by the #84 lead): - engine/crates/layerfs-persistence/src/backend/sqlite/lane/writer_admission.rs (Writer::try_acquire: one non-blocking flock attempt) - engine/crates/layerfs-persistence/src/store/selection.rs (StoreFile::checkpoint) - api/crates/layerfs-control/src/sandbox/route.rs (open::released after a successful sandbox.close) New tests: layerfs-persistence tests/idle_checkpoint.rs (truncate + file equals closed copy, reader busy left once, Linux lane busy left without waiting, racing calls and a new opener lose nothing); layerfs-api tests/store_idle_checkpoint.rs. No tests deleted. Production LOC: 139048 -> 139212 (delta +164); tools/production_loc.py --json on the parent tree (git archive HEAD) and the staged tree (git archive of git write-tree); scopes api 20764 -> 20839 (layerfs-control 18122 -> 18197), engine 63539 -> 63628 (layerfs-persistence 12720 -> 12809), core/connect unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mengmengmengqiang
pushed a commit
to tokenaissance/layerfs
that referenced
this pull request
Nov 6, 2026
…ps the Store (Ephemeral-AI-Lab#86) Regression from Ephemeral-AI-Lab#583 on layerfs_v2_0_0 3384cbb: the one-shot `layerfs` CLI ends every call with `close_idle(ZERO)` so the Stores it opened close and SQLite folds the write-ahead log in. Ephemeral-AI-Lab#583 started an idle checkpoint on its own thread after each Store call and made `close_idle` skip an entry whose checkpoint was still running (`settling > 0`). The CLI's close then closed nothing, the process exited with the Store open, and init + fork left store.sqlite-wal (12,392 B), so `store push` refused the Store. `close_idle` now waits (Condvar, bounded by the one checkpoint statement) for a running checkpoint on any Store it is about to close, then closes it, as `close` already did. No retry: the checkpoint is still one attempt, and the close is the ordinary last-connection close. Regression test: layerfs-api tests/store_oneshot_close.rs (store.create, project.init, branch.fork, then the CLI's close_idle(ZERO): it must close the Store and leave no -wal). It fails on 3384cbb (close_idle closes 0 Stores) and passes with this change. No tests deleted. Production LOC: 139400 -> 139405 (delta +5); tools/production_loc.py --json on the parent tree (git archive HEAD) and the staged tree (git archive of git write-tree); api 20839 -> 20844 (layerfs-control 18197 -> 18202), engine/core/connect unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A quiesced SQLite Store no longer carries an uncheckpointed WAL. When the last call or Sandbox in a process releases a Store, one
wal_checkpoint(TRUNCATE)runs on its open connection. On fs-storage-quick, every SQLite row now haswal_bytes0 at quiesce, and its file equals the closed bytes after the 60 s idle close.Fixes Ephemeral-AI-Lab/layerfs-project#86 (parent Ephemeral-AI-Lab/layerfs-project#84). Base:
layerfs_v2_0_0@b4fa173e40.Change
store/open.rsruns onePRAGMA wal_checkpoint(TRUNCATE):sandbox.close, before the reply, so the nextstore statussees the truncated log.settlingcount stops the reaper orclosefrom dropping the service mid-checkpoint.Session::checkpoint_idle(newbackend/sqlite/session/checkpoint.rs) holds the FIFO turn and the connection mutex for that one statement. It makes one attempt and never retries or waits on a busy handler. If it can't complete, it returns Ok(false) and leaves the log for the idle close:busy=1).Hooks outside the core files:
lane/writer_admission.rs:try_acquire, one non-blocking attempt.store/selection.rs:StoreFile::checkpoint.sandbox/route.rs: callsopen::releasedaftersandbox.close.Production LOC: 139048 -> 139212 (delta +164).
Tests
persistence/tests/idle_checkpoint.rs: macOS 3/3, Linux 4/4.wal_bytesis 0 and the file is byte-equal to the closed copy.api/tests/store_idle_checkpoint.rs: passes.store_reaper,writer_admission(11/11 on Linux), and persistence and store-service (185, plus every store-service binary run on its own) all pass.fs-storage-quick (campaign q86-c1, Docker, late probe 75 s, against the pinned base baseline)
sandbox closewas about +0.16 s per close in the candidate samples, with heavy noise from shared Docker. Store calls are not delayed, because their checkpoint runs after the reply.No format or codec change.
🤖 Generated with Claude Code