Repository navigation
GC: five refactors (-485 LOC) and store.compact progress events - #735
Merged
Merged
Conversation
The eight open-time migrations in backend/sqlite/schema.rs each repeated the same skeleton: return when already applied, read the recorded schema, compare it with an earlier source and with the live definition, apply, record the new source and definition. `Session::migrate` now owns that skeleton; each migration states only its present check, the sources it accepts and records, and what it applies. No behaviour change: the same checks run in the same order inside the same single write transaction, and the sources are still computed only after the present check. `binding_hint_table()` joins the sibling text helpers. It panics if the compiled-in history schema lacks the table, where the old code returned Integrity; the text is a compile-time constant. Checks (macOS host, Rust 1.85.1): cargo check and fmt; persistence tests old_format_columns, old_format_open, gc_backfill, lineage_backfill, binding_hint_blocks, kept_node, diagnostic_schema, store_diagnostic and path_removal pass (46 tests). No test changed or deleted. Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above are from before the rebase. This commit compiles with its tests on the new base; the suites were rerun at the branch tip, not at this commit. Production LOC: 188,972 -> 188,837 (delta -135) core 40,589 -> 40,589 (0) engine 90,631 -> 90,496 (-135) layerfs-persistence -135 control 51,230 -> 51,230 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --json on parent 4c296e9 and the committed tree (recomputed after the rebase onto 4c296e9) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The four open-time migrations in backend/postgresql/schema.rs (kept_node, the GC tables, next_history_serial, the position columns) repeated one skeleton: probe without a transaction, probe again under the lane, read the recorded schema, decide whether it is an earlier revision without or with the acquisition tables, compare the live definition, apply, record. `migrate` now owns it; each migration states its probe, its earlier source and what it applies. One ordering difference, with no visible effect: `add_gc` compared the table set before the live definition and now compares it after. Both are reads that refuse with the same Integrity error before anything is written. Checks (macOS host, Rust 1.85.1, PostgreSQL 17 in a private container with amcheck): Clippy -D warnings on layerfs-persistence, fmt; persistence tests postgresql_position_columns, postgresql_open_check, postgresql_statement_schema, postgresql_gc_mark, postgresql_diagnostic and postgresql_compact pass (52 tests, none NOT_RUN). No test changed or deleted. Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above are from before the rebase. This commit compiles with its tests on the new base; the suites were rerun at the branch tip, not at this commit. Production LOC: 188,837 -> 188,739 (delta -98) core 40,589 -> 40,589 (0) engine 90,496 -> 90,398 (-98) layerfs-persistence -98 control 51,230 -> 51,230 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --json on parent 7ceff82 and the committed tree (recomputed after the rebase onto 4c296e9) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- `RunMetrics`, the phase clock and §11 values of one `store.compact` run, existed twice: backend/sqlite/compact/receipt.rs and backend/postgresql/compact/op.rs differed only in `mark_bytes`. One definition now lives in compact_metrics.rs beside the recorder it feeds; `mark_bytes` is `None` on SQLite and `Some` on PostgreSQL, as each reported before. - `CompactReport`, `CompactGeneration` and the one-implementor trait `CompactReportSource` restated `CompactReceipt` field for field. Both backends already return `CompactReceipt`, so the control serializer `layerfs_store::report::compact_report` now takes it directly. The wire shape is unchanged. Tests: sqlite_compact.rs `the_report_projects_every_section_11_field` is now `the_receipt_carries_every_section_11_field`; its comparisons of the removed projection against the receipt (ten field equalities and the trait/From equality) are gone with the type, and its assertions on the values stay. compact_report_status.rs passes the receipt to the serializer. No test file deleted. Checks (macOS host, Rust 1.85.1, PostgreSQL 17 in a private container): cargo check --workspace --all-targets; Clippy -D warnings on layerfs-persistence and layerfs-store; fmt; product boundary guard; persistence sqlite_compact (33), sqlite_compact_crash (15), compact_metrics (5), postgresql_compact (30), postgresql_status (8), postgresql_compact_units (8); layerfs-store whole package (80). sqlite_compact_capacity ran 0 tests on this host. Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above are from before the rebase. This commit compiles with its tests on the new base; the suites were rerun at the branch tip, not at this commit. Production LOC: 188,739 -> 188,574 (delta -165) core 40,589 -> 40,589 (0) engine 90,398 -> 90,234 (-164) layerfs-persistence -164 control 51,230 -> 51,229 (-1) layerfs-store -1 api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --json on parent 4f136fd and the committed tree (recomputed after the rebase onto 4c296e9) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…alled bind - history/lookup.rs built eight optional text fields of the MountsActive, Unknown and Busy details with `map_or(SdkValue::Null, SdkValue::text)` chains. They now use the existing `SdkValue::opt_text`. Same values. - store/bind.rs: `bind_private_placed` had no caller in engine, control, api or any test; removed. Checks (macOS host, Rust 1.85.1): cargo check --workspace --all-targets; Clippy -D warnings on layerfs-store-service; fmt; store-service tests history_wire_codes, admission, branch_remove, project_delete, epoch_fence and method_refusal_order (23 tests); layerfs-store tests project_delete, project_delete_store and compact_in_process_call (23 tests). No test changed or deleted. Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above are from before the rebase. This commit compiles with its tests on the new base; the suites were rerun at the branch tip, not at this commit. Production LOC: 188,574 -> 188,529 (delta -45) core 40,589 -> 40,589 (0) engine 90,234 -> 90,189 (-45) layerfs-store-service -45 control 51,229 -> 51,229 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --json on parent 52a0697 and the committed tree (recomputed after the rebase onto 4c296e9) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`remove_branch`, `delete_stack` and `remove_path` each wrote the same admission block before their marker: on PostgreSQL the Store key shared, a scan for a live conflicting intent and the scope's own key; on both backends the project epoch hold and the scan that settles dead owners' intents. `admit` now holds that block once. Each removal passes its scan statement and parameters, its exclusive key, and whether it also holds the project key shared (a Branch removal does). Lock order, statements and refusals are unchanged. Checks (macOS host, Rust 1.85.1, PostgreSQL 17 in a private container): Clippy -D warnings on layerfs-persistence (all targets), fmt; persistence tests admission, branch_removal, layerstack_removal, path_removal, project_delete, epoch_fence, visibility_guard, semantic_admission (32 tests), postgresql_gc_fence (19) and postgresql_compact (30). No test changed or deleted. Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above are from before the rebase. This commit compiles with its tests on the new base; the suites were rerun at the branch tip, not at this commit. Production LOC: 188,529 -> 188,487 (delta -42) core 40,589 -> 40,589 (0) engine 90,189 -> 90,147 (-42) layerfs-persistence -42 control 51,229 -> 51,229 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --json on parent 64c76cc and the committed tree (recomputed after the rebase onto 4c296e9) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…r phase (#42 A13) Owner direction 2026-11-18: the caller of a compaction should see its stages as they happen, through the same on_progress events store.push and backup.create already send, so a client's job display needs no polling. What changes - API: store.compact declares the shared PROGRESS option, the twelfth call to do so. Node, Python and the CLI (--progress) get it from the table; index.d.ts, _client.pyi and aio.pyi are regenerated. - Control: compact() uses the event stream it already received and sends { phase, done, total } with null counts as each phase starts. - Engine: RunMetrics, the phase clock both backends share, holds an observer and tells it a phase once, when the phase changes. CompactPhase::step gives the name store.status uses. SqliteHandles and PostgresHandles gain compact_observed; compact() calls it with no observer, so every existing caller is unchanged. - Phases: mark, build, verify, install on SQLite; mark, build, swap, cleanup on PostgreSQL. A none run sends mark alone. The SQLite settle (cleanup in store.status, D4) is inside install and sends no event. What does not change - The op record and store.status: still the only record another process, or a call after a crash, reads. Events are in memory and write nothing. - No byte counts inside a phase, no estimate before a run, no cancel. Spec: the frozen GC spec says there is no PROGRESS option (#42 item 40). The spec file stays byte-identical; the change is recorded as amendment A13 in core/docs/issues/project-41/gc-42-decisions.md for sign-off. Tests - New: an_observed_run_is_told_each_phase_once_in_order (sqlite_compact, postgresql_compact) and store_compact_sends_each_phase_as_a_progress_event (gc_calls, through layerfs_api::call: none then rebuild). - Changed on purpose, because the contract changed: the two "four GC calls take ids only" tests (gc_calls, calls_table) asserted no options at all; they now assert that the three removals have none and that store.compact has exactly the on_progress function. The progress lists (progress_options, node progress.mjs, python test_progress, CLI progress) name twelve calls. No test deleted. - Ran, each under 120 s, macOS host, after the rebase onto 4c296e9 (layerfs#734): all 89 layerfs-persistence test files (sqlite_compact 37); with PostgreSQL 17 in Docker on loopback, postgresql_compact (32), postgresql_gc_fence (19) and nine more PostgreSQL files; layerfs-api gc_calls (9), calls_table (22), progress_options (2), gc_trace (5), store_family (5); layerfs-cli gc (5), progress (5); layerfs-store compact_report_status (6), compact_in_process_call (2); nine layerfs-store-service removal and refusal files; Clippy -D warnings on the five changed crates. gen-node-types --check ran before the rebase only. - The T12.3a matrix report is regenerated: it records the test counts of sqlite_compact and postgresql_compact, which each gained one. - Not run: the Node and Python end-to-end progress tests and gen-coverage --check (they need the Python module and a Linux Store host; the coverage row is edited to what the generator writes), any Linux build, and the macOS worker route for these events. Docs: errors-and-events.md (call list, shape and phase tables), the API README count, sdk/store.md, coverage.md and the JS SDK skill reference. Production LOC: 188,487 -> 188,532 (delta +45) core 40,589 -> 40,589 (0) engine 90,147 -> 90,180 (+33) layerfs-persistence +33 control 51,229 -> 51,241 (+12) layerfs-store +12 api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --json on parent b8ff1af and the committed tree (recomputed after the rebase onto 4c296e9) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six commits on top of
layerfs_v2_0_0at4c296e9297(after #734). Five are refactors of the GC code with no behaviour change; the sixth adds progress events tostore.compact.What changes
7ceff82a5f4f136fdab852a0697cdd64c76ccb34opt_textin refusal details; an uncalled bind removedb8ff1afdfef2e27d7bdfstore.compacttakeson_progressand sends one event per phaseProduction LOC: 188,972 -> 188,532 (delta -440)across the branch, counted withtools/production_loc.py(core, engine, control, api; tests, docs and tools excluded). Each commit message carries its own comparison against its first parent.store.compactprogress (Ephemeral-AI-Lab/layerfs-project#42, amendment A13)store.compactnow acceptson_progress, likestore.push. It sends one{ progress: { phase, done, total } }event when each phase starts;doneandtotalare null.mark,build,verify,installmark,build,swap,cleanupmarkonly.The recorded stage in
store.statusstays the source of truth; the events mirror it. SQLite'scleanup(the settle step, decision D4) shows instore.statusbut is not sent as an event.How it is wired: the shared phase clock in
compact_metrics.rscalls an observer when a phase begins; both backends gaincompact_observed(request, on_phase)andcompact(request)calls it with a no-op;layerfs-storeforwards each phase to the call's event sink.Needs owner sign-off: A13 is a new amendment in
core/docs/issues/project-41/gc-42-decisions.md. The frozen spec file is unchanged.Two test assertions changed on purpose.
gc_calls.rsandcalls_table.rsasserted that the four GC calls take no options. They now assert thatstore.compacttakes exactlyon_progressand the other three take none.progress_options.rscounts twelve progress calls instead of eleven.Generated typings (
api/node/src/index.d.ts, the two Python.pyifiles) and the API docs are updated. Thesqlite_compactmatrix report is regenerated.Checks run (macOS host, cargo 1.85.1,
--locked --offline)layerfs-persistencetest files pass, includingsqlite_compact(37).postgres:17container, includingpostgresql_compact(32) andpostgresql_gc_fence(19).cargo fmt --checkare clean; each of the six commits compiles on this base.markthenbuild, and finished at 405 MB.Not run
api/node/test/progress.mjs,api/python/test/test_progress.py) were edited but not run.gen-coverage.mjs --checkwas not run (it needs the Python native module); thestore.compactrow incoverage.mdwas edited by hand. The Python stubs were rendered from the Node addon's call table.The matching UI change is Ephemeral-AI-Lab/dsh-layerfs branch
claude/store-compact.🤖 Generated with Claude Code