Skip to content

GC: five refactors (-485 LOC) and store.compact progress events - #735

Merged
yifanxuaaa merged 6 commits into
layerfs_v2_0_0from
claude/gc-simplify
Nov 18, 2026
Merged

yifanxuaaa merged 6 commits into
layerfs_v2_0_0from
claude/gc-simplify

Conversation

@yifanxuaaa

Copy link
Copy Markdown
Contributor

Six commits on top of layerfs_v2_0_0 at 4c296e9297 (after #734). Five are refactors of the GC code with no behaviour change; the sixth adds progress events to store.compact.

What changes

Commit Change Production LOC
7ceff82a5f One helper for the SQLite additive migrations −135
4f136fdab8 One helper for the PostgreSQL additive migrations −98
52a0697cdd One phase clock and one report type for both compaction backends −165
64c76ccb34 Store service: opt_text in refusal details; an uncalled bind removed −45
b8ff1afdfe One admission function for the three removals −42
f2e27d7bdf store.compact takes on_progress and sends one event per phase +45

Production LOC: 188,972 -> 188,532 (delta -440) across the branch, counted with tools/production_loc.py (core, engine, control, api; tests, docs and tools excluded). Each commit message carries its own comparison against its first parent.

store.compact progress (Ephemeral-AI-Lab/layerfs-project#42, amendment A13)

store.compact now accepts on_progress, like store.push. It sends one { progress: { phase, done, total } } event when each phase starts; done and total are null.

  • SQLite: mark, build, verify, install
  • PostgreSQL: mark, build, swap, cleanup
  • A run that finds nothing to reclaim sends mark only.

The recorded stage in store.status stays the source of truth; the events mirror it. SQLite's cleanup (the settle step, decision D4) shows in store.status but is not sent as an event.

How it is wired: the shared phase clock in compact_metrics.rs calls an observer when a phase begins; both backends gain compact_observed(request, on_phase) and compact(request) calls it with a no-op; layerfs-store forwards each phase to the call's event sink.

Needs owner sign-off: A13 is a new amendment in core/docs/issues/project-41/gc-42-decisions.md. The frozen spec file is unchanged.

Two test assertions changed on purpose. gc_calls.rs and calls_table.rs asserted that the four GC calls take no options. They now assert that store.compact takes exactly on_progress and the other three take none. progress_options.rs counts twelve progress calls instead of eleven.

Generated typings (api/node/src/index.d.ts, the two Python .pyi files) and the API docs are updated. The sqlite_compact matrix report is regenerated.

Checks run (macOS host, cargo 1.85.1, --locked --offline)

  • All 89 layerfs-persistence test files pass, including sqlite_compact (37).
  • 12 PostgreSQL test files pass against a private postgres:17 container, including postgresql_compact (32) and postgresql_gc_fence (19).
  • SDK, CLI and control compaction tests and nine store-service test files pass.
  • Clippy and cargo fmt --check are clean; each of the six commits compiles on this base.
  • End to end through the dsh-layerfs plugin on a private side server: a 526 MB SQLite Store with one deleted project reported mark then build, and finished at 405 MB.

Not run

  • No Linux build or Linux test run.
  • The Node and Python end-to-end progress tests (api/node/test/progress.mjs, api/python/test/test_progress.py) were edited but not run.
  • gen-coverage.mjs --check was not run (it needs the Python native module); the store.compact row in coverage.md was edited by hand. The Python stubs were rendered from the Node addon's call table.
  • The macOS worker route was not traced for these events.
  • Byte counts within a phase are not built; events carry the phase only.

The matching UI change is Ephemeral-AI-Lab/dsh-layerfs branch claude/store-compact.

🤖 Generated with Claude Code

yifanxuaaa and others added 6 commits November 18, 2026 02:57
The eight open-time migrations in backend/sqlite/schema.rs each repeated the
same skeleton: return when already applied, read the recorded schema, compare
it with an earlier source and with the live definition, apply, record the new
source and definition. `Session::migrate` now owns that skeleton; each
migration states only its present check, the sources it accepts and records,
and what it applies. No behaviour change: the same checks run in the same
order inside the same single write transaction, and the sources are still
computed only after the present check.

`binding_hint_table()` joins the sibling text helpers. It panics if the
compiled-in history schema lacks the table, where the old code returned
Integrity; the text is a compile-time constant.

Checks (macOS host, Rust 1.85.1): cargo check and fmt; persistence tests
old_format_columns, old_format_open, gc_backfill, lineage_backfill,
binding_hint_blocks, kept_node, diagnostic_schema, store_diagnostic and
path_removal pass (46 tests). No test changed or deleted.

Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above
are from before the rebase. This commit compiles with its tests on the new
base; the suites were rerun at the branch tip, not at this commit.

Production LOC: 188,972 -> 188,837 (delta -135)
  core     40,589 ->  40,589 (0)
  engine   90,631 ->  90,496 (-135)  layerfs-persistence -135
  control  51,230 ->  51,230 (0)
  api       6,522 ->   6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --json on parent 4c296e9 and the committed tree (recomputed after the rebase onto 4c296e9)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The four open-time migrations in backend/postgresql/schema.rs (kept_node,
the GC tables, next_history_serial, the position columns) repeated one
skeleton: probe without a transaction, probe again under the lane, read the
recorded schema, decide whether it is an earlier revision without or with
the acquisition tables, compare the live definition, apply, record. `migrate`
now owns it; each migration states its probe, its earlier source and what it
applies.

One ordering difference, with no visible effect: `add_gc` compared the table
set before the live definition and now compares it after. Both are reads
that refuse with the same Integrity error before anything is written.

Checks (macOS host, Rust 1.85.1, PostgreSQL 17 in a private container with
amcheck): Clippy -D warnings on layerfs-persistence, fmt; persistence tests
postgresql_position_columns, postgresql_open_check,
postgresql_statement_schema, postgresql_gc_mark, postgresql_diagnostic and
postgresql_compact pass (52 tests, none NOT_RUN). No test changed or deleted.

Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above
are from before the rebase. This commit compiles with its tests on the new
base; the suites were rerun at the branch tip, not at this commit.

Production LOC: 188,837 -> 188,739 (delta -98)
  core     40,589 ->  40,589 (0)
  engine   90,496 ->  90,398 (-98)  layerfs-persistence -98
  control  51,230 ->  51,230 (0)
  api       6,522 ->   6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --json on parent 7ceff82 and the committed tree (recomputed after the rebase onto 4c296e9)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- `RunMetrics`, the phase clock and §11 values of one `store.compact` run,
  existed twice: backend/sqlite/compact/receipt.rs and
  backend/postgresql/compact/op.rs differed only in `mark_bytes`. One
  definition now lives in compact_metrics.rs beside the recorder it feeds;
  `mark_bytes` is `None` on SQLite and `Some` on PostgreSQL, as each
  reported before.
- `CompactReport`, `CompactGeneration` and the one-implementor trait
  `CompactReportSource` restated `CompactReceipt` field for field. Both
  backends already return `CompactReceipt`, so the control serializer
  `layerfs_store::report::compact_report` now takes it directly. The wire
  shape is unchanged.

Tests: sqlite_compact.rs `the_report_projects_every_section_11_field` is now
`the_receipt_carries_every_section_11_field`; its comparisons of the removed
projection against the receipt (ten field equalities and the trait/From
equality) are gone with the type, and its assertions on the values stay.
compact_report_status.rs passes the receipt to the serializer. No test file
deleted.

Checks (macOS host, Rust 1.85.1, PostgreSQL 17 in a private container):
cargo check --workspace --all-targets; Clippy -D warnings on
layerfs-persistence and layerfs-store; fmt; product boundary guard;
persistence sqlite_compact (33), sqlite_compact_crash (15), compact_metrics
(5), postgresql_compact (30), postgresql_status (8), postgresql_compact_units
(8); layerfs-store whole package (80). sqlite_compact_capacity ran 0 tests on
this host.

Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above
are from before the rebase. This commit compiles with its tests on the new
base; the suites were rerun at the branch tip, not at this commit.

Production LOC: 188,739 -> 188,574 (delta -165)
  core     40,589 ->  40,589 (0)
  engine   90,398 ->  90,234 (-164)  layerfs-persistence -164
  control  51,230 ->  51,229 (-1)  layerfs-store -1
  api       6,522 ->   6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --json on parent 4f136fd and the committed tree (recomputed after the rebase onto 4c296e9)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…alled bind

- history/lookup.rs built eight optional text fields of the MountsActive,
  Unknown and Busy details with `map_or(SdkValue::Null, SdkValue::text)`
  chains. They now use the existing `SdkValue::opt_text`. Same values.
- store/bind.rs: `bind_private_placed` had no caller in engine, control,
  api or any test; removed.

Checks (macOS host, Rust 1.85.1): cargo check --workspace --all-targets;
Clippy -D warnings on layerfs-store-service; fmt; store-service tests
history_wire_codes, admission, branch_remove, project_delete, epoch_fence
and method_refusal_order (23 tests); layerfs-store tests project_delete,
project_delete_store and compact_in_process_call (23 tests). No test changed
or deleted.

Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above
are from before the rebase. This commit compiles with its tests on the new
base; the suites were rerun at the branch tip, not at this commit.

Production LOC: 188,574 -> 188,529 (delta -45)
  core     40,589 ->  40,589 (0)
  engine   90,234 ->  90,189 (-45)  layerfs-store-service -45
  control  51,229 ->  51,229 (0)
  api       6,522 ->   6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --json on parent 52a0697 and the committed tree (recomputed after the rebase onto 4c296e9)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`remove_branch`, `delete_stack` and `remove_path` each wrote the same
admission block before their marker: on PostgreSQL the Store key shared, a
scan for a live conflicting intent and the scope's own key; on both backends
the project epoch hold and the scan that settles dead owners' intents.
`admit` now holds that block once. Each removal passes its scan statement
and parameters, its exclusive key, and whether it also holds the project key
shared (a Branch removal does). Lock order, statements and refusals are
unchanged.

Checks (macOS host, Rust 1.85.1, PostgreSQL 17 in a private container):
Clippy -D warnings on layerfs-persistence (all targets), fmt; persistence
tests admission, branch_removal, layerstack_removal, path_removal,
project_delete, epoch_fence, visibility_guard, semantic_admission (32
tests), postgresql_gc_fence (19) and postgresql_compact (30). No test
changed or deleted.

Rebased onto 4c296e9 (layerfs#734) on 2026-11-18. The test counts above
are from before the rebase. This commit compiles with its tests on the new
base; the suites were rerun at the branch tip, not at this commit.

Production LOC: 188,529 -> 188,487 (delta -42)
  core     40,589 ->  40,589 (0)
  engine   90,189 ->  90,147 (-42)  layerfs-persistence -42
  control  51,229 ->  51,229 (0)
  api       6,522 ->   6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --json on parent 64c76cc and the committed tree (recomputed after the rebase onto 4c296e9)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…r phase (#42 A13)

Owner direction 2026-11-18: the caller of a compaction should see its
stages as they happen, through the same on_progress events store.push and
backup.create already send, so a client's job display needs no polling.

What changes
- API: store.compact declares the shared PROGRESS option, the twelfth call
  to do so. Node, Python and the CLI (--progress) get it from the table;
  index.d.ts, _client.pyi and aio.pyi are regenerated.
- Control: compact() uses the event stream it already received and sends
  { phase, done, total } with null counts as each phase starts.
- Engine: RunMetrics, the phase clock both backends share, holds an
  observer and tells it a phase once, when the phase changes.
  CompactPhase::step gives the name store.status uses. SqliteHandles and
  PostgresHandles gain compact_observed; compact() calls it with no
  observer, so every existing caller is unchanged.
- Phases: mark, build, verify, install on SQLite; mark, build, swap,
  cleanup on PostgreSQL. A none run sends mark alone. The SQLite settle
  (cleanup in store.status, D4) is inside install and sends no event.

What does not change
- The op record and store.status: still the only record another process,
  or a call after a crash, reads. Events are in memory and write nothing.
- No byte counts inside a phase, no estimate before a run, no cancel.

Spec: the frozen GC spec says there is no PROGRESS option (#42 item 40).
The spec file stays byte-identical; the change is recorded as amendment
A13 in core/docs/issues/project-41/gc-42-decisions.md for sign-off.

Tests
- New: an_observed_run_is_told_each_phase_once_in_order (sqlite_compact,
  postgresql_compact) and store_compact_sends_each_phase_as_a_progress_event
  (gc_calls, through layerfs_api::call: none then rebuild).
- Changed on purpose, because the contract changed: the two "four GC calls
  take ids only" tests (gc_calls, calls_table) asserted no options at all;
  they now assert that the three removals have none and that
  store.compact has exactly the on_progress function. The progress lists
  (progress_options, node progress.mjs, python test_progress, CLI
  progress) name twelve calls. No test deleted.
- Ran, each under 120 s, macOS host, after the rebase onto 4c296e9
  (layerfs#734): all 89 layerfs-persistence test files (sqlite_compact
  37); with PostgreSQL 17 in Docker on loopback, postgresql_compact (32),
  postgresql_gc_fence (19) and nine more PostgreSQL files; layerfs-api
  gc_calls (9), calls_table (22), progress_options (2), gc_trace (5),
  store_family (5); layerfs-cli gc (5), progress (5); layerfs-store
  compact_report_status (6), compact_in_process_call (2); nine
  layerfs-store-service removal and refusal files; Clippy -D warnings on
  the five changed crates. gen-node-types --check ran before the rebase
  only.
- The T12.3a matrix report is regenerated: it records the test counts of
  sqlite_compact and postgresql_compact, which each gained one.
- Not run: the Node and Python end-to-end progress tests and
  gen-coverage --check (they need the Python module and a Linux Store
  host; the coverage row is edited to what the generator writes), any
  Linux build, and the macOS worker route for these events.

Docs: errors-and-events.md (call list, shape and phase tables), the API
README count, sdk/store.md, coverage.md and the JS SDK skill reference.

Production LOC: 188,487 -> 188,532 (delta +45)
  core     40,589 ->  40,589 (0)
  engine   90,147 ->  90,180 (+33)  layerfs-persistence +33
  control  51,229 ->  51,241 (+12)  layerfs-store +12
  api       6,522 ->   6,522 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --json on parent b8ff1af and the committed tree (recomputed after the rebase onto 4c296e9)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit f28ab03 into layerfs_v2_0_0 Nov 18, 2026
2 checks passed
@yifanxuaaa
yifanxuaaa deleted the claude/gc-simplify branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant