Skip to content

Reference index of manifest chunks and a read-only verify - #25

Open
Skyfuller1303 wants to merge 30 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:gc-reference-index
Open

Skyfuller1303 wants to merge 30 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:gc-reference-index

Conversation

@Skyfuller1303

@Skyfuller1303 Skyfuller1303 commented Nov 4, 2026 •

Copy link
Copy Markdown

Stacked on #24 (trash), which is stacked on #21. This branch contains their commits; this PR's own commits are git log --no-merges trash..gc-reference-index; the merge commits after them bring in #24's update (a folder's restore and removal take what is in the trash below it). This is the first, non-destructive slice of garbage collection: nothing in it frees, deletes or claims anything. It adds a reference index and a read-only verify. The questions that block any deleting slice are listed at the end.

What

  • Reference index (one migration, nominally 0010; numbered when merged):
    • manifest_chunks (manifest, ordinal, chunk): the chunk list of each manifest, copied from its object in the store.
    • manifest_index: per manifest, how many chunks it names, how many edges are written (ordinals 0 up to that number, never a gap), and whether the list was checked against the manifest (state 1) or is still being built (state 0). A reset in progress is state 2.
    • The index is derived data, never the authority. It answers "which chunks does this manifest need" and "which manifests use this chunk".
  • Writing the index:
    • It is written in the transaction that records a manifest, up to a budget of 5,000 edges per transaction. A longer manifest (up to about a million chunks) is finished by the upload handler in batches of at most that size before the upload is acknowledged, so no write lock is held for a long list.
    • A manifest is marked complete only after every edge was compared, ordinal by ordinal, with its manifest. Equal counts are not enough; repeated chunks, wrong IDs, swapped order, missing, extra and negative-ordinal edges are all found.
    • An index that doesn't match while it's being built is thrown away and rebuilt. The throwing away is also done in batches, behind a marker on the row, so an interrupted run resumes. A complete index that is wrong is only reported.
  • craft-cloud-server index build indexes manifests recorded before the index existed (and any left unfinished). It reads each manifest from the store, checks it against its ID, resumes where it stopped, and may run beside the server.
  • craft-cloud-server verify [--json] [--skip-chunk-reads] reads, reports and changes nothing:
    • The database and the storage directory must already exist. A wrong path is an error, not a new empty server reporting "no problems found". A database with an older schema is refused rather than upgraded.
    • Problems (exit with an error): missing, damaged or non-manifest manifest objects; a complete index that doesn't equal its manifest; edges without an index row; index rows without a recorded manifest; edges or manifests naming an unrecorded chunk; missing or damaged chunk objects; recorded sizes that don't match (stored length of a chunk or manifest object, a chunk length named by a manifest, a manifest's file size); versions or snapshots naming an unrecorded manifest; blob memberships without a blob.
    • Notices (no error): incomplete indexes, manifests no version or snapshot names, chunks no complete manifest names.
  • Store::get_sized (the stored length as read), Meta::open_existing, schema::check_current. Docs in docs/design/sync.md and docs/server.md.

Known gap, stated in the docs

Manifests that no version or snapshot names are not garbage. A live document's binary assets (images and the like, uploaded through the agent's blob interface as a manifest of their own) are named only inside the document's state, which the server stores as opaque bytes. The database holds no reference to them. Until that is decided, anything that frees storage must treat those manifests and their chunks as in use. verify lists them as a notice only.

Questions for you before any deleting slice

I'd propose these slices after this one, each safe on its own and with deletion disabled until its proofs pass: upload reservations and claim-aware publication; Store::list, an inventory and a GC dry run; a maintenance lease with durable claims and fencing; then thinning and trash purge; then account removal. Before any of those can delete, these need your decision:

  1. Roots for app assets: how should a live document's assets be rooted (for example, the agent declares them per document, or the server parses them out of the state)?
  2. Uploads racing a claim: an upload passes its check, writes the object, and only then records it. What should reserve a key across "check, store write, metadata commit"?
  3. A paused deleter: a worker whose lease expired can still finish a store DELETE after another worker took over and the same key was uploaded again. Is a quiescence guarantee acceptable, or do you want fencing (per-incarnation keys, or conditional deletes where the backend supports them)?
  4. Incomplete indexes: should a manifest whose index isn't complete block all collection, or only protect the chunks it might name?
  5. Clock: which time is authoritative for grace periods and leases (server process clock or database time), and what happens on a clock jump?
  6. Purging a file with a live document: does that remove the live document's journal and snapshots, or keep them?

Testing

  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test -p craft-cloud-meta -p craft-cloud-server -p craft-cloud-api -p craft-cloud-proto --locked on SQLite
  • Postgres 17: meta with CRAFT_CLOUD_REQUIRE_BACKENDS=1 (179 tests)
  • cargo test -p craft-cloud-sync --locked; agent tests end_to_end, local_api, binary, network
  • MSRV 1.94 cargo check of proto, meta and server
  • Meta tests on both databases: exact edges, budgeted writes, resumable builds, equal-count wrong indexes, repeated chunks, missing and extra edges, negative ordinals (including i64::MIN), a reset in batches that resumes on a new handle.
  • CLI tests: verify on a clean server, on seeded problems of each kind, with wrong sizes (with and without --skip-chunk-reads), on a wrong database path (nothing created; an empty or foreign file is refused and left alone) and a wrong storage path (nothing created).
  • The fixes for negative ordinals, the batched reset, wrong paths and sizes were reverted once to check that their tests fail without them (SQLite).
  • Edge cases: an extra edge at the highest ordinal (i64::MAX) is reported; a reset whose edge count is an exact multiple of the budget ends and doesn't loop; verify refuses an older and a newer schema and leaves the database file and its WAL bytes unchanged; two concurrent builders on Postgres are held on the index row together (checked in pg_stat_activity) and end with an exact index and no duplicate ordinals. The last two were checked against a deliberately broken build.
  • verify opens an existing SQLite database read-only (mode=ro). A read-write open checkpointed a crashed process's WAL into the database file when its last connection closed: run as a separate process after a crash, the old open changed the database file in 9 of 12 runs. The new tests run the real binary as a child process after a simulated crash, with a row present only in the WAL (and once with -shm deleted first). In every run verify reports that row, and the database file, WAL and blob storage are byte-identical after the process exits (12 of 12 runs each).

Not tested

  • (Since checked: on the merged tree of all open PRs, against MinIO and Postgres 17, verify on S3 passed, and the server's tests ran on Postgres with S3 with no failures. A database created by main upgraded from schema 6 to 14 on Postgres and on SQLite, and verify passed; it then asks for index build, as documented.)
  • Real timing of a million-edge build or reset, and two builders resetting the same manifest at once on Postgres (it uses the existing row lock).
  • verify with a Postgres URL pointing at a missing database.
  • On a SQLite database in WAL mode with no server running, verify may still leave an empty -wal and a -shm file behind (SQLite creates them when it opens the database, even read-only); the database and WAL bytes don't change.

Adds the request and answer types of GET /nodes/{id}/history and POST /nodes/{id}/restore, the opaque history cursor with a single accepted spelling, and the paging limits (default 50, at most 200).
history() pages a file's versions newest first by (seq, id) with an exclusive cursor. restore_version() records a new version with the content of an old one as an ordinary update, checking the source, the current version, deletion and live documents under the same space lock as a commit. No migration is needed.
GET /api/v1/nodes/{id}/history and POST /api/v1/nodes/{id}/restore, resolved in the caller's own space. A stale expected_version answers 409 conflict. Covers permissions, paging, conflicts, restoring old and missing versions, and parallel restores in the API tests.
Updates the API table and the milestone note in the sync design, and adds a short section to the server guide.
A history page now reads versions through an index on (space_id, node, seq, id) in the order of the query, so the page size bounds the work. The migration adds the index on both databases. The restore note in the sync design no longer claims the engine prefers a newer live-document state: a restore is an ordinary new version and files with a live document are refused.
Adds the list, restore and cursor types of GET /trash and POST /trash/{id}/restore, with the same page limits as the history.
…und taken names

A migration adds nodes.deleted_ms and nodes.purge_requested_ms and an index for the trash list, and dates old deletions at the migration. The commit path stamps the time when a node goes from live to deleted and clears it when the node comes back. restore_from_trash brings a deleted node and its deleted folders back in one transaction, merging into live folders of the same name and renaming around taken names with the engine's conflict pattern. request_purge only marks a node, so nothing is freed.
GET /api/v1/trash, POST /api/v1/trash/{id}/restore and DELETE /api/v1/trash/{id}, resolved in the caller's own space, and the [retention] trash_days setting (default 30, 1 to 3650) that sets purge_after_ms.
Adds the trash routes and rules to the sync design and a section with the new setting and the rollout note to the server guide.
…ders within the depth limit

A file restored from the trash now gets a new current version with the content of its last one (no save record). A machine that sent a delete against the old version and never heard the answer can no longer delete the restored file again: the server refuses that delete as a version mismatch, and the engine already lets go of a node that came back at another version. The restore answer reports the new version. A folder that comes back below a folder that has since moved deep is refused as too deep, and nothing changes.
A folder has no version to tell a delete that went through and a later restore from one that never arrived. The engine now remembers the folder deletes it sent without hearing the answer, and when the next pull reports such a folder changed and live, lets it go instead of deleting it again. A folder the feed does not mention was never touched and its delete is sent again. Scenario tests cover a file and a folder restored after a lost answer, at its own path, under a conflict name, and merged into a live folder.
A restored file has a new version, and agents need the folder rule for the rare case of a folder restored after a delete whose answer was lost. Reports the new version in the restore answer.
…d are settled

A folder deleted together with its file, whose answer was lost, and then restored alone, still had the file tracked when the pull judged it, so it was kept and deleted again once the file was let go. The pull now only notes which folders the feed reports live, and the decision is made in apply_remote after the files whose deletes went through are let go. Scenarios cover the non-empty folder restored alone, the folder and its file both restored, and only the file restored.
A folder this machine deleted with a lost answer and that came back from the trash was judged before the folders below it were settled. A tombstoned subfolder still tracked in it kept it, and the next push deleted the restored folder again. The folders of this machine's own delete are now settled deepest first: one the server lists as live, or no longer has and that is gone here too, is let go, and only then are the parents judged.

A relist at the same epoch (the change feed answered resync required) did not note which of those folders the server lists as live, so a restored empty folder stayed tracked and was deleted again. The relist now notes them, so a folder the server has is never pushed as deleted from a stale request.

Regressions cover d/sub/f and d/a/b/f with only the top folder, a middle folder, or only the deepest node restored, and a relist after the restore.
A migration adds manifest_chunks (manifest, ordinal, chunk) and manifest_index (chunk count, edges written, state). The index is written in the transaction that records a manifest, up to a budget of edges per transaction, and a longer manifest is finished in batches by index_complete, which also resumes an interrupted build and indexes manifests recorded before the index existed. An index is marked complete only after every edge was compared with its manifest, so wrong, swapped, missing, extra and collapsed edges are found. index_audit reports what the database alone can tell: orphan edges, index rows without a manifest, unknown dependencies, roots that name unrecorded manifests, and manifests that nothing names. Nothing is deleted except the derived edges of an index that does not match its manifest while it is still being built.
…ore the upload is acknowledged

verify checks the database and the store against each other and reports problems and notices as text or JSON, changing nothing. index build indexes manifests recorded before the index existed, in bounded batches. PUT /manifests completes the manifest's index before answering.
The comparison of a manifest's edges started its pages after ordinal -1, so an edge at a negative ordinal was never seen and an index with such an edge could be marked complete. The comparison now starts at the lowest number there is and reports an edge at a negative ordinal as a mismatch of its own kind, so completion is never published over it.

Starting an index again deleted all of a manifest's edges in one transaction, up to about a million rows, which broke the batch budget and blocked other writers. The reset now marks the index row as resetting (state 2) and deletes at most the index budget per transaction, lowest ordinal first; nothing is written or marked complete until the last edge is gone, and a run that stops between two batches is resumed by the next stage. The old behaviour of a complete row that is wrong being left to the checker is unchanged.

Regressions cover negative ordinals (including the lowest possible), a reset across several batches that resumes on a new handle after a stop, and a build that resets a wrong index by itself.
…es it reads

`verify` opened SQLite with create enabled and made missing storage directories, so a wrong path gave a new empty server and "no problems found". It now opens the database without creating it (`Meta::open_existing`: a missing file, a file with no Craft Cloud data, or a schema older than the program is an error, and nothing is created or upgraded) and requires the storage directory to exist. `index build` still uses the creating open.

A successful read of a chunk or manifest object discarded its bytes, so seeded size corruption passed. Reading now reports the stored length (format byte included) and verify checks it against `blobs.stored_size` for chunks and manifests, the file size recorded for a manifest against the manifest, and the chunk lengths a manifest names against the length recorded for each chunk (one less than the stored size).

Regressions cover a wrong database path, a wrong storage path, and seeded wrong sizes with and without reading chunk objects.
…y claim

The verify schema test now snapshots the database and WAL straight from disk, with no SQLite connection opened between the
setup checkpoint and the check. The concurrent index reset test holds the manifest_index row lock, proves both builders
wait on it, and only then releases it. The i64::MAX edge test is renamed for what it pins; the cursor overflow guard is not
reachable through the public API.
@Skyfuller1303

Copy link
Copy Markdown
Author

Decisions and assumptions for the Craft Cloud PRs

Hi, this collects the decisions that block upcoming work and the assumptions in the open PRs. Each item gives a proposed default; please confirm it or redirect us. We will use the proposed defaults for drafting unless you say otherwise. Destructive GC remains disabled until its safety contract is agreed and demonstrated.

Decisions blocking whole slices

Destructive GC and retention

  1. Upload/claim race: How should an upload be protected between its initial check, the store write, and metadata publication? Options: a durable reservation spanning the whole sequence; claim-aware publication with a proven equivalent protocol; another owner-specified protocol. Recommend: a durable reservation, with generation checks and transactional publication; a pre-write check alone cannot stop a concurrent claim. Unblocks: safe GC publication and later deletion. Raised: Reference index of manifest chunks and a read-only verify #25; CP-06c risk table, upload race.

  2. Orphan grace: How long should an unreferenced object remain protected after its first observation? Options: two observations separated by an agreed wait_hours; a different fixed grace; a durable upload-intent inventory. Recommend: two observations with a persisted first-seen time and an owner-chosen grace; the current dry run cannot measure age without persisted state. Unblocks: orphan collection and its configuration. Raised: CP-06 §8; GC dry-run report.

  3. Paused deleter and backup: What must prevent an expired worker from deleting an object after its claim or maintenance lease is taken over? Options: storage-enforced generation/conditional-delete fencing; a proven protocol that guarantees all previous deleters have quiesced before reuse or backup; keep deletion disabled. Recommend: require enforceable fencing or proven quiescence, and keep deletion disabled until demonstrated; a database owner token and expiring lease alone cannot fence an in-flight store operation. Unblocks: GC, recovery and consistent backups. Raised: Reference index of manifest chunks and a read-only verify #25; CP-06 §7a and CP-06c risks.

  4. Application assets and incomplete indexes: How should manifests referenced by opaque live-document state and children of an incomplete manifest index be protected? Options: explicit client-declared asset roots plus fail-closed incomplete indexes; server-side extraction plus a complete index; preserve all potentially referenced objects. Recommend: preserve unrooted manifests and block collection where children cannot be proven safe until an explicit asset-root protocol is agreed. This avoids deleting objects the server cannot see. Unblocks: live-document GC and complete inventories. Raised: Reference index of manifest chunks and a read-only verify #25; CP-06 §5, §8 and CP-06c risks.

  5. Clock and time jumps: Which clock governs grace periods, leases and claims, and what happens when it jumps? Options: database time with a documented conservative jump policy; a designated server clock with skew limits; no expiry-driven takeover. Recommend: use one authoritative database clock and fail closed when safe elapsed time cannot be established. Unblocks: reliable expiry, recovery and GC scheduling. Raised: Reference index of manifest chunks and a read-only verify #25; CP-06c risks.

  6. Live-document purge: Should permanent file purge also remove its live-document journal and snapshots? Options: purge both with the file; retain them under a separately defined retention policy; defer live-document purge. Recommend: defer their deletion until their roots and recovery requirements are specified; retaining data is reversible. Unblocks: live-document permanent purge. Raised: Reference index of manifest chunks and a read-only verify #25; CP-06c risks.

  7. Version retention: Is the proposed thinning grid right: all versions for 24 hours, then hourly through 7 days, daily through 30 days, and weekly through 365 days, with older versions retained? Options: that grid; another grid; no automatic thinning. Recommend: the proposed grid only after its boundaries and saved/head protections are confirmed; it bounds history without silently removing older records. Unblocks: version thinning. Raised: CP-06 §5 and retention rules.

  8. Trash lifecycle: Should expiry purge a deleted folder and its deleted descendants while preserving an ancestor tombstone with live descendants, and should revival clear its purge mark? Options: this policy; a different descendant/mark policy; defer purge. Recommend: preserve live descendants and clear the mark on revival, then independently test restore/delete cycles. Unblocks: safe trash purge. Raised: Trash: list, restore and mark deleted files and folders #24, Reference index of manifest chunks and a read-only verify #25; CP-06 retention and CP-06c risks.

Backup and restore

  1. Backup format: Should backup use a database-neutral logical snapshot in a local directory with BLAKE3 verification, or database-native snapshots/one archive? Options: logical directory; native database tools; a single archive. Recommend: the logical directory: one streaming verification path and SQLite/Postgres cross-restore. Unblocks: CP-07 format and tooling. Raised: CP-07, open questions 1–2.

  2. Backup confidentiality: Should CP-07 ship an unencrypted backup or include encryption? Options: unencrypted with protected storage and separately backed-up keys; encrypted with a separately designed key/recovery protocol. Recommend: the former for the first slice, explicitly documenting that it contains password and token hashes and encrypted 2FA secrets; encryption needs its own recovery design. Unblocks: backup format and operator guidance. Raised: CP-07, question 3.

  3. Restore isolation and destination: Should restore require enforced instance exclusion and an empty database and store prefix? Options: instance locking plus empty target; operator-only shutdown plus empty target; a separately designed destructive replacement. Recommend: enforce the lock and reject any nonempty target, including foreign S3 objects, before writing; this prevents overwrite and mixing. Unblocks: safe restore. Raised: CP-07, questions 4–6; CP-06 §7a. A stale/incomplete S3 listing cannot by itself prove emptiness; that guarantee must be resolved first.

  4. Backup scope and recovery: Should the first version copy all referenced objects, decline resumable restore, use backup create|verify|restore, and accept only current/previous schema versions? Options: this bounded first version; metadata-only/external bucket backup; resumable or broader-version restore with a verified journal and fixtures. Recommend: the bounded first version, with interrupted restores retried into a clean target; it avoids unverified partial imports. Unblocks: CP-07 scope. Raised: CP-07, questions 5, 7–9.

Email and teams

  1. Email: Is opt-in operator-configured SMTP in scope, and may a background task retry an outbox? Options: opt-in SMTP with an outbox worker; opt-in SMTP with bounded request-path delivery; no email. Recommend: keep it disabled by default and use an outbox worker if you accept a background task; delivery retries then do not block account actions. Unblocks: CP-09 design. Raised: CP-09, scope and background-task questions.

  2. Team spaces: May we first propose a design with owner/editor/viewer membership and separate team-space routes, leaving multi-space agent behavior for your decision? Options: design first; defer teams; another model. Recommend: design only, no implementation until accepted, because authorization, migration and sync behavior are cross-cutting. Unblocks: CP-10 design. Raised: CP-10, Phase A and sync-engine sections.

Delayed saves

  1. Delayed save: When a save arrives for a version already thinned, should the agent discard the save after the server returns conflict, or surface it for recovery? Options: discard; surface the failure; prevent thinning until an explicit acknowledgement. Recommend: surface the failure rather than silently lose a save, pending a defined recovery path. Unblocks: thinning and agent behavior. Raised: CP-06 retention rules, delayed saves.

PR assumptions — confirm or redirect

The groups follow the proposed merge order; grouped choices include all listed PRs. Items without a separate owner question are still listed for scope confirmation.

  1. Docker: hardened image and Compose deployment #13, Device sessions: account info, password change, signing devices out #14 and Agent: build historical live snapshots without Loro's checkout (fixes #15) #16: Confirm Compose packaging, device-session defaults and the bounded-memory snapshot approach? Default: ship local-build Compose without a registry workflow (Docker: hardened image and Compose deployment #13); keep non-expiring device tokens, 1 MiB login/join limits and no idle expiry (Device sessions: account info, password change, signing devices out #14); replay historical operations with memory proportional to history (Agent: build historical live snapshots without Loro's checkout (fixes #15) #16). Confirm / redirect.

  2. Enforce the admin role and add an HTTP admin API #17–Desktop admin pages, audit log and activity #20 and Web admin pages with browser sessions #22/Name web admin sign-ins in the audit log #27: Confirm admin, audit, 2FA and desktop/web administration scope? Default: deleted accounts await GC (Enforce the admin role and add an HTTP admin API #17); audit retention defaults to 365 days and append-only is application-enforced (Security audit log #18); 2FA uses TOTP plus one-use recovery codes with key held separately (Two-factor sign-in (TOTP and recovery codes) #19); desktop shows IDs and omits time-filter UI (Desktop admin pages, audit log and activity #20); web uses server-rendered HTML, SameSite Strict, 8-hour idle/7-day absolute sessions, one-time code display and no browser-session list (Web admin pages with browser sessions #22). Name web admin sign-ins in the audit log #27 supersedes Web admin pages with browser sessions #22's original missing web audit kind; keep the Name web admin sign-ins in the audit log #27 behavior. Confirm / redirect.

  3. Version history and restoring an old version #21 history: Should restore copy the old mtime, permit restoring the head as a new version, keep save as a boolean, reject deleted-file and live-document restores, and omit quota/rate limits because bytes are reused? Default: yes; restore deleted files through trash instead. Confirm / redirect.

  4. Trash: list, restore and mark deleted files and folders #24 trash: Should tombstones carry deletion timestamps and purge marks rather than a new table, with unknown times assigned on a later read rather than immediately expired? Default: yes; set time only on live-to-deleted transition, clear time/mark on revival, use a snapshot read, and never log names. Confirm / redirect.

  5. Trash: list, restore and mark deleted files and folders #24 restore and remove: Should restore merge deleted ancestors into same-name live folders, use the sync conflict name (up to 100 attempts), create a new file version, and keep same-epoch uncertain deletes when relisting finds the folder live? Default: yes; do not merge the restored target itself, and DELETE marks for later GC without freeing data. Confirm / redirect.

  6. Server tests: run the API scenarios on Postgres with a disk store #23 and Raise the minimum Rust version to 1.95 and check it in CI #26: Confirm the Postgres-plus-disk test matrix and raising declared MSRV to 1.95 with a matching CI job? Default: both: Server tests: run the API scenarios on Postgres with a disk store #23 exercises Postgres without S3, and egui 0.36 requires Rust 1.95. Confirm / redirect.

  7. Reference index of manifest chunks and a read-only verify #25 and Store: list the blobs of a kind in pages #29: Confirm paged store listing as the inventory foundation, with invalid object-store keys failing listing, an opaque cursor bounded to 4096 bytes, and the public hidden test counter? Default: retain these; feature-gate the counter instead if preferred. All destructive decisions remain gated by questions 1–8. Confirm / redirect.

  8. File versions in the Craft Cloud app, through the agent #28 history UI and Trash in the agent's local API and the Craft Cloud app #30 trash UI: Is a path text field preferable to a file picker for version history, with restore disabled while loading and its confirmation preserved? Default: yes; keep trash addressed by node ID and use optional path hints. Confirm / redirect.

  9. Trash: list, restore and mark deleted files and folders #24 folder delete in doubt: On a same-epoch relist, a folder whose delete is in doubt and that the server lists live is kept. A listing cannot tell a lost answer from a delete that never reached the server, so this also undoes a folder delete that was never sent; the user deletes it again. Default: keep the folder (losing a delete is recoverable, deleting a restored folder is not). Alternative: remember whether the request left the machine and resend a delete that was never sent. Confirm / redirect.

  10. Reference index of manifest chunks and a read-only verify #25 verify and old schemas: verify refuses a database with an older (or newer) schema instead of upgrading it, so a read-only check never migrates. Default: refuse; the operator upgrades by starting the server once, after a backup. Confirm / redirect.

  11. Reference index of manifest chunks and a read-only verify #25 stored chunk length: verify treats a chunk as sound when its stored size equals its decoded length plus one format byte (stored_size = len + 1), matching how uploads are counted today. Compression (planned later) will change the stored size; the check will then compare against the decoded length after unpacking. Default: keep the len + 1 rule until compression lands. Confirm / redirect.

Merge order and stacks

Suggested order: independent #13, #16, #23 and #26 as convenient; #14 → #17 → #18 → #19 → #20; #22 after its admin/2FA base, then #27; #21 → #24 → #25, with #28 after #21 and #29 independently before inventory work. #30 (trash UI) is stacked on both #24 and #28. Exact PR bases and migration numbers should be checked again when landing.

Current local branch comparisons show trash versus admin-api conflicts in server main/state/tests, metadata schema and tests, and design/server docs; migration order requires deliberate integration. Current integrated audit/2FA/desktop branches merge textually cleanly, unlike the earlier stack analysis; text-clean merges still require semantic migration and audit review. The history/trash, trash/reference-index, history-UI/trash, web/admin and audit-web/audit comparisons were also textually clean.

Already answered

No owner comments or reviews answering these questions were found in the available PRs. The web audit-kind omission was resolved in #27 by code, not by an owner answer.

…nnot checkpoint a crashed WAL

Closing the last read-write connection to a WAL-mode database checkpoints the
WAL of a crashed process into the database file and removes it. Whether that
finished before verify exited was a race, and a command that only reads must
not make that write. Meta::open_existing now opens SQLite with mode=ro, which
still reads the WAL's committed data. The tests run verify as a process of its
own and compare the database file, the WAL and the storage tree after it has
exited, with a row that only the WAL holds and with the -shm file gone.
@Skyfuller1303

Copy link
Copy Markdown
Author

Hi, following up on the questions above with a tested merge order for the open PRs.

All of them were merged together locally, in the order below, and the result passes fmt, clippy -D warnings, the SQLite test suites (meta, server, store, proto, api, app), meta and server on Postgres, the agent's local_api and lib tests, the desktop tests and a cargo check on Rust 1.95. A database created by the current main server upgraded cleanly to the final schema, and verify passed on it afterwards (SQLite; the same check from a Postgres database created by main was not run).

The merged result is on the fork as integration, for reference only; it is not meant to be merged as is.

Order

Each line is one PR tip; a later tip already contains the PRs in brackets, so merging in this order means each PR lands on top of what it was built on.

  1. Docker: hardened image and Compose deployment #13 Docker
  2. Agent: build historical live snapshots without Loro's checkout (fixes #15) #16 agent memory bound
  3. Server tests: run the API scenarios on Postgres with a disk store #23 Postgres plus disk tests
  4. Raise the minimum Rust version to 1.95 and check it in CI #26 MSRV 1.95
  5. Desktop admin pages, audit log and activity #20 desktop administration (contains Device sessions: account info, password change, signing devices out #14 → Enforce the admin role and add an HTTP admin API #17 → Security audit log #18 → Two-factor sign-in (TOTP and recovery codes) #19)
  6. Name web admin sign-ins in the audit log #27 audit kind for web sessions (contains Web admin pages with browser sessions #22)
  7. Trash in the agent's local API and the Craft Cloud app #30 trash UI (contains Version history and restoring an old version #21, Trash: list, restore and mark deleted files and folders #24, File versions in the Craft Cloud app, through the agent #28)
  8. Store inventory and gc --dry-run, which changes nothing #31 GC dry run (contains Reference index of manifest chunks and a read-only verify #25, Store: list the blobs of a kind in pages #29)
  9. Account usage: stored bytes, quota and trash/history estimates (read-only) #32 account usage (on Reference index of manifest chunks and a read-only verify #25; not yet in integration, it has small textual conflicts in crates/meta/src/lib.rs, crates/proto/src/api.rs and docs/design/sync.md against it)

#13, #16, #23 and #26 are independent and merged cleanly in any position. Steps 5 to 8 had conflicts; nothing was dropped to resolve them. The only real one is step 7 (trash against the admin stack, in the server, meta and docs). The resolution commits on integration are named integration: ... and can be reused.

Migration numbers

Each branch numbers its migrations from main, so they collide. In this order they become:

Migration On its branch Final
device sessions (#14) 0007 0007
admin resets (#17) 0008 0008
audit log (#18) 0009 0009
two-factor (#19) 0010 0010
web-session generation (#22/#27) 0011 0011
version sequence index (#21) 0008 0012
trash times and purge marks (#24) 0009 0013
manifest reference index (#25) 0010 0014

The final schema version is 14. Only the file names, the migration list and the schema-version expectations in tests change; the SQL itself stays as it is. #32 adds no migration.

If you'd rather merge in a different order, tell me, and I'll renumber the affected branches before you merge them. Alternatively I can rebase each PR onto the previous one as they land.

Known after merging

…he trash

Restoring a folder from the trash now also brings back every node below it that
is in the trash (deleted and not removed from the trash), however long before it
was deleted, in the same transaction; the trash list shows such a node only
through its folder; and removing a folder from the trash marks them too. A
folder with more than 10,000 such nodes below it is refused whole as invalid
for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first

Restoring a folder from the trash now brings back all that is in the trash below
it, so the lost-delete tests that restored a folder alone and expected its file
to stay deleted take that file (or sub folder) out of the trash first.
Conflict in crates/meta/src/lib.rs: both new Meta fields (index_rows and
trash_tree_cap) are kept.
… rest is listed

Instead of refusing a folder with more than 10,000 items in the trash below it
(which left it impossible to restore or remove from a list that hides what is
below a trashed folder), a restore or removal now takes the folder and at most
the cap of items below it, shallowest first. The rest stays in the trash and is
listed on its own because its folder is live or removed, so a further call takes
it. The per-row work under the write lock is bounded by the cap.
Another test helper in a later branch is called sorted; a distinct name keeps the
trash tests compiling wherever they are merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant