Repository navigation
Store inventory and gc --dry-run, which changes nothing - #31
Open
Skyfuller1303 wants to merge 38 commits into
Open
Skyfuller1303 wants to merge 38 commits into
Skyfuller1303 wants to merge 38 commits into
Conversation
Adds the request and answer types of GET /nodes/{id}/history and POST /nodes/{id}/restore, the opaque history cursor with a single accepted spelling, and the paging limits (default 50, at most 200).
history() pages a file's versions newest first by (seq, id) with an exclusive cursor. restore_version() records a new version with the content of an old one as an ordinary update, checking the source, the current version, deletion and live documents under the same space lock as a commit. No migration is needed.
GET /api/v1/nodes/{id}/history and POST /api/v1/nodes/{id}/restore, resolved in the caller's own space. A stale expected_version answers 409 conflict. Covers permissions, paging, conflicts, restoring old and missing versions, and parallel restores in the API tests.
Updates the API table and the milestone note in the sync design, and adds a short section to the server guide.
A history page now reads versions through an index on (space_id, node, seq, id) in the order of the query, so the page size bounds the work. The migration adds the index on both databases. The restore note in the sync design no longer claims the engine prefers a newer live-document state: a restore is an ordinary new version and files with a live document are refused.
Adds the list, restore and cursor types of GET /trash and POST /trash/{id}/restore, with the same page limits as the history.
…und taken names A migration adds nodes.deleted_ms and nodes.purge_requested_ms and an index for the trash list, and dates old deletions at the migration. The commit path stamps the time when a node goes from live to deleted and clears it when the node comes back. restore_from_trash brings a deleted node and its deleted folders back in one transaction, merging into live folders of the same name and renaming around taken names with the engine's conflict pattern. request_purge only marks a node, so nothing is freed.
GET /api/v1/trash, POST /api/v1/trash/{id}/restore and DELETE /api/v1/trash/{id}, resolved in the caller's own space, and the [retention] trash_days setting (default 30, 1 to 3650) that sets purge_after_ms.
Adds the trash routes and rules to the sync design and a section with the new setting and the rollout note to the server guide.
…ders within the depth limit A file restored from the trash now gets a new current version with the content of its last one (no save record). A machine that sent a delete against the old version and never heard the answer can no longer delete the restored file again: the server refuses that delete as a version mismatch, and the engine already lets go of a node that came back at another version. The restore answer reports the new version. A folder that comes back below a folder that has since moved deep is refused as too deep, and nothing changes.
A folder has no version to tell a delete that went through and a later restore from one that never arrived. The engine now remembers the folder deletes it sent without hearing the answer, and when the next pull reports such a folder changed and live, lets it go instead of deleting it again. A folder the feed does not mention was never touched and its delete is sent again. Scenario tests cover a file and a folder restored after a lost answer, at its own path, under a conflict name, and merged into a live folder.
A restored file has a new version, and agents need the folder rule for the rare case of a folder restored after a delete whose answer was lost. Reports the new version in the restore answer.
…d are settled A folder deleted together with its file, whose answer was lost, and then restored alone, still had the file tracked when the pull judged it, so it was kept and deleted again once the file was let go. The pull now only notes which folders the feed reports live, and the decision is made in apply_remote after the files whose deletes went through are let go. Scenarios cover the non-empty folder restored alone, the folder and its file both restored, and only the file restored.
A folder this machine deleted with a lost answer and that came back from the trash was judged before the folders below it were settled. A tombstoned subfolder still tracked in it kept it, and the next push deleted the restored folder again. The folders of this machine's own delete are now settled deepest first: one the server lists as live, or no longer has and that is gone here too, is let go, and only then are the parents judged. A relist at the same epoch (the change feed answered resync required) did not note which of those folders the server lists as live, so a restored empty folder stayed tracked and was deleted again. The relist now notes them, so a folder the server has is never pushed as deleted from a stale request. Regressions cover d/sub/f and d/a/b/f with only the top folder, a middle folder, or only the deepest node restored, and a relist after the restore.
A migration adds manifest_chunks (manifest, ordinal, chunk) and manifest_index (chunk count, edges written, state). The index is written in the transaction that records a manifest, up to a budget of edges per transaction, and a longer manifest is finished in batches by index_complete, which also resumes an interrupted build and indexes manifests recorded before the index existed. An index is marked complete only after every edge was compared with its manifest, so wrong, swapped, missing, extra and collapsed edges are found. index_audit reports what the database alone can tell: orphan edges, index rows without a manifest, unknown dependencies, roots that name unrecorded manifests, and manifests that nothing names. Nothing is deleted except the derived edges of an index that does not match its manifest while it is still being built.
…ore the upload is acknowledged verify checks the database and the store against each other and reports problems and notices as text or JSON, changing nothing. index build indexes manifests recorded before the index existed, in bounded batches. PUT /manifests completes the manifest's index before answering.
The comparison of a manifest's edges started its pages after ordinal -1, so an edge at a negative ordinal was never seen and an index with such an edge could be marked complete. The comparison now starts at the lowest number there is and reports an edge at a negative ordinal as a mismatch of its own kind, so completion is never published over it. Starting an index again deleted all of a manifest's edges in one transaction, up to about a million rows, which broke the batch budget and blocked other writers. The reset now marks the index row as resetting (state 2) and deletes at most the index budget per transaction, lowest ordinal first; nothing is written or marked complete until the last edge is gone, and a run that stops between two batches is resumed by the next stage. The old behaviour of a complete row that is wrong being left to the checker is unchanged. Regressions cover negative ordinals (including the lowest possible), a reset across several batches that resumes on a new handle after a stop, and a build that resets a wrong index by itself.
…es it reads `verify` opened SQLite with create enabled and made missing storage directories, so a wrong path gave a new empty server and "no problems found". It now opens the database without creating it (`Meta::open_existing`: a missing file, a file with no Craft Cloud data, or a schema older than the program is an error, and nothing is created or upgraded) and requires the storage directory to exist. `index build` still uses the creating open. A successful read of a chunk or manifest object discarded its bytes, so seeded size corruption passed. Reading now reports the stored length (format byte included) and verify checks it against `blobs.stored_size` for chunks and manifests, the file size recorded for a manifest against the manifest, and the chunk lengths a manifest names against the length recorded for each chunk (one less than the stored size). Regressions cover a wrong database path, a wrong storage path, and seeded wrong sizes with and without reading chunk objects.
…y claim The verify schema test now snapshots the database and WAL straight from disk, with no SQLite connection opened between the setup checkpoint and the check. The concurrent index reset test holds the manifest_index row lock, proves both builders wait on it, and only then releases it. The i64::MAX edge test is renamed for what it pins; the cursor overflow guard is not reachable through the public API.
Adds Backend::list_raw(prefix, after, limit) and Store::list(kind, cursor, limit), which returns a page of blob keys, the foreign objects found under the kind's prefix (reported, never dropped), and an opaque cursor for the next page. Later work needs it: a store inventory and a GC dry run, and the "store is empty" check of a backup restore. It only lists; nothing is deleted. - Every backend lists in the same byte order of the relative path, strictly after the cursor, and never outside its root or prefix. A page holds at most 1,000 keys; 0 is refused, a larger limit is clamped. - Disk reads and sorts one directory at a time, skips directories that lie wholly before the cursor and stops when the page is full, so a page costs one directory listing plus the page. It skips symlinks and only the writer's own temporary files (`.tmp-` and a UUIDv7); other names are foreign. - S3 lists with object_store's list_with_offset and stops when the page is full, relying on S3's ascending key order (an out-of-order key is an error). A key object_store can't represent (`a//b`) fails the listing with a specific error, and `a/b/` is seen as `a/b`; both limits are documented. A listing may be incomplete or stale on some S3-compatible servers. - Memory reads a bounded range of its map. - A cursor names at most a 4,096-byte path and is checked for size before it is decoded; a malformed or wrong-kind cursor is an error.
…nnot checkpoint a crashed WAL Closing the last read-write connection to a WAL-mode database checkpoints the WAL of a crashed process into the database file and removes it. Whether that finished before verify exited was a race, and a command that only reads must not make that write. Meta::open_existing now opens SQLite with mode=ro, which still reads the WAL's committed data. The tests run verify as a process of its own and compare the database file, the WAL and the storage tree after it has exited, with a row that only the WAL holds and with the -shm file gone.
…thout the object Store::get_sized now reads at most the kind's limit plus one byte, so a damaged or foreign object much larger than any blob is refused without being read in full. Backend::size_raw gives an object's stored size from metadata (file metadata on disk, HEAD on S3); a backend without it answers with an error rather than a guess. The agent's counting test backend forwards both.
… nothing craft-cloud-server gc --dry-run [--json] pages through every blob kind of the store, looks each page's keys up in one batched query, and classifies every object as recorded, orphan (no blobs row, such as an upload the quota refused after the store write) or foreign (not a valid key). It then reports what a later collector would consider and why: orphans only after two runs apart, recorded chunks no complete-indexed manifest names only when every manifest index is complete and the database checks pass, and never a manifest no version or snapshot names, since it may be a live document's asset. Everything a current or saved version, a trashed node, a live snapshot or an import snapshot names is protected. Without --dry-run the command refuses. The database and the store are opened existing-only, like verify, and nothing is written. The tests run the real binary as a child process and compare the database (SQLite file and WAL bytes; every Postgres table and sequence) and the whole store tree before and after it exits.
This was referenced Nov 5, 2026
…he trash Restoring a folder from the trash now also brings back every node below it that is in the trash (deleted and not removed from the trash), however long before it was deleted, in the same transaction; the trash list shows such a node only through its folder; and removing a folder from the trash marks them too. A folder with more than 10,000 such nodes below it is refused whole as invalid for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first Restoring a folder from the trash now brings back all that is in the trash below it, so the lost-delete tests that restored a folder alone and expected its file to stay deleted take that file (or sub folder) out of the trash first.
Conflict in crates/meta/src/lib.rs: both new Meta fields (index_rows and trash_tree_cap) are kept.
… rest is listed Instead of refusing a folder with more than 10,000 items in the trash below it (which left it impossible to restore or remove from a list that hides what is below a trashed folder), a restore or removal now takes the folder and at most the cap of items below it, shallowest first. The rest stays in the trash and is listed on its own because its folder is live or removed, so a further call takes it. The per-row work under the write lock is bounded by the cap.
Another test helper in a later branch is called sorted; a distinct name keeps the trash tests compiling wherever they are merged.
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #25 (reference index and
verify) and #29 (store listing). This branch is #25 with #29 merged in (merge commit 544c563); this PR's own commits are the three ofgit log --no-merges gc-dry-run ^gc-reference-index ^store-list; the later merge commits bring in #25's update.verify(#25) compares the database against the store. This adds the other direction: an inventory of the store compared against the database, and a GC dry run that reports what a later collector would consider, and why. Nothing is deleted, claimed, marked or recorded, and there is no new table or migration. The deleting slices wait for the decisions listed in the comment on #25.What
craft-cloud-server gc --dry-run [--json] --config <path>. Without--dry-runit refuses with "deleting is not implemented yet; run with --dry-run". It opens the database and the store likeverify: existing only, schema checked, nothing created, SQLite read-only (Reference index of manifest chunks and a read-only verify #25).Inventory. It pages through every blob kind with
Store::list, 1,000 keys at a time, and looks each page up in one batched query. Each object is:recorded: it has ablobsrow. Count and stored bytes per kind, taken from the database'sstored_size.orphan: it has noblobsrow, for example an upload the quota refused after the store write. Count, stored bytes (read from object metadata, not the object), and up to 20 IDs as examples.foreign: not a valid key for its kind. Count and pseudonymous examples, never file names.A listing error stops the inventory and is reported with its kind and page, never as an empty store. Missing objects reuse
verify's checks.Candidates, each with a reason and the protections that applied:
versions), live snapshots and import snapshots are roots. Totals, index state, roots and candidates come from one read-only database snapshot. The store listing is a separate point in time, so a later collector must recheck roots before it claims anything. The code and docs say so.Output: a text summary with counts and bytes per class and reason, the protections, and "nothing was changed"; or
--jsonwithschema_version: 1. Exit 0 when no problem is found (candidates aren't problems); non-zero on problems such as missing objects or a listing or size error.Store:
Store::get_sizednow reads at most the kind's limit plus one byte, so a far too large object is refused without being read whole.Backend::size_rawgives an object's size from metadata (file metadata on disk,HEADon S3). A backend without it answers with an error, which the dry run reports as a problem. The agent's counting test backend forwards both.Memory: one page plus fixed summaries and at most 20 examples per class.
Docs:
docs/server.mdanddocs/design/sync.md(classes, reasons, protections, and why grace and claims are left to a later slice).Assumptions (please confirm or redirect)
get_sizedand the newsize_rawlive in this PR because the inventory needs them. They could move to their own PR.Testing
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --locked -- -D warnings.cargo test -p craft-cloud-meta -p craft-cloud-server -p craft-cloud-store --lockedon SQLite: meta 185, server 61 unit, 86 API and 27 CLI, store 6 unit and 17 conformance.Postgres 17:
craft-cloud-metawithCRAFT_CLOUD_REQUIRE_BACKENDS=1;craft-cloud-server --test cli, where the scenario test ran on Postgres as well as SQLite.cargo checkof proto, meta, store and server on Rust 1.95.One scenario test, run on SQLite and on Postgres. It covers:
"Changes nothing": every dry run in the tests runs the real binary as a child process. After it has exited, the test compares:
This includes a SQLite database a crashed process left with committed data only in the WAL. With Reference index of manifest chunks and a read-only verify #25's read-only open, the crash tests passed 12 of 12 repeated runs.
Each rule was checked against a deliberately broken build, and each break made the tests fail:
blobs,space_blobsandspacesduring the run;putduring the run;Not tested
size_rawmatched a 2 MiB object, andgc --dry-runleft the whole bucket listing and every Postgres table and sequence unchanged; a foreign and an orphan object were classified as such.)