Repository navigation
Reference index of manifest chunks and a read-only verify - #25
Skyfuller1303 wants to merge 30 commits into
Conversation
Adds the request and answer types of GET /nodes/{id}/history and POST /nodes/{id}/restore, the opaque history cursor with a single accepted spelling, and the paging limits (default 50, at most 200).
history() pages a file's versions newest first by (seq, id) with an exclusive cursor. restore_version() records a new version with the content of an old one as an ordinary update, checking the source, the current version, deletion and live documents under the same space lock as a commit. No migration is needed.
GET /api/v1/nodes/{id}/history and POST /api/v1/nodes/{id}/restore, resolved in the caller's own space. A stale expected_version answers 409 conflict. Covers permissions, paging, conflicts, restoring old and missing versions, and parallel restores in the API tests.
Updates the API table and the milestone note in the sync design, and adds a short section to the server guide.
A history page now reads versions through an index on (space_id, node, seq, id) in the order of the query, so the page size bounds the work. The migration adds the index on both databases. The restore note in the sync design no longer claims the engine prefers a newer live-document state: a restore is an ordinary new version and files with a live document are refused.
Adds the list, restore and cursor types of GET /trash and POST /trash/{id}/restore, with the same page limits as the history.
…und taken names A migration adds nodes.deleted_ms and nodes.purge_requested_ms and an index for the trash list, and dates old deletions at the migration. The commit path stamps the time when a node goes from live to deleted and clears it when the node comes back. restore_from_trash brings a deleted node and its deleted folders back in one transaction, merging into live folders of the same name and renaming around taken names with the engine's conflict pattern. request_purge only marks a node, so nothing is freed.
GET /api/v1/trash, POST /api/v1/trash/{id}/restore and DELETE /api/v1/trash/{id}, resolved in the caller's own space, and the [retention] trash_days setting (default 30, 1 to 3650) that sets purge_after_ms.
Adds the trash routes and rules to the sync design and a section with the new setting and the rollout note to the server guide.
…ders within the depth limit A file restored from the trash now gets a new current version with the content of its last one (no save record). A machine that sent a delete against the old version and never heard the answer can no longer delete the restored file again: the server refuses that delete as a version mismatch, and the engine already lets go of a node that came back at another version. The restore answer reports the new version. A folder that comes back below a folder that has since moved deep is refused as too deep, and nothing changes.
A folder has no version to tell a delete that went through and a later restore from one that never arrived. The engine now remembers the folder deletes it sent without hearing the answer, and when the next pull reports such a folder changed and live, lets it go instead of deleting it again. A folder the feed does not mention was never touched and its delete is sent again. Scenario tests cover a file and a folder restored after a lost answer, at its own path, under a conflict name, and merged into a live folder.
A restored file has a new version, and agents need the folder rule for the rare case of a folder restored after a delete whose answer was lost. Reports the new version in the restore answer.
…d are settled A folder deleted together with its file, whose answer was lost, and then restored alone, still had the file tracked when the pull judged it, so it was kept and deleted again once the file was let go. The pull now only notes which folders the feed reports live, and the decision is made in apply_remote after the files whose deletes went through are let go. Scenarios cover the non-empty folder restored alone, the folder and its file both restored, and only the file restored.
A folder this machine deleted with a lost answer and that came back from the trash was judged before the folders below it were settled. A tombstoned subfolder still tracked in it kept it, and the next push deleted the restored folder again. The folders of this machine's own delete are now settled deepest first: one the server lists as live, or no longer has and that is gone here too, is let go, and only then are the parents judged. A relist at the same epoch (the change feed answered resync required) did not note which of those folders the server lists as live, so a restored empty folder stayed tracked and was deleted again. The relist now notes them, so a folder the server has is never pushed as deleted from a stale request. Regressions cover d/sub/f and d/a/b/f with only the top folder, a middle folder, or only the deepest node restored, and a relist after the restore.
A migration adds manifest_chunks (manifest, ordinal, chunk) and manifest_index (chunk count, edges written, state). The index is written in the transaction that records a manifest, up to a budget of edges per transaction, and a longer manifest is finished in batches by index_complete, which also resumes an interrupted build and indexes manifests recorded before the index existed. An index is marked complete only after every edge was compared with its manifest, so wrong, swapped, missing, extra and collapsed edges are found. index_audit reports what the database alone can tell: orphan edges, index rows without a manifest, unknown dependencies, roots that name unrecorded manifests, and manifests that nothing names. Nothing is deleted except the derived edges of an index that does not match its manifest while it is still being built.
…ore the upload is acknowledged verify checks the database and the store against each other and reports problems and notices as text or JSON, changing nothing. index build indexes manifests recorded before the index existed, in bounded batches. PUT /manifests completes the manifest's index before answering.
The comparison of a manifest's edges started its pages after ordinal -1, so an edge at a negative ordinal was never seen and an index with such an edge could be marked complete. The comparison now starts at the lowest number there is and reports an edge at a negative ordinal as a mismatch of its own kind, so completion is never published over it. Starting an index again deleted all of a manifest's edges in one transaction, up to about a million rows, which broke the batch budget and blocked other writers. The reset now marks the index row as resetting (state 2) and deletes at most the index budget per transaction, lowest ordinal first; nothing is written or marked complete until the last edge is gone, and a run that stops between two batches is resumed by the next stage. The old behaviour of a complete row that is wrong being left to the checker is unchanged. Regressions cover negative ordinals (including the lowest possible), a reset across several batches that resumes on a new handle after a stop, and a build that resets a wrong index by itself.
…es it reads `verify` opened SQLite with create enabled and made missing storage directories, so a wrong path gave a new empty server and "no problems found". It now opens the database without creating it (`Meta::open_existing`: a missing file, a file with no Craft Cloud data, or a schema older than the program is an error, and nothing is created or upgraded) and requires the storage directory to exist. `index build` still uses the creating open. A successful read of a chunk or manifest object discarded its bytes, so seeded size corruption passed. Reading now reports the stored length (format byte included) and verify checks it against `blobs.stored_size` for chunks and manifests, the file size recorded for a manifest against the manifest, and the chunk lengths a manifest names against the length recorded for each chunk (one less than the stored size). Regressions cover a wrong database path, a wrong storage path, and seeded wrong sizes with and without reading chunk objects.
…y claim The verify schema test now snapshots the database and WAL straight from disk, with no SQLite connection opened between the setup checkpoint and the check. The concurrent index reset test holds the manifest_index row lock, proves both builders wait on it, and only then releases it. The i64::MAX edge test is renamed for what it pins; the cursor overflow guard is not reachable through the public API.
a18503e to
1f9610a
Compare
Decisions and assumptions for the Craft Cloud PRsHi, this collects the decisions that block upcoming work and the assumptions in the open PRs. Each item gives a proposed default; please confirm it or redirect us. We will use the proposed defaults for drafting unless you say otherwise. Destructive GC remains disabled until its safety contract is agreed and demonstrated. Decisions blocking whole slicesDestructive GC and retention
Backup and restore
Email and teams
Delayed saves
PR assumptions — confirm or redirectThe groups follow the proposed merge order; grouped choices include all listed PRs. Items without a separate owner question are still listed for scope confirmation.
Merge order and stacksSuggested order: independent #13, #16, #23 and #26 as convenient; #14 → #17 → #18 → #19 → #20; #22 after its admin/2FA base, then #27; #21 → #24 → #25, with #28 after #21 and #29 independently before inventory work. #30 (trash UI) is stacked on both #24 and #28. Exact PR bases and migration numbers should be checked again when landing. Current local branch comparisons show Already answeredNo owner comments or reviews answering these questions were found in the available PRs. The |
…nnot checkpoint a crashed WAL Closing the last read-write connection to a WAL-mode database checkpoints the WAL of a crashed process into the database file and removes it. Whether that finished before verify exited was a race, and a command that only reads must not make that write. Meta::open_existing now opens SQLite with mode=ro, which still reads the WAL's committed data. The tests run verify as a process of its own and compare the database file, the WAL and the storage tree after it has exited, with a row that only the WAL holds and with the -shm file gone.
|
Hi, following up on the questions above with a tested merge order for the open PRs. All of them were merged together locally, in the order below, and the result passes The merged result is on the fork as OrderEach line is one PR tip; a later tip already contains the PRs in brackets, so merging in this order means each PR lands on top of what it was built on.
#13, #16, #23 and #26 are independent and merged cleanly in any position. Steps 5 to 8 had conflicts; nothing was dropped to resolve them. The only real one is step 7 (trash against the admin stack, in the server, meta and docs). The resolution commits on Migration numbersEach branch numbers its migrations from
The final schema version is 14. Only the file names, the migration list and the schema-version expectations in tests change; the SQL itself stays as it is. #32 adds no migration. If you'd rather merge in a different order, tell me, and I'll renumber the affected branches before you merge them. Alternatively I can rebase each PR onto the previous one as they land. Known after merging
|
…he trash Restoring a folder from the trash now also brings back every node below it that is in the trash (deleted and not removed from the trash), however long before it was deleted, in the same transaction; the trash list shows such a node only through its folder; and removing a folder from the trash marks them too. A folder with more than 10,000 such nodes below it is refused whole as invalid for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first Restoring a folder from the trash now brings back all that is in the trash below it, so the lost-delete tests that restored a folder alone and expected its file to stay deleted take that file (or sub folder) out of the trash first.
Conflict in crates/meta/src/lib.rs: both new Meta fields (index_rows and trash_tree_cap) are kept.
… rest is listed Instead of refusing a folder with more than 10,000 items in the trash below it (which left it impossible to restore or remove from a list that hides what is below a trashed folder), a restore or removal now takes the folder and at most the cap of items below it, shallowest first. The rest stays in the trash and is listed on its own because its folder is live or removed, so a further call takes it. The per-row work under the write lock is bounded by the cap.
Another test helper in a later branch is called sorted; a distinct name keeps the trash tests compiling wherever they are merged.
Stacked on #24 (trash), which is stacked on #21. This branch contains their commits; this PR's own commits are
git log --no-merges trash..gc-reference-index; the merge commits after them bring in #24's update (a folder's restore and removal take what is in the trash below it). This is the first, non-destructive slice of garbage collection: nothing in it frees, deletes or claims anything. It adds a reference index and a read-onlyverify. The questions that block any deleting slice are listed at the end.What
manifest_chunks (manifest, ordinal, chunk): the chunk list of each manifest, copied from its object in the store.manifest_index: per manifest, how many chunks it names, how many edges are written (ordinals 0 up to that number, never a gap), and whether the list was checked against the manifest (state 1) or is still being built (state 0). A reset in progress is state 2.craft-cloud-server index buildindexes manifests recorded before the index existed (and any left unfinished). It reads each manifest from the store, checks it against its ID, resumes where it stopped, and may run beside the server.craft-cloud-server verify [--json] [--skip-chunk-reads]reads, reports and changes nothing:Store::get_sized(the stored length as read),Meta::open_existing,schema::check_current. Docs indocs/design/sync.mdanddocs/server.md.Known gap, stated in the docs
Manifests that no version or snapshot names are not garbage. A live document's binary assets (images and the like, uploaded through the agent's blob interface as a manifest of their own) are named only inside the document's state, which the server stores as opaque bytes. The database holds no reference to them. Until that is decided, anything that frees storage must treat those manifests and their chunks as in use.
verifylists them as a notice only.Questions for you before any deleting slice
I'd propose these slices after this one, each safe on its own and with deletion disabled until its proofs pass: upload reservations and claim-aware publication;
Store::list, an inventory and a GC dry run; a maintenance lease with durable claims and fencing; then thinning and trash purge; then account removal. Before any of those can delete, these need your decision:Testing
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --locked -- -D warningscargo test -p craft-cloud-meta -p craft-cloud-server -p craft-cloud-api -p craft-cloud-proto --lockedon SQLiteCRAFT_CLOUD_REQUIRE_BACKENDS=1(179 tests)cargo test -p craft-cloud-sync --locked; agent testsend_to_end,local_api,binary,networkcargo checkof proto, meta and serveri64::MIN), a reset in batches that resumes on a new handle.verifyon a clean server, on seeded problems of each kind, with wrong sizes (with and without--skip-chunk-reads), on a wrong database path (nothing created; an empty or foreign file is refused and left alone) and a wrong storage path (nothing created).i64::MAX) is reported; a reset whose edge count is an exact multiple of the budget ends and doesn't loop;verifyrefuses an older and a newer schema and leaves the database file and its WAL bytes unchanged; two concurrent builders on Postgres are held on the index row together (checked inpg_stat_activity) and end with an exact index and no duplicate ordinals. The last two were checked against a deliberately broken build.verifyopens an existing SQLite database read-only (mode=ro). A read-write open checkpointed a crashed process's WAL into the database file when its last connection closed: run as a separate process after a crash, the old open changed the database file in 9 of 12 runs. The new tests run the real binary as a child process after a simulated crash, with a row present only in the WAL (and once with-shmdeleted first). In every runverifyreports that row, and the database file, WAL and blob storage are byte-identical after the process exits (12 of 12 runs each).Not tested
verifyon S3 passed, and the server's tests ran on Postgres with S3 with no failures. A database created bymainupgraded from schema 6 to 14 on Postgres and on SQLite, andverifypassed; it then asks forindex build, as documented.)verifywith a Postgres URL pointing at a missing database.verifymay still leave an empty-waland a-shmfile behind (SQLite creates them when it opens the database, even read-only); the database and WAL bytes don't change.