Repository navigation
Trash: list, restore and mark deleted files and folders - #24
Open
Skyfuller1303 wants to merge 19 commits into
Open
Skyfuller1303 wants to merge 19 commits into
Skyfuller1303 wants to merge 19 commits into
Conversation
Adds the request and answer types of GET /nodes/{id}/history and POST /nodes/{id}/restore, the opaque history cursor with a single accepted spelling, and the paging limits (default 50, at most 200).
history() pages a file's versions newest first by (seq, id) with an exclusive cursor. restore_version() records a new version with the content of an old one as an ordinary update, checking the source, the current version, deletion and live documents under the same space lock as a commit. No migration is needed.
GET /api/v1/nodes/{id}/history and POST /api/v1/nodes/{id}/restore, resolved in the caller's own space. A stale expected_version answers 409 conflict. Covers permissions, paging, conflicts, restoring old and missing versions, and parallel restores in the API tests.
Updates the API table and the milestone note in the sync design, and adds a short section to the server guide.
A history page now reads versions through an index on (space_id, node, seq, id) in the order of the query, so the page size bounds the work. The migration adds the index on both databases. The restore note in the sync design no longer claims the engine prefers a newer live-document state: a restore is an ordinary new version and files with a live document are refused.
Adds the list, restore and cursor types of GET /trash and POST /trash/{id}/restore, with the same page limits as the history.
…und taken names A migration adds nodes.deleted_ms and nodes.purge_requested_ms and an index for the trash list, and dates old deletions at the migration. The commit path stamps the time when a node goes from live to deleted and clears it when the node comes back. restore_from_trash brings a deleted node and its deleted folders back in one transaction, merging into live folders of the same name and renaming around taken names with the engine's conflict pattern. request_purge only marks a node, so nothing is freed.
GET /api/v1/trash, POST /api/v1/trash/{id}/restore and DELETE /api/v1/trash/{id}, resolved in the caller's own space, and the [retention] trash_days setting (default 30, 1 to 3650) that sets purge_after_ms.
Adds the trash routes and rules to the sync design and a section with the new setting and the rollout note to the server guide.
…ders within the depth limit A file restored from the trash now gets a new current version with the content of its last one (no save record). A machine that sent a delete against the old version and never heard the answer can no longer delete the restored file again: the server refuses that delete as a version mismatch, and the engine already lets go of a node that came back at another version. The restore answer reports the new version. A folder that comes back below a folder that has since moved deep is refused as too deep, and nothing changes.
A folder has no version to tell a delete that went through and a later restore from one that never arrived. The engine now remembers the folder deletes it sent without hearing the answer, and when the next pull reports such a folder changed and live, lets it go instead of deleting it again. A folder the feed does not mention was never touched and its delete is sent again. Scenario tests cover a file and a folder restored after a lost answer, at its own path, under a conflict name, and merged into a live folder.
A restored file has a new version, and agents need the folder rule for the rare case of a folder restored after a delete whose answer was lost. Reports the new version in the restore answer.
…d are settled A folder deleted together with its file, whose answer was lost, and then restored alone, still had the file tracked when the pull judged it, so it was kept and deleted again once the file was let go. The pull now only notes which folders the feed reports live, and the decision is made in apply_remote after the files whose deletes went through are let go. Scenarios cover the non-empty folder restored alone, the folder and its file both restored, and only the file restored.
A folder this machine deleted with a lost answer and that came back from the trash was judged before the folders below it were settled. A tombstoned subfolder still tracked in it kept it, and the next push deleted the restored folder again. The folders of this machine's own delete are now settled deepest first: one the server lists as live, or no longer has and that is gone here too, is let go, and only then are the parents judged. A relist at the same epoch (the change feed answered resync required) did not note which of those folders the server lists as live, so a restored empty folder stayed tracked and was deleted again. The relist now notes them, so a folder the server has is never pushed as deleted from a stale request. Regressions cover d/sub/f and d/a/b/f with only the top folder, a middle folder, or only the deepest node restored, and a relist after the restore.
…he trash Restoring a folder from the trash now also brings back every node below it that is in the trash (deleted and not removed from the trash), however long before it was deleted, in the same transaction; the trash list shows such a node only through its folder; and removing a folder from the trash marks them too. A folder with more than 10,000 such nodes below it is refused whole as invalid for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first Restoring a folder from the trash now brings back all that is in the trash below it, so the lost-delete tests that restored a folder alone and expected its file to stay deleted take that file (or sub folder) out of the trash first.
… rest is listed Instead of refusing a folder with more than 10,000 items in the trash below it (which left it impossible to restore or remove from a list that hides what is below a trashed folder), a restore or removal now takes the folder and at most the cap of items below it, shallowest first. The rest stays in the trash and is listed on its own because its folder is live or removed, so a further call takes it. The per-row work under the write lock is bounded by the cap.
Another test helper in a later branch is called sorted; a distinct name keeps the trash tests compiling wherever they are merged.
Author
|
Update: restoring a folder from the trash now brings back everything in the trash below it, including a file that was deleted on its own inside it earlier. Before, the folder came back empty. Removing a folder from the trash marks its contents too, and the list shows only the folder. One call handles at most 10,000 nodes below the folder, shallowest first. Anything beyond that is listed on its own and can be handled with a further call. The details are in the description (new items under "What", assumptions 12 and 13, and the testing notes). The change is merged into #25, #30, #31, #32, #33 and #34. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21 (version history and restore): this branch contains #21's commit, so please review #21 first; the diff of this PR alone is
ec83bd7..trash. The history/retention design (CP-06) isn't agreed yet. The trash only lists, restores and marks; it frees nothing. The assumptions are listed below.What
GET /api/v1/trash?cursor&limit: the deleted nodes of your space that were not removed from the trash, newest deletion first by(deleted_ms, id), paged (default 50, max 200). A node whose folder is itself in the trash is not listed: the folder stands for it. Each item hasid,parent,name,is_dir,size,deleted_msandpurge_after_ms(deleted_msplus[retention] trash_days).POST /api/v1/trash/{id}/restore: brings a node back with its deleted folders, root-most first, keeping their IDs, in one transaction under the space's write lock and with one sequence number.name (conflict, <device>, <UTC date and minute>)like the engine's conflict copies, with, 2to, 100. No free name gives409 conflictand changes nothing.DELETE /api/v1/trash/{id}: only marks the node (purge_requested_ms, 204, idempotent). Versions, manifests, chunks and the tombstone stay; the change feed and the sequence number don't change. For a folder it marks what is in the trash below it too, with the same 10,000-per-call limit; the rest is then listed on its own.[retention] trash_days = 30(1 to 3650). It is shown to clients and removes nothing yet.nodes.deleted_ms,nodes.purge_requested_ms, and a partial index for the trash list. Nodes deleted before the migration are dated at the migration.Sync engine
A restore must not be undone by a delete that was in flight:
State::sent_dir_deletes). When the next pull or a relist shows such a folder live, it lets it go instead of deleting it again. Nested folders are settled deepest first, and a parent is judged only after its children.docs/server.mdsays to update agents with or soon after the server.Assumptions (please confirm or redirect)
deleted_msis set when a node goes from live to deleted (not on a second delete) and cleared when it comes back by any means.DELETE /trash/{id}is a mark only, until GC exists.Testing
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --locked -- -D warningscargo test -p craft-cloud-sync --locked; agent testsend_to_end,local_api,binary,network(turmoil),disk,run_loopcargo test -p craft-cloud-meta -p craft-cloud-server -p craft-cloud-api -p craft-cloud-proto --lockedon SQLiteCRAFT_CLOUD_REQUIRE_BACKENDS=1cargo checkof proto, meta and serverd/sub/fandd/a/b/fwith only the top, a middle or the deepest node restored; an empty folder restored before a relist. The fixes were reverted once to check that the regressions fail without them (4 of the new folder tests fail; the other 3 are guards).Folder contents (added later):
a_deleted_folder_comes_back_and_goes_with_what_was_deleted_in_it, and an agent end-to-end test in which two agents get a restored folder's files and subfolder back on disk.cargo test -p craft-cloud-sync(142 engine tests).Not tested
live_*,documents*and the like), and the full workspace test run.cargo check -p craft-cloud-syncon Rust 1.94 fails on main already (the locked egui crates need 1.95, and so does theif letguard incrates/sync/src/engine/apply.rs), unrelated to this PR; Raise the minimum Rust version to 1.95 and check it in CI #26 raises the declared version to 1.95.