Repository navigation
Make a team space from a folder (after P1-6) - #71
Open
Skyfuller1303 wants to merge 387 commits into
Open
Skyfuller1303 wants to merge 387 commits into
Skyfuller1303 wants to merge 387 commits into
Conversation
history_bytes counted the manifest and exclusive chunks of an old version that carries a save record, although an app saved that version on purpose. A manifest named by a saved version of a live node is no longer history, and a chunk shared with it stays out as well. A saved version of a deleted node still counts as trash, since purging the node removes it. Tests cover both backends; sync.md and server.md state the rule.
# Conflicts: # crates/meta/src/lib.rs # crates/proto/src/api.rs # docs/design/sync.md
… readers The account override, else the server-wide cap. Behaviour is unchanged; the usage routes use the same rule next.
itrycodinganditsbad#33 (account usage in the desktop app) was built on itrycodinganditsbad#32 only, while this branch also carries the admin stack. Merging it here resolves their overlap once, so itrycodinganditsbad#33 and itrycodinganditsbad#34 can land in either order. Conflicts kept both sides: the agent's local API routes and imports, the desktop model (usage fields, events and effects next to the admin, history and trash ones), the app and proto re-exports, and the overview snapshots, which were redrawn. Fixes needed after the textual merge: - crates/api/src/api.rs: drop the second copy of AccountUsage, UsageEstimates and UsageEstimateReason that the merge duplicated. - Tests: add `name_taken: None` to Error::Refused and ErrorAnswer values, and give admin_server_limited the `task` field of Server.
The locked egui 0.36 crates need rustc 1.95, so the image built from rust:1.94-trixie fails once the workspace's rust-version is raised to 1.95: "rustc 1.94.1 is not supported by the following packages". Rust 1.95 also builds the current workspace, so this does not depend on the order in which the two changes land.
The server (device sessions) shows a device's client_kind, platform and app_version in the device list, but the agent never sent them. Every sign-in and join of the agent now says it is an agent, which operating system it runs on and its own version, cut to the server's 32-byte limit. A server-backed test checks that the device list shows them after both.
Conflicts in apps/craft-cloud-agent/src/http.rs: join_over_with keeps the second factor fields and now also sends the device details. The first step of a two-factor sign-in (login_step_over) sends them too; the server keeps them in the challenge, so the second-factor request needs none.
…he trash Restoring a folder from the trash now also brings back every node below it that is in the trash (deleted and not removed from the trash), however long before it was deleted, in the same transaction; the trash list shows such a node only through its folder; and removing a folder from the trash marks them too. A folder with more than 10,000 such nodes below it is refused whole as invalid for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first Restoring a folder from the trash now brings back all that is in the trash below it, so the lost-delete tests that restored a folder alone and expected its file to stay deleted take that file (or sub folder) out of the trash first.
Conflict in crates/meta/src/lib.rs: both new Meta fields (index_rows and trash_tree_cap) are kept.
…never replaces Putting a moved-aside file back and landing a copy used a hard link and gave up on any error, so on FAT, exFAT or a network share every changed file stayed in the quarantine, and a copy could not land. Only an error that says the volume has no hard links now falls back to the move that renames to a name that was absent a moment before; any other error of the link keeps the file where it is. A folder that took a file's place is moved the same way. A test hook forces either error.
… and what was moved aside is checked The quarantine put every file in a '<name>.ccq' folder, which made a name of 252 characters or more impossible to take aside and could clash with a folder of that name. A file now goes to '<quarantine>/<slot>/<path in the old folder>', the slot being a folder made new for it, so the name keeps its length and nothing a person named is in the way; recovery drops the slot and puts the rest back below the old folder. What was moved aside must be a regular file with the identity of the one that was looked at; a folder, a link or another file that took the name in that moment goes straight back and counts as kept. A quarantine folder that cannot be read (a drive that is not there yet, access denied) is no longer taken for empty: the message about files that could not be put back stays and the work is not marked done.
… moves aside The copy decided that a team file was the job's own before it read the old file, and renamed the new copy over it at the end, so an edit made in the team folder (or downloaded by the team pass) while a big file was read was overwritten. The old team file is now moved aside first, with a note that says where it belongs, the moved file is checked, and the new copy lands only if it still is the content the job wrote (by a link that fails on a taken name). The moved file is read again unless it plainly is the file that was read at the start (same file ID, size and time): the team pass fails on a file that is absent while it is read, so the absence is kept to the two renames. Anything else is put back without replacing and the new version goes beside it as a conflict copy. A crash in between is put right when the next copy starts. What the copy wrote is saved as it goes (about once a second, and when it ends), so a crash no longer makes the job's own files look like team edits. A conflict copy is noted too, and a retry does not make another identical one beside the same edit. Everything is written through the resolved team folder, the one that was checked.
…e no-link fallback
…the replacement is verified after the new copy lands A copy killed between moving a team file aside and landing its new version left the place empty until the person asked for the work again, and the team space could commit the deletion meanwhile. The start of the agent now puts such files back (using the journal's team folder and the space's record) before the team space's sync looks at the folder, and reports any that cannot be put back; the next copy stops on them. The note beside a moved-aside file keeps the path as the folder spells it (an NFD name stays NFD) and the record's key apart, a note goes only after its file, and a file with no note is reported. The replacement of the job's own team file is looked at where it stands first (file ID, size, times and, on Unix, the inode change time), read only if that differs, then moved aside; the new copy lands at once, so the place is empty for two renames, and only then is the moved file read before it is deleted. Anything but the job's own content is kept beside the new copy as a conflict copy and the old file stays. A personal folder that is not found no longer lets the stranded-files message be rewritten or the work be marked done. A link above the team folder that is retargeted during the copy is tested not to lead the writes elsewhere.
…cement's look-move-land-read order
…d of failing the copy, and a foreign journal's asides are put back The start of the agent hashed moved-aside team files on the async loop, so a big one held the start and Ctrl-C up. It now puts back only the files whose place is free and reports the others, which the next copy reads (the job's own old copy is dropped, anything else is kept next to the new file). A team file that is touched again and again while the replacement looks at it stays as it is and counts as changed, where it used to fail the copy. The moved-aside files of a journal of another account are put back before that journal is dropped.
…the space's setting
…the copy The wait for the team space ended the work when the team pass's last pass had skipped a file with a reason it does not get over by itself. A pass that came to read a team file while the copy had it moved aside (its replacement of its own older copy, or the recovery at the start) reports "can't be read: <file>: not found", and under load that report was still the last one when the wait looked, so a copy that was going well failed at Syncing. A file that was not found when the pass came is now treated like one that is still changing: the next pass sees it. A file that stays missing never reaches the server, and the wait still ends in its stall.
The estimate was stored as JSON with the measured floats as decimal text, and serde_json without float_roundtrip does not promise that a parsed f64 has the bits that were written. A compress speed therefore sometimes came back one bit off, which made estimating_is_read_only_and_remembered fail now and then on both backends. The estimate's saving and speed numbers are now rounded to hundredths when the estimate is built and stored as whole hundredths, so the stored value reads back to the very same f64. An estimate stored as plain decimals by an earlier server is still read. The CLI and admin output are unchanged apart from the speed in KiB/s being taken from the rounded value. A unit test sends 10 000 pseudo-random speeds and percentages through write and parse.
With links not followed, inotify watches a link given as the synced folder and not the folder behind it, so a local change in it raised no event and waited for the next periodic pass. The folder the link leads to is watched instead.
The watcher and the folder's roots were made once, when the space started. When a pass found that the folder leads somewhere else (a link pointed at another folder), only the generation was raised, so changes in the new folder raised no event. The space now makes its watcher and roots again on that event, under a new watcher generation.
…does (macOS looks at the fonts)
…er has made it, not after a fixed time The test logged out 800 ms after the call began. On a loaded machine the request that makes the space was not through by then, so the token was gone before it and no space was made, which the test then reported as a failure. The test now waits until the space exists.
…old folder is deleted The old folder's files are deleted one by one. A personal pass that looks in the middle of that sees a few files gone, below the guard's floor of ten, and deletes those from the server. On a slow machine the folder was deleted from the server in such pieces and the pass never held 14 deletions. The test now pauses the sync for the deleting, so the pass that follows sees the whole folder gone at once, as it would for a folder a person deleted.
…a short while On Windows a folder cannot be renamed while a program has a file in it open. The agent's personal pass reads the project folder, so the rename in the link-swap test was refused with access denied once in a while. The rename is tried again for up to ten seconds, and fails with the error after that.
… short while In some runs under load a whole-suite run of this target did not end: one test had finished its body and was stuck dropping its temporary folders (a folder of the server's blobs, or an account's folder), with the process idle and, in one run, no agent running at all. The agent's own shutdown had finished in every such run. The folders are now removed in a thread that is waited for 20 seconds; a folder that is still not removed is left behind.
…, not taken for another account's The journal was read with a single attempt, and any failure of the read counted as no journal. At the start of the agent that deleted the file as another account's, and a call for the same folder and name then asked for a password and would have made a second space. The journal is now looked at as one of: absent, unreadable, not a journal, another account's, this account's. The read is tried again for a moment on Windows when another program holds the file. A journal that stays unreadable is kept and the status says so; a call is refused with conflict until it can be read. A file that is no journal is set aside as from-folder.json.broken-<time>; another account's is removed as before. A journal that says the work is running, found when no work of the session runs, is told as interrupted by the call too, so a journal that could not be read at the start still resumes the same space once it can.
…nd a lost creation answer keeps the journal A failed journal write was only logged, so the server space was made and the copy and deletion went on with no record to recover from. The write before the space is made now has to succeed (otherwise nothing is made and the call fails with a clear message), and so does every checkpoint of the job: the job fails before the next step, and nothing is deleted without its record. When the request that makes the space went out and no answer came, the server may have made it. The journal is kept at Creating (with the server time it holds), the status says it is not known, and the next try, or the next start, looks for the space first and adopts it instead of failing on the taken name.
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge after #69 (P1-6). This branch is P1-6 plus the commits of this change. The diff of this PR alone: team-6-desktop...team-8-from-folder.
P1-8 of the collaboration design (the optional follow-up, Q2): "Make a team space from this folder". A folder of the personal folder becomes a new team space, and the files do not travel again: the team space takes their chunks from the personal space on the server.
What
POST /api/v1/chunks/claim {from_space, ids}→{claimed, missing}.craft-spaceheader names, as on every chunk route, and the caller needseditorthere. The caller also needs at least read access tofrom_space(an archived source counts); without it the answer is404 not_found, byte for byte what a space that does not exist gets, so the call reveals nothing the caller could not download anyway.from_spacereally has.claimedis what the working space has afterwards (added now, or held before),missingwhat neither space has; the client uploads those. At most 10,000 ids per request (400 invalidover), likechunks/missing.507 quota_exceededand records nothing. A claim again changes nothing. No audit event (the design asks for none).docs/design/sync.mdhas the route in its table.Transport::claim_chunks(default: claims nothing). Before a file's missing chunks are uploaded, the engine asks to claim them; what is not claimed is uploaded as before. The HTTP transport asksPOST /chunks/claimonly while it was told a source. A server that lacks the call (it answers404 no such endpointfrom outside the space layer, so without the space's echo), or refuses the source, is not an error: the files are uploaded, and a refused source is not asked again.POST /v1/spaces/from-folder{folder, name, quota_bytes?, password?, …}andStatus.from_folder. Creating a team space is an administrator's call (Q1), so the call carries the administrator's confirmation, and the account becomes the first owner. The order is the safety argument, anddocs/agent.mdand the module doc offrom_folder.rshave it in full:Craft Cloud Sharedfolder, no other synced folder and no name the sync cannot carry. Nothing is written.Craft Cloud Shared/<name>is set up and its sync turned on, with claims allowed from the personal space..tmpfile); both are counted inkept..craft-cloud, read once more and compared with the verified manifest before it is deleted, and its folders are checked for links of any kind (a crash's leftovers are put back at the next start); the work has a cancel flag that sign-out, an account switch or stopping the agent sets and waits for, so it writes nothing under another session; the stall counts bytes the team space stores, not only files committed (one big upload is no stall); the destination of a copy is compared by content; what stayed is told by reason (changed, held by a program but in the team space, not synced by design); a crashed creation adopts only a space that is the account's alone and newer than the call; the tree is read less often while nothing grows; a Trashing journal whose folder is gone is done.<name>.cc-kept); everything a crash left in the quarantine is put back at every start and before every scan, whatever the journal says, and what cannot be put back makes the workFailedwith the quarantine folder named; the old folder and every folder above it up to the personal folder are checked (identity, no link) before every batch and file; the call checks its session after every wait and writes nothing if it ended; per batch only a cheap direct check of the space (full tree check first and every ten minutes); the adoption window of a crashed creation is the server's clock, kept in the journal; only "not found" means the old folder is gone.root.txtis quarantined like any other), is written atomically and validated on recovery (an unsound one leaves the quarantine alone and reported), and nothing is moved aside without it; no server time means none stored and nothing adopted; the stale "move them back yourself" message goes once the files are back; the session number is read first and the job registered under the lock; folders are identified by the sync's own file ID (NTFS file ID on Windows).<slot>/<path>(names up to 255 characters work, nothing can clash with a person's folder); the copy's record is saved as it goes and a conflict copy is recorded, so a crash or a retry makes neither false team edits nor a second identical conflict copy; writes go through the resolved team folder; a quarantine that cannot be read is no longer taken for empty.Failed,failed_at,message,space), never deleted. A journal (from-folder.jsonin the data folder) records the step and the space: when the agent stops in the middle, the next start reports the work asInterrupted, and asking again for the same folder and name goes on with the same space without a password. Only one such work runs at a time.Client::make_space_from_folder; an agent without the call isagent_too_old, a server without team spacesserver_too_old.Checks
On Windows, through a memory-limited runner (16 GiB, never raised):
cargo fmt --all -- --checkcargo xtask licensescargo clippy --workspace --all-targets --locked -- -D warningscargo test --workspace --locked --no-fail-fastagainst Postgres 17 and SeaweedFS 4.48 withCRAFT_CLOUD_REQUIRE_BACKENDS=1cargo +1.95 check --workspace --all-targets --locked(MSRV)All passed (see the latest check below). The agent's from-folder tests (43) pass alone on the last code. The desktop snapshots passed without the update flag; 24 are new and none of the existing ones changed.
New tests:
Craft Cloud Sharedfolder inside, another synced folder inside, a link on Unix, a team folder), a failure at every step and a crash (agent killed) at every step, each leaving the old folder byte for byte and going on with the same space, a file changed under the copy, a file changed after the check (it stays), a taken folder name, and the mass-delete guard holding the deletion of a big folder.Debug, the answer, a refusal, an old agent and an old server, stale answers, every step's notice, done with files kept, failed and interrupted, try again without a password, dismiss), and snapshots in light, dark and the smallest window.Latest check of this head
Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and
CRAFT_CLOUD_REQUIRE_BACKENDS=1under a 16 GiB memory limit at0ffaf3d:cargo fmt --all -- --checkandcargo clippy --workspace --all-targets --locked -- -D warningspass;cargo test --workspace --locked --no-fail-fast: 2348 passed, 0 failed, 4 ignored.Not in this PR