Skip to content

Make a team space from a folder (after P1-6) - #71

Open
Skyfuller1303 wants to merge 387 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:team-8-from-folder
Open

Skyfuller1303 wants to merge 387 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:team-8-from-folder

Conversation

@Skyfuller1303

Copy link
Copy Markdown

Merge after #69 (P1-6). This branch is P1-6 plus the commits of this change. The diff of this PR alone: team-6-desktop...team-8-from-folder.

P1-8 of the collaboration design (the optional follow-up, Q2): "Make a team space from this folder". A folder of the personal folder becomes a new team space, and the files do not travel again: the team space takes their chunks from the personal space on the server.

What

  • Server: POST /api/v1/chunks/claim {from_space, ids} → {claimed, missing}.
    • The working space is the one the craft-space header names, as on every chunk route, and the caller needs editor there. The caller also needs at least read access to from_space (an archived source counts); without it the answer is 404 not_found, byte for byte what a space that does not exist gets, so the call reveals nothing the caller could not download anyway.
    • The server adds to the working space only the blobs from_space really has. claimed is what the working space has afterwards (added now, or held before), missing what neither space has; the client uploads those. At most 10,000 ids per request (400 invalid over), like chunks/missing.
    • The logical bytes of what was added count against the working space's limit, as an upload does, inside the same write transaction and with the same role re-check. A claim that would pass the limit is refused whole with 507 quota_exceeded and records nothing. A claim again changes nothing. No audit event (the design asks for none).
    • docs/design/sync.md has the route in its table.
  • Engine and agent transport: Transport::claim_chunks (default: claims nothing). Before a file's missing chunks are uploaded, the engine asks to claim them; what is not claimed is uploaded as before. The HTTP transport asks POST /chunks/claim only while it was told a source. A server that lacks the call (it answers 404 no such endpoint from outside the space layer, so without the space's echo), or refuses the source, is not an error: the files are uploaded, and a refused source is not asked again.
  • Agent: POST /v1/spaces/from-folder {folder, name, quota_bytes?, password?, …} and Status.from_folder. Creating a team space is an administrator's call (Q1), so the call carries the administrator's confirmation, and the account becomes the first owner. The order is the safety argument, and docs/agent.md and the module doc of from_folder.rs have it in full:
    1. Checking: the folder is inside the personal folder and is not it, is no link, is not empty, and holds no link or special file, no team folder, no Craft Cloud Shared folder, no other synced folder and no name the sync cannot carry. Nothing is written.
    2. Creating: the team space is made on the server. A wrong password, a taken name, an account that is no administrator and a server without team spaces are errors of the call, and nothing exists then.
    3. SettingUp: the team folder Craft Cloud Shared/<name> is set up and its sync turned on, with claims allowed from the personal space.
    4. Copying: each file is read once, chunked, and written into the team folder through a temporary file and a rename (the team pass never sees half a file); a file that changes under the copy is read again.
    5. Syncing: the team space's tree on the server is read until it holds every file with the manifest the copy was chunked into, and every folder. That, not the pass's own report, is "the team commit succeeded".
    6. Verifying: the old folder is read once more and must still be exactly what was copied; if not, the work goes back to copying (three rounds at most).
    7. Trashing: the old folder's files are deleted here one by one, each only while it is still as verified. The personal sync sees an ordinary deletion of a folder: it goes to the server's trash, where it can be restored, and the mass-delete guard sees it like any other (a big folder is held until approved, and the server keeps it until then). A file changed after the check stays, and so does a file the sync never carries (a .tmp file); both are counted in kept.
    • Review round (the commits after the first three): the team space is checked again before the first deletion and after every batch of 1,000 files or 60 s (listed, active, not archived, the account may change it, the server's tree holds every file still to be deleted); each file is renamed into a quarantine inside the personal folder's .craft-cloud, read once more and compared with the verified manifest before it is deleted, and its folders are checked for links of any kind (a crash's leftovers are put back at the next start); the work has a cancel flag that sign-out, an account switch or stopping the agent sets and waits for, so it writes nothing under another session; the stall counts bytes the team space stores, not only files committed (one big upload is no stall); the destination of a copy is compared by content; what stayed is told by reason (changed, held by a program but in the team space, not synced by design); a crashed creation adopts only a space that is the account's alone and newer than the call; the tree is read less often while nothing grows; a Trashing journal whose folder is gone is done.
    • Second review round: no move in the deletion ever replaces a file (a hard link, then the old name goes; a taken name refuses, a taken place gets <name>.cc-kept); everything a crash left in the quarantine is put back at every start and before every scan, whatever the journal says, and what cannot be put back makes the work Failed with the quarantine folder named; the old folder and every folder above it up to the personal folder are checked (identity, no link) before every batch and file; the call checks its session after every wait and writes nothing if it ended; per batch only a cheap direct check of the space (full tree check first and every ten minutes); the adoption window of a crashed creation is the server's clock, kept in the journal; only "not found" means the old folder is gone.
    • Third review round: the quarantine's marker sits beside the quarantine folder (a file called root.txt is quarantined like any other), is written atomically and validated on recovery (an unsound one leaves the quarantine alone and reported), and nothing is moved aside without it; no server time means none stored and nothing adopted; the stale "move them back yourself" message goes once the files are back; the session number is read first and the job registered under the lock; folders are identified by the sync's own file ID (NTFS file ID on Windows).
    • Codex items: a file goes into the quarantine by one rename into a folder made new for it (a file saved at the old place in the meantime is never lost); the copy never writes through a link in the team folder (its chain is recorded and checked before every write, and a new file lands by a link that fails on a taken name); a team file edited after the copy stays, the old version goes beside it as a conflict copy and the old file is not deleted.
    • Fourth review round: the copy replaces its own older team file only after moving it aside and reading it (a team edit made while a big file is read is kept, the new version goes beside it; a crash in between is put right at the next start); a volume without hard links falls back to a rename to an absent name instead of stranding changed files, and any other link error keeps the file; what was moved aside must be a regular file with the identity that was looked at; the quarantine keeps a file's own name under <slot>/<path> (names up to 255 characters work, nothing can clash with a person's folder); the copy's record is saved as it goes and a conflict copy is recorded, so a crash or a retry makes neither false team edits nor a second identical conflict copy; writes go through the resolved team folder; a quarantine that cannot be read is no longer taken for empty.
    • Fifth review round: team files an interrupted copy had moved aside are put back when the agent starts, before the team space's sync looks at its folder (and reported, and stopping the next copy, if one cannot be); the note beside such a file keeps the path as the folder spells it (NFD names), and a removal order keeps the note until the file is gone; the replacement of the job's own team file is looked at first (file ID, size, times and, on Unix, the inode change time), then the file is moved aside, the new copy lands at once, and the moved file is read before it is deleted (anything but the job's own content is kept beside the new copy as a conflict copy); an unmounted personal folder no longer turns a stranded-files message into "done"; the writes through the resolved team folder are tested against a link that is retargeted mid-copy.
    • Sixth review round: the start of the agent reads no file (it puts back only moved-aside team files whose place is free and reports the rest, which the next copy reads); a team file that is touched again and again is kept and counted as changed instead of failing the copy; the moved-aside files of a journal of another account are put back before that journal is dropped.
    • A failure before step 7 leaves the old folder as it was. The half-made team space is reported (Failed, failed_at, message, space), never deleted. A journal (from-folder.json in the data folder) records the step and the space: when the agent stops in the middle, the next start reports the work as Interrupted, and asking again for the same folder and name goes on with the same space without a password. Only one such work runs at a time.
    • The claim source is kept in memory only; after a restart the files are uploaded, which is slower and as safe.
  • App: Client::make_space_from_folder; an agent without the call is agent_too_old, a server without team spaces server_too_old.
  • Desktop: on a row of the home page's "On this computer" folder list, administrators get "Team space…". It opens a form (the name, the administrator's password and, with two-factor sign-in, a code) that says the files are copied into a new team space, that they don't travel again, and that the old folder is deleted from this computer only after the team space is confirmed to hold every file, and then sits in the trash on the server. The work shows as a notice on the home page, drawn from the agent's status alone (so it also shows after the app is reopened): the step, a bar for the steps that count files, and for a run that stopped where and why, that the old folder was not changed, and "Try again". Nobody but an administrator sees the button.

Checks

On Windows, through a memory-limited runner (16 GiB, never raised):

  • cargo fmt --all -- --check
  • cargo xtask licenses
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked --no-fail-fast against Postgres 17 and SeaweedFS 4.48 with CRAFT_CLOUD_REQUIRE_BACKENDS=1
  • cargo +1.95 check --workspace --all-targets --locked (MSRV)

All passed (see the latest check below). The agent's from-folder tests (43) pass alone on the last code. The desktop snapshots passed without the update flag; 24 are new and none of the existing ones changed.

New tests:

  • Server (SQLite and Postgres): the access matrix (viewer of the target, an editor of an archived target, an account only invited, removed, disabled, signed out), no access to the source answered like an unknown space, a viewer as the source, an archived source, a member removed from the source or the target, partial claims (some held, some in the source, some in neither), duplicates, idempotence and charging once, the quota (refused whole, nothing recorded, then what fits), the id limit and a body without a source, and that nothing flows from the target to the source.
  • Engine (in process): claimed chunks are not uploaded and are really in the space (another member downloads them), a source the account may not read falls back to the upload, an engine not told to claim uploads everything.
  • Agent (against a real server on a loopback port, with a counting layer in front of it): the whole thing with no chunk uploaded again (counted), a server that does not know the claim, the administrator's confirmation (wrong, none, a taken name, a member who is no administrator), nested and invalid folders (the personal folder, outside it, relative, missing, a file, empty, a Craft Cloud Shared folder inside, another synced folder inside, a link on Unix, a team folder), a failure at every step and a crash (agent killed) at every step, each leaving the old folder byte for byte and going on with the same space, a file changed under the copy, a file changed after the check (it stays), a taken folder name, and the mass-delete guard holding the deletion of a big folder.
  • Desktop: model tests for every state and event of the form and the notice (offered to administrators only, the confirmation and two-factor, the request sent and the password kept out of Debug, the answer, a refusal, an old agent and an old server, stale answers, every step's notice, done with files kept, failed and interrupted, try again without a password, dismiss), and snapshots in light, dark and the smallest window.

Latest check of this head

Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and CRAFT_CLOUD_REQUIRE_BACKENDS=1 under a 16 GiB memory limit at 0ffaf3d: cargo fmt --all -- --check and cargo clippy --workspace --all-targets --locked -- -D warnings pass; cargo test --workspace --locked --no-fail-fast: 2348 passed, 0 failed, 4 ignored.

Not in this PR

  • Making a team space from a folder for an account that is not an administrator (Q1: administrators only in P1, with a config switch later).
  • A call that forgets a team space the account lost, and the page for removing the old folder's copy on the server's trash; the server's trash page already has it.

history_bytes counted the manifest and exclusive chunks of an old version
that carries a save record, although an app saved that version on purpose.
A manifest named by a saved version of a live node is no longer history, and
a chunk shared with it stays out as well. A saved version of a deleted node
still counts as trash, since purging the node removes it. Tests cover both
backends; sync.md and server.md state the rule.
# Conflicts:
#	crates/meta/src/lib.rs
#	crates/proto/src/api.rs
#	docs/design/sync.md
… readers

The account override, else the server-wide cap. Behaviour is unchanged; the usage routes use the same rule next.
itrycodinganditsbad#33 (account usage in the desktop app) was built on itrycodinganditsbad#32 only, while this
branch also carries the admin stack. Merging it here resolves their
overlap once, so itrycodinganditsbad#33 and itrycodinganditsbad#34 can land in either order.

Conflicts kept both sides: the agent's local API routes and imports, the
desktop model (usage fields, events and effects next to the admin,
history and trash ones), the app and proto re-exports, and the overview
snapshots, which were redrawn.

Fixes needed after the textual merge:
- crates/api/src/api.rs: drop the second copy of AccountUsage,
  UsageEstimates and UsageEstimateReason that the merge duplicated.
- Tests: add `name_taken: None` to Error::Refused and ErrorAnswer values,
  and give admin_server_limited the `task` field of Server.
The locked egui 0.36 crates need rustc 1.95, so the image built from
rust:1.94-trixie fails once the workspace's rust-version is raised to
1.95: "rustc 1.94.1 is not supported by the following packages".
Rust 1.95 also builds the current workspace, so this does not depend on
the order in which the two changes land.
The server (device sessions) shows a device's client_kind, platform and
app_version in the device list, but the agent never sent them. Every
sign-in and join of the agent now says it is an agent, which operating
system it runs on and its own version, cut to the server's 32-byte limit.
A server-backed test checks that the device list shows them after both.
Conflicts in apps/craft-cloud-agent/src/http.rs: join_over_with keeps the second
factor fields and now also sends the device details. The first step of a
two-factor sign-in (login_step_over) sends them too; the server keeps them in
the challenge, so the second-factor request needs none.
…he trash

Restoring a folder from the trash now also brings back every node below it that
is in the trash (deleted and not removed from the trash), however long before it
was deleted, in the same transaction; the trash list shows such a node only
through its folder; and removing a folder from the trash marks them too. A
folder with more than 10,000 such nodes below it is refused whole as invalid
for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first

Restoring a folder from the trash now brings back all that is in the trash below
it, so the lost-delete tests that restored a folder alone and expected its file
to stay deleted take that file (or sub folder) out of the trash first.
Conflict in crates/meta/src/lib.rs: both new Meta fields (index_rows and
trash_tree_cap) are kept.
…never replaces

Putting a moved-aside file back and landing a copy used a hard link and gave up on any
error, so on FAT, exFAT or a network share every changed file stayed in the quarantine,
and a copy could not land. Only an error that says the volume has no hard links now falls
back to the move that renames to a name that was absent a moment before; any other error
of the link keeps the file where it is. A folder that took a file's place is moved the same
way. A test hook forces either error.
… and what was moved aside is checked

The quarantine put every file in a '<name>.ccq' folder, which made a name of 252 characters or
more impossible to take aside and could clash with a folder of that name. A file now goes to
'<quarantine>/<slot>/<path in the old folder>', the slot being a folder made new for it, so
the name keeps its length and nothing a person named is in the way; recovery drops the
slot and puts the rest back below the old folder.

What was moved aside must be a regular file with the identity of the one that was looked at;
a folder, a link or another file that took the name in that moment goes straight back and
counts as kept. A quarantine folder that cannot be read (a drive that is not there yet,
access denied) is no longer taken for empty: the message about files that could not be put
back stays and the work is not marked done.
… moves aside

The copy decided that a team file was the job's own before it read the old file, and renamed
the new copy over it at the end, so an edit made in the team folder (or downloaded by the
team pass) while a big file was read was overwritten. The old team file is now moved aside
first, with a note that says where it belongs, the moved file is checked, and the new copy
lands only if it still is the content the job wrote (by a link that fails on a taken name).
The moved file is read again unless it plainly is the file that was read at the start (same
file ID, size and time): the team pass fails on a file that is absent while it is read, so
the absence is kept to the two renames. Anything else is put back without replacing and the
new version goes beside it as a conflict copy. A crash in between is put right when the next
copy starts.

What the copy wrote is saved as it goes (about once a second, and when it ends), so a crash no
longer makes the job's own files look like team edits. A conflict copy is noted too, and a retry
does not make another identical one beside the same edit.

Everything is written through the resolved team folder, the one that was checked.
…the replacement is verified after the new copy lands

A copy killed between moving a team file aside and landing its new version left the place
empty until the person asked for the work again, and the team space could commit the deletion
meanwhile. The start of the agent now puts such files back (using the journal's team folder
and the space's record) before the team space's sync looks at the folder, and reports any
that cannot be put back; the next copy stops on them. The note beside a moved-aside file keeps
the path as the folder spells it (an NFD name stays NFD) and the record's key apart, a note
goes only after its file, and a file with no note is reported.

The replacement of the job's own team file is looked at where it stands first (file ID, size,
times and, on Unix, the inode change time), read only if that differs, then moved aside; the
new copy lands at once, so the place is empty for two renames, and only then is the moved file
read before it is deleted. Anything but the job's own content is kept beside the new copy as a
conflict copy and the old file stays.

A personal folder that is not found no longer lets the stranded-files message be rewritten or
the work be marked done. A link above the team folder that is retargeted during the copy is
tested not to lead the writes elsewhere.
…d of failing the copy, and a foreign journal's asides are put back

The start of the agent hashed moved-aside team files on the async loop, so a big one held the
start and Ctrl-C up. It now puts back only the files whose place is free and reports the
others, which the next copy reads (the job's own old copy is dropped, anything else is kept
next to the new file). A team file that is touched again and again while the replacement looks
at it stays as it is and counts as changed, where it used to fail the copy. The moved-aside
files of a journal of another account are put back before that journal is dropped.
…the copy

The wait for the team space ended the work when the team pass's last pass had skipped a file
with a reason it does not get over by itself. A pass that came to read a team file while the
copy had it moved aside (its replacement of its own older copy, or the recovery at the start)
reports "can't be read: <file>: not found", and under load that report was still the last
one when the wait looked, so a copy that was going well failed at Syncing. A file that was
not found when the pass came is now treated like one that is still changing: the next pass
sees it. A file that stays missing never reaches the server, and the wait still ends in its
stall.
The estimate was stored as JSON with the measured floats as decimal text, and serde_json without float_roundtrip
does not promise that a parsed f64 has the bits that were written. A compress speed therefore sometimes came back
one bit off, which made estimating_is_read_only_and_remembered fail now and then on both backends.

The estimate's saving and speed numbers are now rounded to hundredths when the estimate is built and stored as whole
hundredths, so the stored value reads back to the very same f64. An estimate stored as plain decimals by an earlier
server is still read. The CLI and admin output are unchanged apart from the speed in KiB/s being taken from the
rounded value. A unit test sends 10 000 pseudo-random speeds and percentages through write and parse.
With links not followed, inotify watches a link given as the synced folder and not the folder behind it, so a local change
in it raised no event and waited for the next periodic pass. The folder the link leads to is watched instead.
The watcher and the folder's roots were made once, when the space started. When a pass found that the folder leads somewhere
else (a link pointed at another folder), only the generation was raised, so changes in the new folder raised no event. The
space now makes its watcher and roots again on that event, under a new watcher generation.
…er has made it, not after a fixed time

The test logged out 800 ms after the call began. On a loaded machine the request that makes the space was not through by then,
so the token was gone before it and no space was made, which the test then reported as a failure. The test now waits until
the space exists.
…old folder is deleted

The old folder's files are deleted one by one. A personal pass that looks in the middle of that sees a few files gone, below the
guard's floor of ten, and deletes those from the server. On a slow machine the folder was deleted from the server in such
pieces and the pass never held 14 deletions. The test now pauses the sync for the deleting, so the pass that follows sees the
whole folder gone at once, as it would for a folder a person deleted.
…a short while

On Windows a folder cannot be renamed while a program has a file in it open. The agent's personal pass reads the project folder,
so the rename in the link-swap test was refused with access denied once in a while. The rename is tried again for up to ten
seconds, and fails with the error after that.
… short while

In some runs under load a whole-suite run of this target did not end: one test had finished its body and was stuck dropping its
temporary folders (a folder of the server's blobs, or an account's folder), with the process idle and, in one run, no agent
running at all. The agent's own shutdown had finished in every such run. The folders are now removed in a thread that is waited
for 20 seconds; a folder that is still not removed is left behind.
…, not taken for another account's

The journal was read with a single attempt, and any failure of the read counted as no journal. At the start of the agent that
deleted the file as another account's, and a call for the same folder and name then asked for a password and would have made a
second space. The journal is now looked at as one of: absent, unreadable, not a journal, another account's, this account's.
The read is tried again for a moment on Windows when another program holds the file. A journal that stays unreadable is kept
and the status says so; a call is refused with conflict until it can be read. A file that is no journal is set aside as
from-folder.json.broken-<time>; another account's is removed as before. A journal that says the work is running, found when no
work of the session runs, is told as interrupted by the call too, so a journal that could not be read at the start still resumes
the same space once it can.
…nd a lost creation answer keeps the journal

A failed journal write was only logged, so the server space was made and the copy and deletion went on with no record to
recover from. The write before the space is made now has to succeed (otherwise nothing is made and the call fails with a clear
message), and so does every checkpoint of the job: the job fails before the next step, and nothing is deleted without its
record.

When the request that makes the space went out and no answer came, the server may have made it. The journal is kept at
Creating (with the server time it holds), the status says it is not known, and the next try, or the next start, looks for the
space first and adopts it instead of failing on the taken name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant