Skip to content

Desktop: conflict copies, folder usage, compression and server health (after the extras PR and the PRs it merges) - #56

Open
Skyfuller1303 wants to merge 302 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:desktop-insights
Open

Skyfuller1303 wants to merge 302 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:desktop-insights

Conversation

@Skyfuller1303

Copy link
Copy Markdown

Merge after #55 (extras), #51 (conflicts), #53 (server health, which carries the server-full PR) and #54 (compression admin). The diff of this PR alone: desktop-extras...desktop-insights. The branch is the extras branch plus a merge of each of the others, then its own commits; the compare also lists the merged commits.

The agent and the server now report things the desktop app never showed: the conflict copies in the synced folder, how much the files take on this computer, how the server compresses what it stores, and how healthy the server is. This PR shows them. It is desktop only (apps/craft-cloud-desktop); the merges bring in the endpoints.

What

  • Conflict copies (Files tab, new segment "Conflict copies" beside "Recent files"):
    • one row per copy: name, "Looks like a conflict copy of ", computer, time, size;
    • Open works as it does for the recent files: the path is joined part by part, only the kinds of file on the Files page's document allow-list are opened, the disk is looked at again when it is clicked and the path that check found is the one opened; Show original opens the folder the original is in (the synced folder for a top-level file), enabled when the original is here, also when it is a kind of file that is not opened from the page. The folder gets only the checks of a path (no escape, no bundle suffix such as .app), so a folder called Photos.2024 is shown;
    • the hint says a file you named that way yourself is listed too, and that the list is the last finished sync pass;
    • empty state, error state, and "Showing the newest N" when the agent says truncated;
    • loaded off the UI thread with the recents (one Refresh asks for both), with its own ticket so a stale answer is dropped.
  • Overview tag: when the agent lists conflict copies, the Sync cell gets a link "3 conflict copies ↗" that opens the Files tab on the copies. Asked at sign-in and with Refresh; an error shows nothing.
  • Folder usage ("On this computer ↗" in the Storage cell opens a section under the cells): the top folders by local logical bytes and the eight kinds, each with a share bar, size and file count. The folders beyond the first eight and the agent's own "other" are one "Other (n folders)" row. It says plainly these are local sizes, not what the server stores. Asked when the section opens and with Refresh. The answer is taken wherever the overview is, also under the Security page, adding a computer, the admin pages or the activity, and an overview that comes back from under another screen no longer spins.
  • Compression (admin, Storage part): algorithm, level in use (and whether from the config or set by an admin), logical size, stored size of the known part, saving, known share, the last estimate with its date and the per-level table. The level is a menu from 1 to 19 and "Change level"; the change goes through the same password (and two-factor code) confirmation as the other admin changes, and the server audits it as admin.compression_changed (the activity label was already there). With compression = "none" in the config the panel says "Set by the server config" and has no control.
  • Server health (admin, Accounts part): uptime from started_ms, "Database ok", "Store ok · checked …", and "Refusing uploads" in the danger tone. A field an older server does not send shows nothing, and with none of them there is no row.
  • Server full on the user side: the agent pauses with its own text when the server answers server_full ("The server is full; the admin needs to free space ..."). The overview and the footer now say "The server is full. Uploads wait until its admin frees space" and keep the agent's text under Details, instead of the first sentence of a message that reads like the account's quota. The match is on the pause's reason_code (server_full; added to the agent's status in the server-full PR) and, for an agent that sends none, on the text (one shared constant); a different code, quota_exceeded included, is never taken for it.

Merges

fix/agent-conflicts, fix/server-health and fix/compression-admin merge without conflicts in the files I touch. fix/compression-admin conflicts with fix/server-health in the server (config.rs, state.rs, api.rs, tests/api.rs): both add [storage] settings, AppState fields and tests next to each other, and I kept both sides. The compression migration is 0015_physical_size here; no number clashed.

Checks

On Windows, through the memory-limited build queue (16 GiB job limit, two build jobs):

  • cargo fmt --all -- --check
  • cargo xtask licenses
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked --no-fail-fast on the merged tree before any UI change (without the desktop crate, which I had not touched yet) and again at the end for the whole workspace, desktop included (peak 2.4 GiB; live_documents fit in 16 GiB here)
  • cargo +1.95 check --workspace --all-targets --locked (MSRV)

All passed. New tests: the model (conflict copies load with the recents and ignore stale answers, Open and Show original only for what is here and only through the safe resolver, the overview count and its link, folder sizes load when the section opens), the folder rows ("Other"), the compression change end to end against the fake agent (range 1 to 19, wrong password, password sent, answer applied), the uptime wording, the full-server line, and kittest snapshots for each new screen in light and dark and at the smallest window size. 30 existing snapshots changed because the Files page has its new segment and the Storage cell its link; the others stayed within the snapshot tolerance.

Open points

  • "Show original" opens the original's folder, not the file; revealing and selecting the file needs a platform-specific call the app does not have.

Latest check of this head

Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and CRAFT_CLOUD_REQUIRE_BACKENDS=1 under a 16 GiB memory limit at dd50688: cargo fmt --all -- --check and cargo clippy --workspace --all-targets --locked -- -D warnings pass; cargo test --workspace --locked --no-fail-fast: 2042 passed, 0 failed, 4 ignored.

Adds /v1/admin/... to the local API, one route for each route of the server's admin API. The agent checks no role and
keeps no confirmation: each body is read as the shared type (so its Debug output hides the admin's password), limited
to 8 KiB, and written to the server with the device's token. The server's answer is the answer, with its HTTP status,
error code and message, and Retry-After for a 429. A 403 therefore stays a 403; the server's 401, which on a signed-in
route means this device was revoked, is told as every other call tells it (409 needs_you), never as a local 401.

Tests run the agent against a real server on loopback: an admin's reads and changes, a missing and a wrong confirmation,
the last admin, the server's rate limit, a member's 403, a revoked device, a signed-out agent and oversized bodies, and
that neither the password nor a code is written to the data folder or printed.
An account the server answers the admin counters to gets an Administration button: a paged user list with search, a
new account (name and role; its quota is set afterwards on the account's page, as a change of its own), one account's
role, quota, disable, enable, removal, invite, password reset code and computers, and the codes shown once.

Every change asks for the admin's password in a confirmation page and sends it in the call's body; the model clears
it as soon as it is sent, and its Debug output hides it. The confirmation is one type, so a second factor's code can
be added to it without touching the pages, events or calls. The server decides every permission: its refusals
(wrong_password, last_admin, account_disabled, forbidden, rate_limited with its wait) are shown in plain words.

Adds the model tests, kittest interaction tests and snapshots of the new screens (the existing ones are unchanged).
Describes the /v1/admin routes of the agent's local API in docs/agent.md (how the server's status and codes pass
through, and that the confirmation is never kept), mentions them in the local API's call list, and lists the new
modules in the workspace map.
…oke any open code

Review fixes for the admin pages.

The password field's undo history, which egui keeps in memory with the text as it was typed, is now dropped every frame
unless a confirmation is open with something typed in it: once a password was sent, or the page was cancelled or left,
Undo can't bring it back. The field has a stable id. A kittest test types a password, sends it, cancels, leaves and
presses Ctrl+Z each time, and checks that the field stays empty and that egui keeps no history. egui keeps this state in
memory only; the app doesn't enable eframe's persistence and the undo history is not serialized.

The numbers that match an answer to its call (the user list, a user's devices) never restart, so a late answer from
before the pages were opened again matches nothing. The admin check carries the number of the sign-in it was made for,
so a late answer for an earlier account can't change the current one's access.

"Revoke open code" is offered always, since the list's invite flag doesn't cover password reset codes. A revoke that
finds nothing open says so and looks at the list again.
craft-cloud-api gets the closed list of audit event names with a check for an audit filter (a name, or a prefix ending in a
dot), the local API's paths for the account, its activity and the admin audit and its export, and the serde, default and
redaction tests of the admin bodies (with the second factor and the reset flag) and of the audit bodies, kept with the
types.

craft-cloud-app gets Client::account, account_activity and admin_audit, and admin_audit_export, which reads the agent's
chunked NDJSON stream line by line into a writer: one line and a small buffer at a time, an event checked as an event, and
the export counted as whole only with the server's complete line and the number of events it says. A stream that ends without
it, breaks, or holds a line that doesn't read is an error. A truncated line is a good export that says so. The body reader
that takes the chunks off is tested against cut-off and malformed streams. The reset code call takes the whole
AdminPasswordReset and a call for reset-two-factor is added.
…ocal API

Adds GET /v1/account (the signed-in account's role and two-factor state), GET /v1/account/activity (any account's own
events), GET /v1/admin/audit (filtered, paged, newest first) and GET /v1/admin/audit/export, which passes the server's NDJSON
through chunk by chunk without ever holding more than the chunk in hand; a server stream that breaks ends the local one without
its last chunk, so the client sees a broken export. Cursors are passed as they came, limited in length and characters; a user
must be an account ID, an action one of the closed list or a prefix of it, and times and limits are bounded, all before
anything is sent. The server's statuses and codes (403 forbidden for a member, 429 with Retry-After) pass through as
everywhere, and the account's own activity reads none of the admin's filters.

Tests run against a real server on loopback: an admin with two-factor sign-in confirming changes with a code or a recovery
code (missing and wrong ones are two_factor_invalid and never sign the agent out, too many are stopped with the server's 429), a
reset code with reset_two_factor, resetting an account's two-factor sign-in, and the audit list, filters, own activity,
member refusals and a whole and a truncated export. The app's tests read an export over a real connection: whole, cut off
before the terminating chunk, cut in the middle of a chunk, without a last line, and refused.
…off, role from the account

The admin confirmation takes a code beside the password when the account has two-factor sign-in (the page asks for it once the
server says two_factor_invalid, too): one field, six digits are a code from the app, anything else a recovery code. Password
and code live in Confirmation, which clears both when they are sent; both fields have stable ids and go through the Security
page's forgetting of secrets in egui's undo history (one mechanism, with a test for the code field next to the password's).
The server's wrong or locked code and its rate limit are said as the Security page says them, with the wait.

Password reset codes go through AdminPasswordReset end to end, so a reset code can turn two-factor off when it is used, and
a change of its own turns an account's two-factor sign-in off and signs its computers out. An account's page shows whether
it has two-factor sign-in.

The app learns whether the account is an admin from GET /v1/account (the server's role, and whether the account has
two-factor sign-in) instead of probing an admin route; the server's counters are asked for when the admin pages open. The
Refusal and error shapes are the two-factor ones: a refusal keeps its challenge and its wait from the JSON or the header.
…the export to a file

The admin pages get an audit log: events newest first as words with the typed details the server writes, paged with the
server's cursors given back as they came, filtered by an event name or group from the closed list (never anything else) and by
an account (opened from its page), with a refresh. Its export goes to a file the person picks: the app writes the agent's
stream to <file>.part and moves it into place only when the stream ended with the server's complete line, so a broken stream
or a full disk fails the export, removes the part file and leaves what was at the path; a stream the server cut at its limit
is a good file the page says is not the whole log, never a plain success.

Every signed-in account gets an Activity page with its own events (the signed-in view's Activity button), whatever its role.

Late answers match nothing (the numbers that tell them apart never restart). The Security, Activity and Administration
buttons are all in the signed-in view. New screens are drawn by kittest: the audit log (light and dark), an export in progress,
truncated and failed, the activity page, an account with two-factor sign-in and the confirmation that asks for a code. The
signed-in snapshots and the admin ones whose screens gained a button are regenerated.
…nd factor in the admin confirmation

Adds GET /v1/account, the account's activity, the audit list and its streamed export, and reset-two-factor to the table of the
local API's admin calls, and describes the second factor in the confirmation, a reset code that turns two-factor off, and why
and how an export is counted as whole.
…their export, read the end of a chunked body, show who and what

The audit export is written to a file created for it alone in the destination's folder (it must not exist, so nothing is
truncated and no link is followed), moved over the chosen path only when the stream ended properly, and removed on any failure;
a file that happens to be called <path>.part, and an existing destination, are never touched by a failed export.

The picker's answer and the export's answer carry the number of the export that asked for them, so after leaving and opening the
pages again, or starting a second export, a late answer of the first finishes nothing of the second.

The reader of a chunked body now needs the last line break after the last chunk and its trailers (which are bounded): a
connection that ends after "0\r\n" or inside a trailer is a cut-off body, not a complete one, which matters because an export
counts as whole only when its body does. Tested with unit cases and over a real connection.

Each audit event line now says who did it (the account's short ID, "the command line", or "unknown" for a sign-in nobody proved
anything for) and what it was about (account or computer, with the computer an act names in its details); the whole IDs are on
hover. The audit snapshots that show events are regenerated.
Adds the request and answer types of GET /nodes/{id}/history and POST /nodes/{id}/restore, the opaque history cursor with a single accepted spelling, and the paging limits (default 50, at most 200).
history() pages a file's versions newest first by (seq, id) with an exclusive cursor. restore_version() records a new version with the content of an old one as an ordinary update, checking the source, the current version, deletion and live documents under the same space lock as a commit. No migration is needed.
GET /api/v1/nodes/{id}/history and POST /api/v1/nodes/{id}/restore, resolved in the caller's own space. A stale expected_version answers 409 conflict. Covers permissions, paging, conflicts, restoring old and missing versions, and parallel restores in the API tests.
Updates the API table and the milestone note in the sync design, and adds a short section to the server guide.
A history page now reads versions through an index on (space_id, node, seq, id) in the order of the query, so the page size bounds the work. The migration adds the index on both databases. The restore note in the sync design no longer claims the engine prefers a newer live-document state: a restore is an ordinary new version and files with a live document are refused.
Adds the list, restore and cursor types of GET /trash and POST /trash/{id}/restore, with the same page limits as the history.
…und taken names

A migration adds nodes.deleted_ms and nodes.purge_requested_ms and an index for the trash list, and dates old deletions at the migration. The commit path stamps the time when a node goes from live to deleted and clears it when the node comes back. restore_from_trash brings a deleted node and its deleted folders back in one transaction, merging into live folders of the same name and renaming around taken names with the engine's conflict pattern. request_purge only marks a node, so nothing is freed.
GET /api/v1/trash, POST /api/v1/trash/{id}/restore and DELETE /api/v1/trash/{id}, resolved in the caller's own space, and the [retention] trash_days setting (default 30, 1 to 3650) that sets purge_after_ms.
Adds the trash routes and rules to the sync design and a section with the new setting and the rollout note to the server guide.
Migration 0011 adds web_sessions, which keeps only the BLAKE3 hash of a session token. A session is made by a password
sign-in of an administrator (with a second-factor challenge of its own kind when the account has one), is valid while
the account's auth_gen is the one it was made under, and ends by idle time, absolute lifetime or logout. Expired rows are
pruned in bounded batches. AdminActor now says whether a device or a browser session acts, and the admin changes check
under the account's lock that the session is still valid at the time of the change (not logged out, same auth_gen,
inside its idle and absolute lifetimes). AdminUser carries removed_ms. Tests run on SQLite and Postgres.
Adds the opt-in admin pages under /web (config section web_admin, off by default; it needs behind_proxy and an https
public_url without a path): sign-in with the JSON sign-in's password checks and limits (limiters of their own) and the
server's two-factor challenge and lock, the __Host- session cookie (Secure, HttpOnly, SameSite=Strict), CSRF tokens
derived from the session, an Origin check on every post, a strict CSP and the other security headers on every answer
(the /web/ redirect too), logout, and pages for the overview, users (search, create, role and quota, disable, enable,
remove, invites and reset codes shown once, reset of two-factor sign-in, devices) and the audit log with its NDJSON
download. Each change is handed to the same store call as the JSON admin route with the same confirmation. The JSON
admin routes and the pages refuse a request with both a bearer token and the cookie. A sign-in for an account without a
password (an open invite) spends the same password check as an unknown name, in the pages and in POST /auth/login.
Expired sessions are deleted a few at a time at sign-in and by the new command web-sessions prune.
Replaces "no web UI in v1" for administrators in the design document (sessions, sign-in, CSRF and origin, headers,
statuses, cleanup) and adds the operator's guide to the admin pages (config, TLS proxy, upgrade, cleaning up).
…ders within the depth limit

A file restored from the trash now gets a new current version with the content of its last one (no save record). A machine that sent a delete against the old version and never heard the answer can no longer delete the restored file again: the server refuses that delete as a version mismatch, and the engine already lets go of a node that came back at another version. The restore answer reports the new version. A folder that comes back below a folder that has since moved deep is refused as too deep, and nothing changes.
A folder has no version to tell a delete that went through and a later restore from one that never arrived. The engine now remembers the folder deletes it sent without hearing the answer, and when the next pull reports such a folder changed and live, lets it go instead of deleting it again. A folder the feed does not mention was never touched and its delete is sent again. Scenario tests cover a file and a folder restored after a lost answer, at its own path, under a conflict name, and merged into a live folder.
A restored file has a new version, and agents need the folder rule for the rare case of a folder restored after a delete whose answer was lost. Reports the new version in the restore answer.
…d are settled

A folder deleted together with its file, whose answer was lost, and then restored alone, still had the file tracked when the pull judged it, so it was kept and deleted again once the file was let go. The pull now only notes which folders the feed reports live, and the decision is made in apply_remote after the files whose deletes went through are let go. Scenarios cover the non-empty folder restored alone, the folder and its file both restored, and only the file restored.
A folder this machine deleted with a lost answer and that came back from the trash was judged before the folders below it were settled. A tombstoned subfolder still tracked in it kept it, and the next push deleted the restored folder again. The folders of this machine's own delete are now settled deepest first: one the server lists as live, or no longer has and that is gone here too, is let go, and only then are the parents judged.

A relist at the same epoch (the change feed answered resync required) did not note which of those folders the server lists as live, so a restored empty folder stayed tracked and was deleted again. The relist now notes them, so a folder the server has is never pushed as deleted from a stale request.

Regressions cover d/sub/f and d/a/b/f with only the top folder, a middle folder, or only the deepest node restored, and a relist after the restore.
A migration adds manifest_chunks (manifest, ordinal, chunk) and manifest_index (chunk count, edges written, state). The index is written in the transaction that records a manifest, up to a budget of edges per transaction, and a longer manifest is finished in batches by index_complete, which also resumes an interrupted build and indexes manifests recorded before the index existed. An index is marked complete only after every edge was compared with its manifest, so wrong, swapped, missing, extra and collapsed edges are found. index_audit reports what the database alone can tell: orphan edges, index rows without a manifest, unknown dependencies, roots that name unrecorded manifests, and manifests that nothing names. Nothing is deleted except the derived edges of an index that does not match its manifest while it is still being built.
…re dropped

A tab, another account, the list, the audit log, Sign out or Quit asked for while the account page holds a typed role or quota that was not saved now shows the page's inline question, "Discard the unsaved changes?", with Discard and Keep editing. The question's cancel button is a parameter of the confirmation panel now.
Ctrl+1 to Ctrl+9 go to the visible tabs in the header's order (Cmd on macOS), Ctrl+R does what the page's Refresh does, and Ctrl+F puts the cursor in the page's search field. A locked page keeps its tabs locked. A tab's hover text names its shortcut. Esc still closes a confirmation or a menu.
The tray's menu starts with a disabled line in the words of the window's footer, such as "Online · Up to date", and the line follows the state on Windows, macOS and Linux (the D-Bus menu sends a layout update when it changes).
…ivity

Both tables get a menu of the kinds of event (sign-in, computers, account, admin), a menu of the computers the loaded events name (by the names the app knows) and a search of the event's words. They narrow the events already loaded and ask the server for nothing, which the page says under the controls: "Filters the loaded events".
The scale goes from 80 % to 160 % in steps of 10 %. It is kept by the model for the session: the app has no settings of its own and eframe's storage is not built in, so it starts at 100 % again.
Every name of the API's list already has its words; the compression change, which the server writes once its setting is in, now reads "Compression level changed" instead of the raw name. A test keeps the list and the labels together.
The audit log inside the admin pages' switcher had Refresh on a row of its own. It sits on the switcher's row now, as it does on the Accounts and Storage parts.
… branches changed

The Computers and admin branches drew their narrow-window snapshots before the Files tab existed. The merged stack has it, so the tab row of the smallest-window snapshots differs; nothing else on those pages changed.
The question "Discard the unsaved changes?" was asked for a tab, another
account, Sign out and Quit, but not when the window's close button ends the
app (no tray) or when the Linux tray process's Exit does. Both are now held
back like the others: the event stays pending, the window is shown for the
question and nothing is carried out until Discard.
The menu's host takes the first underscore of a dbusmenu label as the mark
of a keyboard shortcut and drops it. The labels (the state line and the
entries) now send `_` as `__`.
…th and a full server

Files tab: a "Conflict copies" segment lists the copies the agent finds in the synced
folder (name, original, computer, time, size) with Open and Show original, an empty and
an error state and "Showing the newest N" when the list is truncated; the overview links
to it with "n conflict copies". The Storage cell opens "On this computer": the top
folders and the eight kinds by local logical size.

Admin: the Storage part shows the compression (algorithm, level, logical and stored
size, saving, known share, the last estimate) and changes the level 1 to 19 behind the
usual password confirmation; the Accounts part shows the server's health (uptime,
database, store, refusing uploads), and nothing for fields an older server leaves out.

A pause because the server is full now says so instead of reading like the account's
quota. Snapshots of the changed and new screens.
# Conflicts:
#	apps/craft-cloud-desktop/src/ui/overview.rs
#	apps/craft-cloud-server/src/state.rs
#	apps/craft-cloud-server/tests/api.rs
…other screen

The answer to the folder sizes was dropped unless the overview was the
current screen, so a request still under way when the Security page, adding a
computer, the admin pages or the activity was opened never ended and its
spinner stayed on. The answer is matched through the overview wherever it
is, as the conflict copies' answer is, and an overview that comes back from
under another screen no longer waits for a request.
…nal's folder is checked as a folder

Open on a conflict copy looks at the disk again when it is clicked and opens
the path that check found, as Open on a recent file does, so a copy swapped
for a link in between is not followed. Show original applied the program
check to the original's folder name, which refused a folder called
`Photos.2024` and would refuse any folder under the document allow-list; it
now applies the checks of a path only (no escape, no bundle suffix), and
shows the folder of an original that is not a kind of file opened from the
page.
…econd

The paused run state now carries `reason_code` (the agent's server-full
change), so the overview and the footer take a pause for the server being full
when the code is `server_full`, and not for any other code, `quota_exceeded`
included. For an agent that sends no code the words are matched as before,
now through one constant.
The recent files' rows have a Folder button beside Open, in the lists of
the Files tab on the stacked branch.
The estimate was stored as JSON with the measured floats as decimal text, and serde_json without float_roundtrip
does not promise that a parsed f64 has the bits that were written. A compress speed therefore sometimes came back
one bit off, which made estimating_is_read_only_and_remembered fail now and then on both backends.

The estimate's saving and speed numbers are now rounded to hundredths when the estimate is built and stored as whole
hundredths, so the stored value reads back to the very same f64. An estimate stored as plain decimals by an earlier
server is still read. The CLI and admin output are unchanged apart from the speed in KiB/s being taken from the
rounded value. A unit test sends 10 000 pseudo-random speeds and percentages through write and parse.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant