Skip to content

Account usage: stored bytes, quota and trash/history estimates (read-only) - #32

Open
Skyfuller1303 wants to merge 38 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:account-usage
Open

Skyfuller1303 wants to merge 38 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:account-usage

Conversation

@Skyfuller1303

@Skyfuller1303 Skyfuller1303 commented Nov 5, 2026 •

Copy link
Copy Markdown

Stacked on #25 (reference index and verify). This PR's own commits are the five of git log --no-merges gc-reference-index..account-usage; the later merge commits bring in #25's update.

A signed-in device can ask how much its account stores and roughly how much of that is held only by the trash or by old versions. This is the read side for a later usage display in the desktop app and the web page; it changes nothing, adds no table, no migration and no counter.

What

  • GET /api/v1/account/usage for the device's own space:
    {"stored_bytes": 11, "quota_bytes": null, "trash_bytes": 0, "history_bytes": 0,
     "estimates": {"index_complete": true}}
    • stored_bytes is the existing spaces.stored_bytes: chunks and manifests of the space, each once, with their format byte.
    • quota_bytes is the configured stored_bytes_per_user cap, or null. A per-account override (Enforce the admin role and add an HTTP admin API #17) should take precedence once both are merged.
    • trash_bytes: blobs named only by versions of deleted nodes. No live version, live-document snapshot or import snapshot may name them. A node with a pending purge still counts, because the purge frees nothing yet.
    • history_bytes: blobs named only by old versions of live nodes. No current version, saved version (one with a save record), deleted node or live-document snapshot may name them.
    • A chunk counts only when every manifest that names it is in the class. Shared blobs count once per space. Unrooted manifests (possible app assets, see Reference index of manifest chunks and a read-only verify #25) are not counted.
    • Both are labelled, overlapping estimates, never added to each other or to stored_bytes. If any manifest of the space has an incomplete reference index, both are null with estimates: {index_complete: false, reason: "index_incomplete"}.
    • 401 without a session.
  • Implementation: one SQL aggregate (crates/meta/src/usage_estimate.sql) in one read snapshot; no manifest or chunk rows are loaded into Rust. The SQLite query plan uses versions_node_seq, manifest_chunks_by_chunk and primary keys; a test pins that.
  • Read-only: the auth middleware skips its last_used_ms touch for this route, so a usage poll writes nothing.
  • Docs: docs/server.md, docs/design/sync.md.

Assumptions (please confirm or redirect)

  1. Saved versions are treated as kept in the history estimate, although nothing thins versions yet. Overstating what could be freed seemed worse than understating it. A saved version of a deleted node still counts as trash, since purging the node removes it.
  2. The estimates are per space, matching stored_bytes; a blob shared with another account counts for both.
  3. Skipping the last_used_ms update for this route is acceptable. The session check itself is unchanged.
  4. No usage UI yet; it follows as its own PR on top of this one.

Testing

  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets --locked -- -D warnings.
  • SQLite: cargo test -p craft-cloud-meta -p craft-cloud-server -p craft-cloud-proto --locked (meta 194 behaviour, server 90 API and the unit and CLI suites, proto).
  • Postgres 17: craft-cloud-meta with CRAFT_CLOUD_REQUIRE_BACKENDS=1, and craft-cloud-server --test api on Postgres with the disk store.
  • cargo check of proto, meta and server on Rust 1.95.
  • Behaviour tests on both backends: empty and current-only accounts, an old-only manifest and chunk, chunks shared between old and current versions, between trash and live content, and with a saved version, live-document snapshots as roots, unrooted manifests, a pending purge, per-space deduplication, and an incomplete index.
  • Writes nothing: on SQLite the test compares the database file and WAL bytes before and after requests; on Postgres it hashes every row of every table.
  • Each of these rules was checked against a deliberately broken build, and each break failed its test: a chunk shared with live content, a chunk shared with the current version, an incomplete index, a snapshot-named old manifest, and a saved old version.

Not tested

  • The Postgres query plan (only the indexes were compared). (Postgres with S3 was since checked against MinIO: cargo test -p craft-cloud-server --test api, 90 passed, including the Postgres + S3 scenarios.)
  • Linux and macOS; this was built on Windows.

Adds the request and answer types of GET /nodes/{id}/history and POST /nodes/{id}/restore, the opaque history cursor with a single accepted spelling, and the paging limits (default 50, at most 200).
history() pages a file's versions newest first by (seq, id) with an exclusive cursor. restore_version() records a new version with the content of an old one as an ordinary update, checking the source, the current version, deletion and live documents under the same space lock as a commit. No migration is needed.
GET /api/v1/nodes/{id}/history and POST /api/v1/nodes/{id}/restore, resolved in the caller's own space. A stale expected_version answers 409 conflict. Covers permissions, paging, conflicts, restoring old and missing versions, and parallel restores in the API tests.
Updates the API table and the milestone note in the sync design, and adds a short section to the server guide.
A history page now reads versions through an index on (space_id, node, seq, id) in the order of the query, so the page size bounds the work. The migration adds the index on both databases. The restore note in the sync design no longer claims the engine prefers a newer live-document state: a restore is an ordinary new version and files with a live document are refused.
Adds the list, restore and cursor types of GET /trash and POST /trash/{id}/restore, with the same page limits as the history.
…und taken names

A migration adds nodes.deleted_ms and nodes.purge_requested_ms and an index for the trash list, and dates old deletions at the migration. The commit path stamps the time when a node goes from live to deleted and clears it when the node comes back. restore_from_trash brings a deleted node and its deleted folders back in one transaction, merging into live folders of the same name and renaming around taken names with the engine's conflict pattern. request_purge only marks a node, so nothing is freed.
GET /api/v1/trash, POST /api/v1/trash/{id}/restore and DELETE /api/v1/trash/{id}, resolved in the caller's own space, and the [retention] trash_days setting (default 30, 1 to 3650) that sets purge_after_ms.
Adds the trash routes and rules to the sync design and a section with the new setting and the rollout note to the server guide.
…ders within the depth limit

A file restored from the trash now gets a new current version with the content of its last one (no save record). A machine that sent a delete against the old version and never heard the answer can no longer delete the restored file again: the server refuses that delete as a version mismatch, and the engine already lets go of a node that came back at another version. The restore answer reports the new version. A folder that comes back below a folder that has since moved deep is refused as too deep, and nothing changes.
A folder has no version to tell a delete that went through and a later restore from one that never arrived. The engine now remembers the folder deletes it sent without hearing the answer, and when the next pull reports such a folder changed and live, lets it go instead of deleting it again. A folder the feed does not mention was never touched and its delete is sent again. Scenario tests cover a file and a folder restored after a lost answer, at its own path, under a conflict name, and merged into a live folder.
A restored file has a new version, and agents need the folder rule for the rare case of a folder restored after a delete whose answer was lost. Reports the new version in the restore answer.
…d are settled

A folder deleted together with its file, whose answer was lost, and then restored alone, still had the file tracked when the pull judged it, so it was kept and deleted again once the file was let go. The pull now only notes which folders the feed reports live, and the decision is made in apply_remote after the files whose deletes went through are let go. Scenarios cover the non-empty folder restored alone, the folder and its file both restored, and only the file restored.
A folder this machine deleted with a lost answer and that came back from the trash was judged before the folders below it were settled. A tombstoned subfolder still tracked in it kept it, and the next push deleted the restored folder again. The folders of this machine's own delete are now settled deepest first: one the server lists as live, or no longer has and that is gone here too, is let go, and only then are the parents judged.

A relist at the same epoch (the change feed answered resync required) did not note which of those folders the server lists as live, so a restored empty folder stayed tracked and was deleted again. The relist now notes them, so a folder the server has is never pushed as deleted from a stale request.

Regressions cover d/sub/f and d/a/b/f with only the top folder, a middle folder, or only the deepest node restored, and a relist after the restore.
A migration adds manifest_chunks (manifest, ordinal, chunk) and manifest_index (chunk count, edges written, state). The index is written in the transaction that records a manifest, up to a budget of edges per transaction, and a longer manifest is finished in batches by index_complete, which also resumes an interrupted build and indexes manifests recorded before the index existed. An index is marked complete only after every edge was compared with its manifest, so wrong, swapped, missing, extra and collapsed edges are found. index_audit reports what the database alone can tell: orphan edges, index rows without a manifest, unknown dependencies, roots that name unrecorded manifests, and manifests that nothing names. Nothing is deleted except the derived edges of an index that does not match its manifest while it is still being built.
…ore the upload is acknowledged

verify checks the database and the store against each other and reports problems and notices as text or JSON, changing nothing. index build indexes manifests recorded before the index existed, in bounded batches. PUT /manifests completes the manifest's index before answering.
The comparison of a manifest's edges started its pages after ordinal -1, so an edge at a negative ordinal was never seen and an index with such an edge could be marked complete. The comparison now starts at the lowest number there is and reports an edge at a negative ordinal as a mismatch of its own kind, so completion is never published over it.

Starting an index again deleted all of a manifest's edges in one transaction, up to about a million rows, which broke the batch budget and blocked other writers. The reset now marks the index row as resetting (state 2) and deletes at most the index budget per transaction, lowest ordinal first; nothing is written or marked complete until the last edge is gone, and a run that stops between two batches is resumed by the next stage. The old behaviour of a complete row that is wrong being left to the checker is unchanged.

Regressions cover negative ordinals (including the lowest possible), a reset across several batches that resumes on a new handle after a stop, and a build that resets a wrong index by itself.
…es it reads

`verify` opened SQLite with create enabled and made missing storage directories, so a wrong path gave a new empty server and "no problems found". It now opens the database without creating it (`Meta::open_existing`: a missing file, a file with no Craft Cloud data, or a schema older than the program is an error, and nothing is created or upgraded) and requires the storage directory to exist. `index build` still uses the creating open.

A successful read of a chunk or manifest object discarded its bytes, so seeded size corruption passed. Reading now reports the stored length (format byte included) and verify checks it against `blobs.stored_size` for chunks and manifests, the file size recorded for a manifest against the manifest, and the chunk lengths a manifest names against the length recorded for each chunk (one less than the stored size).

Regressions cover a wrong database path, a wrong storage path, and seeded wrong sizes with and without reading chunk objects.
…y claim

The verify schema test now snapshots the database and WAL straight from disk, with no SQLite connection opened between the
setup checkpoint and the check. The concurrent index reset test holds the manifest_index row lock, proves both builders
wait on it, and only then releases it. The i64::MAX edge test is renamed for what it pins; the cursor overflow guard is not
reachable through the public API.
…nnot checkpoint a crashed WAL

Closing the last read-write connection to a WAL-mode database checkpoints the
WAL of a crashed process into the database file and removes it. Whether that
finished before verify exited was a race, and a command that only reads must
not make that write. Meta::open_existing now opens SQLite with mode=ro, which
still reads the WAL's committed data. The tests run verify as a process of its
own and compare the database file, the WAL and the storage tree after it has
exited, with a row that only the WAL holds and with the -shm file gone.
history_bytes counted the manifest and exclusive chunks of an old version
that carries a save record, although an app saved that version on purpose.
A manifest named by a saved version of a live node is no longer history, and
a chunk shared with it stays out as well. A saved version of a deleted node
still counts as trash, since purging the node removes it. Tests cover both
backends; sync.md and server.md state the rule.
Skyfuller1303 added a commit to Skyfuller1303/craft-cloud that referenced this pull request Nov 6, 2026
itrycodinganditsbad#33 (account usage in the desktop app) was built on itrycodinganditsbad#32 only, while this
branch also carries the admin stack. Merging it here resolves their
overlap once, so itrycodinganditsbad#33 and itrycodinganditsbad#34 can land in either order.

Conflicts kept both sides: the agent's local API routes and imports, the
desktop model (usage fields, events and effects next to the admin,
history and trash ones), the app and proto re-exports, and the overview
snapshots, which were redrawn.

Fixes needed after the textual merge:
- crates/api/src/api.rs: drop the second copy of AccountUsage,
  UsageEstimates and UsageEstimateReason that the merge duplicated.
- Tests: add `name_taken: None` to Error::Refused and ErrorAnswer values,
  and give admin_server_limited the `task` field of Server.
…he trash

Restoring a folder from the trash now also brings back every node below it that
is in the trash (deleted and not removed from the trash), however long before it
was deleted, in the same transaction; the trash list shows such a node only
through its folder; and removing a folder from the trash marks them too. A
folder with more than 10,000 such nodes below it is refused whole as invalid
for both, and Meta::with_trash_tree_cap lowers the limit for tests.
… trash first

Restoring a folder from the trash now brings back all that is in the trash below
it, so the lost-delete tests that restored a folder alone and expected its file
to stay deleted take that file (or sub folder) out of the trash first.
Conflict in crates/meta/src/lib.rs: both new Meta fields (index_rows and
trash_tree_cap) are kept.
… rest is listed

Instead of refusing a folder with more than 10,000 items in the trash below it
(which left it impossible to restore or remove from a list that hides what is
below a trashed folder), a restore or removal now takes the folder and at most
the cap of items below it, shallowest first. The rest stays in the trash and is
listed on its own because its folder is live or removed, so a further call takes
it. The per-row work under the write lock is bounded by the cap.
Another test helper in a later branch is called sorted; a distinct name keeps the
trash tests compiling wherever they are merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant