Repository navigation
Server: optionally refuse uploads when the disk or the capacity is full (after #38) - #52
Open
Skyfuller1303 wants to merge 202 commits into
Open
Skyfuller1303 wants to merge 202 commits into
Skyfuller1303 wants to merge 202 commits into
Conversation
`craft-cloud-server healthcheck [--addr 127.0.0.1:8080]` sends GET /api/info over a plain TcpStream and exits 0 only on a 200 status line, within 3 s. Otherwise it exits 1 with one line on stderr. It prints nothing on stdout and adds no dependency. A container image without a shell or curl can use it as its health check. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ment packaging/docker holds a Dockerfile (Debian trixie build, distroless cc runtime as uid 65532, base images pinned by digest) and a Compose deployment: the server behind Caddy for TLS (acme, internal or custom certificates), with SQLite and the local disk by default and overlays for Postgres and S3. Secrets never reach a service's environment. A one-shot secrets-init service renders config.toml from a template and writes the secret files into a volume; every other service mounts only its own directory read-only. Every service has a read-only root, no capabilities and no-new-privileges. Caddy has a fixed address that the config lists in trusted_proxies. install.sh writes .env with random passwords, builds the image, starts the stack and creates the first admin from a password read on stdin. The Docker section in docs/server.md covers use, TLS, Postgres, S3, upgrading and what is not covered. The Dockerfile uses the healthcheck subcommand. Derived from the deployment in Skyfuller1303/craftcloud (deploy/compose), written by the same author, MIT OR Apache-2.0, adapted to this repository's TOML-only configuration. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Migration 0007 gives a device its client kind, platform and app version and, once revoked, the time and the reason, and gives a user the time of the last password change (plus a disabled time nothing reads yet). All of it is additive, BIGINT on Postgres, and the flag stays the check. Meta gets create_device_with and redeem_with, rename_device, revoke_device_as, revoke_other_devices, revoke_all_devices, account, change_password and reset_password. Flag, time and reason are set together and only by the first revoke. A password change or reset sets the hash, revokes the devices and deletes the account's invite and pairings in one transaction. Tests run on both databases, including the migration from 0006. Co-Authored-By: Claude <noreply@anthropic.com>
New: GET /account, POST /account/password, PATCH /devices/{id} and
POST /devices/revoke-others. Sign-in and join take an optional client
kind, platform and app version, checked before a try is counted. The
device list shows them with the time and reason of a revoke; all new
fields are optional on the wire.
Changing the password answers a wrong current password with 403
wrong_password, never 401, since the agent reads every 401 on a
signed-in route as a revoked device. It is limited to five tries per
account in 15 minutes, takes a password-check slot, and ends the revoked
devices' live connections at once. Password and name bodies are read up
to the new TINY_JSON_BYTES (4 KiB).
wrong_password and forbidden move into the shared error codes; the local
API refers to them. The agent, app and desktop only change where they
build the extended types.
Co-Authored-By: Claude <noreply@anthropic.com>
user reset-password now revokes every device of the account (reason reset) and deletes its open invite and pairings, so a stolen token does not outlive the old password. --keep-devices restores the old behaviour for the devices. device revoke-all signs out an account's devices without touching the password. Both print how many devices they ended; device revoke records the admin as the reason. Co-Authored-By: Claude <noreply@anthropic.com>
The API table and error codes in the design document, the password and sign-out commands, the live-connection delay of the command line and the rollout note (an older server refuses the newer database) in the server guide. Co-Authored-By: Claude <noreply@anthropic.com>
…evice A /changes long-poll that was waiting when its device was signed out through the API now ends at once, and one that wakes for any other reason looks the device up again before it answers, so a device signed out by the command line gets 401 instead of the next change set. The live endpoint listens for revocations and looks the device up before the upgrade, so a revoke between the sign-in check and the subscription is not missed. Co-Authored-By: Claude <noreply@anthropic.com>
change_password checks, inside its transaction, that the calling device is still signed in (it may have been revoked while the password was hashed) and answers 401 if not. A password that changed meanwhile is now 409 conflict instead of 400 invalid. Both are new MetaError variants mapped in the server. set_password_hash had no caller outside a test and is removed. A new test breaks the pairing deletes with a trigger and checks that a password change, an admin reset and a revoke-others undo everything on SQLite and Postgres. Co-Authored-By: Claude <noreply@anthropic.com>
The request used bare line feeds instead of CRLF, and an answer that is not HTTP at all was reported as a bad status. Send CRLF line endings and report "not an HTTP answer" unless the first word is an HTTP version. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The long-poll now listens for revocations and looks its device up before it waits, so a revoke between the sign-in check and the wait is not missed, and looks it up once more after the change set is fetched, as the last step before the answer, so a revoke during the fetch is not answered with changes. Co-Authored-By: Claude <noreply@anthropic.com>
…changes a password change_password locked the caller's device row and then the user's row, while revoke_other_devices from another device locked its own device and then the user's, and reset_password locked the user before the devices. On Postgres two of them at once could deadlock and fail with a database error. All of them now take the user's row first (a no-op update), then the devices, then the pairings. A new scenario runs password changes, sign-outs and an admin reset at the same moment, over and over, on both databases and expects only successes, conflicts and signed-out callers. The test trigger helper also drops its Postgres function. Co-Authored-By: Claude <noreply@anthropic.com>
An answer that stopped inside the status line ("HTTP/1.1 200" and then EOF) or
filled the 64 byte buffer without a line end counted as healthy or as a bad
status. Now only a status line that ends in CRLF is read, and the other cases
are errors. Tests cover a truncated line, a line that does not end and a bare
line feed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When the server is down, Caddy's error log entry for a failed request carried the full request URI and headers, so an invite code (/join/<code>), a query or a bearer token could end up in the log. The default log now drops request>uri and request>headers. Caddy also skips installing its internal CA into the container, which only produced an error line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
db-init exported PGPASSWORD, which put the superuser password in the environment of the shell and of psql. It now writes a mode 0600 password file on a tmpfs, passes only PGPASSFILE and removes the file on exit. The db-init service gets a small noexec tmpfs for it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…docs The server started before Caddy and was given 172.29.215.2, the address reserved for Caddy, so Caddy failed to start. The frontend network now hands out dynamic addresses from CC_FRONTEND_POOL (the upper half by default). The docs name the new setting, the error of a data directory with the wrong owner and what was tried with S3. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
render-config.sh read only the first line of a secret file, so a database password with a second line passed the check while db-init, which reads the whole file, set a different password than the one in config.toml. The whole file is read now, one trailing line feed is allowed and any other line feed is refused. The tests cover a second line, an empty second line and the allowed trailing line feed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Compose gives a container its secrets from .env when it is created. After `docker compose run --rm secrets-init`, a later `up -d db-init` started the original secrets-init container again, which wrote the old values back. The docs and the script comment now say to recreate it (`up -d --force-recreate secrets-init`) before restarting the services that read the files. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…-in adds a device redeem consumed the invite and then updated the user, while a password reset locks the user and then deletes the invite, so the two could deadlock on Postgres. redeem now finds the invite or pairing with a plain read, locks the account, and consumes the code again under the lock; a code that is gone by then is invalid_code. create_pairing locks the account first too. A password sign-in verified the password outside any transaction, so a reset or change that committed meanwhile could be followed by a new device. sign_in_device locks the account and compares the stored hash again before it creates the device; if it differs, login answers as for a wrong password. New scenarios run redeems, a sign-in, a pairing and a reset at the same moment on both databases and expect no database error and no live device once the reset is done. Co-Authored-By: Claude <noreply@anthropic.com>
The invite delete and the pairing update now name the account the plain read found, so a code hash removed and made again for another account between that read and the lock is refused as invalid_code instead of adding a device to the wrong account. Co-Authored-By: Claude <noreply@anthropic.com>
The admin API's server overview shows whether blobs are on disk, in S3 or in memory. A backend that does not say is reported as other, so nothing outside the store crate has to change. Co-Authored-By: Claude <noreply@anthropic.com>
New request and answer types for the routes under /api/v1/admin, with a redacting Debug for everything that holds a password or a code, and the error codes account_disabled and last_admin. The confirmation is a body with an optional password today, so an optional totp_code can join it later without breaking a client. Co-Authored-By: Claude <noreply@anthropic.com>
Migration 0008 adds the purpose of an invite (invite or reset), a quota per space and a removal mark per user. The new Meta methods for the admin API each re-check the acting admin inside their own transaction and take the locks in one order, with a transaction-wide lock before the last-admin count. Disabled accounts are refused by the token check, sign-in, redeem, pairing and device creation, decided under the account's row lock. A reset code always sets a new password, signs every device out and then signs the redeeming computer in. A space's quota replaces the server-wide cap in the preflight check and in the count that follows. Co-Authored-By: Claude <noreply@anthropic.com>
The require_admin layer and the routes under /api/v1/admin: counters, a paged and searchable user list, creating accounts, changing role and quota, disabling, enabling and removing, invites, password reset codes and a user's devices. Every change asks for the acting admin's password in its body. Sign-in and join answer account_disabled for a disabled account, and a redeemed reset code ends the live connections of the devices it signed out. Documents the API, the confirmation and the rollout in the design document and the server guide. Co-Authored-By: Claude <noreply@anthropic.com>
Limiter::count added the attempt before comparing it with the maximum, so tries refused at the limit still raised the count. A refund of an attempt that had been accepted at the same moment then left the key above its limit with fewer wrong passwords than allowed. A refused try now leaves the count alone. Co-Authored-By: Claude <noreply@anthropic.com>
Disabling and enabling an account leaves the password hash as it was, so a sign-in that verified the password before the disable could still create a usable device after the enable. Users get an auth_gen counter (migration 0008) that a disable, an enable, a removal and every password change or reset bump. The sign-in reads it with the hash and sign_in_device compares both under the account's lock. The server's refunds test covers the limiter change. Co-Authored-By: Claude <noreply@anthropic.com>
Compaction asks a replica for a snapshot of a live document at an older version. Loro's SnapshotAt export checks the document out at that version, and on a long text history that needs many GiB: the live_documents test an_app_that_misbehaves_is_refused_or_dropped_and_the_rest_go_on, whose app sends 60 updates of 200,000 characters, ran out of a 16 GiB limit (itrycodinganditsbad#15). snapshot_at now exports the operations up to the version, imports them into a fresh document and takes that document's snapshot: the same history and state, with memory linear in the history. A version whose operations need others past it is refused instead of silently widened, a shallow document is refused at once (Loro can't fork one at a version), and a scratch document that Loro panicked in is let go of without being dropped. Co-Authored-By: Claude <noreply@anthropic.com>
The typed details of a security event, an event, a page of them and the last line of the NDJSON export. Every field of the details is a closed list of words or a number, and fields a later release adds are ignored. Co-Authored-By: Claude <noreply@anthropic.com>
Migration 0009 adds audit_events, with no foreign keys so that an event outlives its account. Every change that the log names writes its event in its own transaction before the commit, so a change whose event cannot be written is not made. A change that changes nothing writes none. A failed sign-in is one row per account and minute whose counter goes up, written in a transaction of its own. The values of an event are typed and come from closed lists, never from what a client typed. Old events are pruned in batches outside those transactions. Meta methods that had no way to say who asked get one: revoke_device_by takes the origin, and admin_enable_user and admin_revoke_invite take the time. Co-Authored-By: Claude <noreply@anthropic.com>
GET /account/activity, GET /admin/audit and the streamed GET /admin/audit/export; a failed sign-in is written after the answer is decided and never changes it. [audit] retention_days (1 to 3650, default 365) and audit list and audit prune on the command line. The export reads fixed pages through a bounded channel up to a fixed end of the log and ends with a line that says complete or truncated, or breaks the connection. Co-Authored-By: Claude <noreply@anthropic.com>
Every ui test built its harness with `.wgpu()`, which creates a fresh wgpu instance and device (WARP on Windows) per test. The tests run on parallel threads, and creating and dropping several D3D12 devices at the same time crashed the test process with STATUS_ACCESS_VIOLATION (0xc0000005) in the middle of the run, in most runs of `cargo test -p craft-cloud-desktop --lib` on Windows. With `--test-threads=1` it never happened. Create the instance, adapter, device and queue once, keep them in a `OnceLock` (so no device is torn down while other threads render) and hand them to each harness with `wgpu_setup(WgpuSetup::Existing(..))`. Every harness still gets its own renderer. Snapshots are unchanged. Test code only; this works on every backend kittest picks (lavapipe, WARP, Metal).
Another test helper in a later branch is called sorted; a distinct name keeps the trash tests compiling wherever they are merged.
Other test modules of the app (such as a Security page test module in a stacked pull request) build their own kittest harnesses. Make shared_wgpu() visible to them, so all of the app's harnesses render on one device.
…ce between the desktop tests
The Security page tests (from itrycodinganditsbad#19) built their kittest harness with .wgpu(), a new wgpu device per test, which crashed the test process with STATUS_ACCESS_VIOLATION when run in parallel on Windows, like the other UI tests did before itrycodinganditsbad#35. Use the shared device from itrycodinganditsbad#35 here too.
…ersions Back from the trash or file versions built a fresh overview, which starts with the storage usage loading and the admin check unknown, but only asked for the computers. The usage stayed on "Loading storage usage…" with Refresh disabled until the app restarted, and the admin pages link stayed hidden. Both pages now go back through one helper that asks for everything a fresh overview shows, like signing in does.
Adds [storage] capacity (and init --capacity), cheap filesystem and SQLite size measurements (fs4), optional fields on AdminStorageOverview, docs and tests.
…, one statvfs per directory
[storage] refuse_when_full (default off) makes the server answer 507 server_full to a new chunk or manifest when the store's filesystem would drop below min_free (disk, default 1 GiB) or the stored bytes would pass capacity (disk or S3). Reads, deletes, GC and admin calls stay open. The agent pauses with its own message instead of the account's quota text.
…already has, bound the free-space read - A request's declared size never sets or logs the full state: it is judged on a one-byte upload, and a declared body over the route's limit is 413 before the full check. - A chunk or manifest the account already has is accepted while the server is full, and not counted. - The bytes the server stored are counted after the record is committed, so a refresh in between can't lose them. - The free-space read has a 2 s timeout and at most one read runs at a time; unknown free space is logged once and the reserve is not checked meanwhile. - Starts of refusing are logged at most once a minute. - docs: capacity is soft, the total sums the accounts, behaviour with large bodies.
The paused run state in the local API's status carries an optional `reason_code`: `server_full` where the server refuses uploads because it is full, `quota_exceeded` where the account is over its quota, and nothing for any other pause. The loop state in status.json keeps it too. An app can match the code instead of the words of `reason`, which are free to change. The field is serde-optional, so an older desktop ignores it and an older agent sends none.
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge after #38. The diff of this PR alone: storage-capacity...server-full.
A server whose disk fills up fails uploads halfway, and a store past its configured capacity says nothing. This adds an opt-in setting that makes the server refuse new uploads in good time, with an answer of its own that the agent does not confuse with the account's quota. It answers question 3 in #37 ("refuse uploads when the capacity is reached"). Server, API, sync engine and agent only.
What
[storage] refuse_when_full, defaultfalse, andmin_free, disk only, default"1 GiB":min_freeafter the write (disk), or when the stored bytes would passcapacity(disk or S3, when set).ignore_forwarded_for:min_freewithoutrefuse_when_full,min_freeon S3, andrefuse_when_fullon S3 withoutcapacityare errors, since they would do nothing. Errors formin_freenamestorage.min_free(the size parser now takes the setting's name).docs/server.md.apps/craft-cloud-server/src/full.rs): the stored total (the existing counts query) and the free space (spawn_blocking) are read at most every 5 seconds; the bytes the server wrote itself since are added in between. A refusal ends within about 5 s of space being freed. The free-space read is a seam in the tests, so no test fills a disk. A filesystem that can't be read never refuses.507 server_full: a new code incodes, andErrorBody.server(ServerRoom { free_bytes, total_bytes, capacity_bytes }, each optional) for what the server knows. It is notquota_exceeded. The check runs before the body is read (byContent-Length) and again with the real size.TransportError::ServerFull(a unit variant: the engine ends the pass, as for a full account, and the agent pauses, rechecked every 10 minutes). The message is "The server is full; the admin needs to free space or raise its capacity", never the account's. Old agents don't know the code and see a plain HTTP 507 server error: they show the server's message as an error and retry with growing pauses (up to 5 minutes), losing nothing. Documented.GET /v1/status(run_state:paused) gains an optionalreason_code(server_full, orquota_exceededfor an account over its quota), set where those two transport errors pause the pass and left out for every other pause. It is#[serde(default, skip_serializing_if = "Option::is_none")], so an older desktop ignores it and an older agent sends none. An app can match the code instead of the words ofreason. Also kept instatus.json's loop state.413first. A chunk or manifest the account already has is accepted while full and not counted. The server's own bytes are counted after the record is committed. The free-space read has a 2 s timeout and runs one at a time (a dead mount means "unknown", logged once, reserve not checked).capacityis soft (bodies in flight) and counts the sum over accounts; documented.warnwhen refusing starts (with the reason, capacity or free space) and one when it stops. No file names.docs/server.md("A full server", upgrade note),docs/agent.md(the pause message).Checks
On Windows, through a memory-limited runner, on the head of this branch:
cargo fmt --all -- --checkcargo xtask licensescargo clippy --workspace --all-targets --locked -- -D warningscargo test -p craft-cloud-server -p craft-cloud-api -p craft-cloud-app -p craft-cloud-agent -p craft-cloud-sync -p craft-cloud-meta -p craft-cloud-store --lockedcargo test -p craft-cloud-desktop --locked -- --test-threads=1CRAFT_CLOUD_REQUIRE_BACKENDS=1)cargo +1.95 check --workspace --all-targets --locked(MSRV)Results (Windows, memory-limited runner, head of this branch):
fmt --check,clippy -D warningsand the seven-cratetestrun all passed (exit 0; test run 744 s, peak 2.6 GiB). The backends, desktop, licenses and MSRV runs are on the next PR's head, which has the same code plus its own commits, and passed there.New tests: off by default (a small
capacityalone refuses nothing); refusal at the capacity (chunk and manifest) with the numbers in the body and no quota; refusal at the disk reserve; reads and admin calls still work while full; the reserve and capacity arithmetic and the cache; config parsing and the three refused combinations; the agent's mapping, exit code and pause message; the engine ends a pass atServerFullwith its own message and uploads again once space is back.Latest check of this head
Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and
CRAFT_CLOUD_REQUIRE_BACKENDS=1under a 16 GiB memory limit at11154e7:cargo fmt --all -- --checkandcargo clippy --workspace --all-targets --locked -- -D warningspass;cargo test --workspace --locked --no-fail-fast: 1770 passed, 1 failed, 2 ignored. The failure,on_postgres_s3::oversized_manifests_are_refused, is a timing test; it passes when run alone (3 of 3 runs).Not in this PR
refusing_uploads, store reachability): the next PR, on top of this one.