Skip to content

Server: compression settings, store tools and the admin's view of what compression saves (after #36 and #38) - #54

Open
Skyfuller1303 wants to merge 225 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:compression-admin
Open

Skyfuller1303 wants to merge 225 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:compression-admin

Conversation

@Skyfuller1303

Copy link
Copy Markdown

Merge after #36 and #38. The diff of this PR alone: storage-capacity...compression-admin.

The compression of #36 (zstd, settable level and minimum saving, store recompress, estimate and measure) merged into the storage capacity branch of #38, plus the admin's view of what compression saves and a way to change the level. Most of the commits in the compare are #36's, which this branch contains up to c49ca59; the 12 commits of this PR alone are the admin's view and level setting (API, meta, server, app, agent, docs, tests) and the exact round trip of the remembered estimate. It does not contain #36's last commit (e0cbe8d, which compares the estimate's floats within a tolerance); this branch fixes the same flake differently (d8fb03a stores the estimate as whole hundredths, so it reads back exactly). The two change the same code in apps/craft-cloud-server/src/storage_tools.rs, so whichever of the two lands second needs a small conflict resolution: keep the whole-hundredths version.

What

The admin storage overview now says how blobs are stored and what that saves, and an admin can change the compression level without restarting the server. The desktop pages for it come in a separate change.

  • GET /admin/storage has an optional compression part: the algorithm (from the config), the level in use, the config's level and the admin's, the minimum saving, the logical size of the blobs, the physical size where it is known, the share of the data that covers, the saving over it, and the last store estimate with its date ("level 19: 80 % possible"). Fractions are permille, so the type stays Eq. It is optional and left out by an older server; an older client ignores it. Blobs from before sizes were tracked are unknown, not guessed, and the share says so. The totals sum the blobs table, so the overview keeps them for 30 seconds.
  • PUT /admin/storage/compression {level} sets the level (1 to 19) for new writes. It needs the admin's password (and second factor) like every admin change, is kept in the database, applied at once, and taken again at the next start. The config file stays the source of truth for the algorithm: compression = "none" wins over any level, and the overview shows that. Each change is an admin.compression_changed audit event with both levels (none when the level doesn't change).
  • The client call is Client::admin_set_compression_level in crates/app, and the agent's local API passes PUT /v1/admin/storage/compression on unchanged.
  • Another server process on the same database takes the new level at its next start; this one applies it immediately.
  • Merge order: this part carries the compression migration as 0015, and the first PR carries it as 0007. Whichever lands upstream first keeps its number and the other renumbers before any release; the first PR's 0007 must not be released alone if the open stack follows. Team spaces take 0016 on the integration branch.

Testing: the overview with and without the new part (an old body without it still reads, an unknown future field is ignored), the confirmation rules (member 403, no password 400, wrong password 403, levels outside 1 to 19 400), the audit event once per change, the next write at the new level, none winning, and the level coming back after a restart. Meta tests on SQLite and Postgres cover the audited change and its refusals. App and agent tests cover the client call and the pass-through.

Latest check of this head

Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and CRAFT_CLOUD_REQUIRE_BACKENDS=1 under a 16 GiB memory limit at d8fb03a: cargo fmt --all -- --check and cargo clippy --workspace --all-targets --locked -- -D warnings pass; cargo test --workspace --locked --no-fail-fast: 1793 passed, 1 failed, 4 ignored. The one failure was the desktop test audit::tests::filters_are_the_closed_list_and_start_the_list_over (hardcoded 36 choices against 37 audit actions); the head moved to e99c47d, which counts the choices from the action list instead. At e99c47d the audit tests pass (9 of 9) and clippy of the desktop crate is clean; the rest of the tree is the checked head.

Skyfuller1303 and others added 30 commits November 3, 2026 22:55
`craft-cloud-server healthcheck [--addr 127.0.0.1:8080]` sends GET /api/info
over a plain TcpStream and exits 0 only on a 200 status line, within 3 s.
Otherwise it exits 1 with one line on stderr. It prints nothing on stdout and
adds no dependency. A container image without a shell or curl can use it as
its health check.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ment

packaging/docker holds a Dockerfile (Debian trixie build, distroless cc
runtime as uid 65532, base images pinned by digest) and a Compose deployment:
the server behind Caddy for TLS (acme, internal or custom certificates), with
SQLite and the local disk by default and overlays for Postgres and S3.

Secrets never reach a service's environment. A one-shot secrets-init service
renders config.toml from a template and writes the secret files into a volume;
every other service mounts only its own directory read-only. Every service has
a read-only root, no capabilities and no-new-privileges. Caddy has a fixed
address that the config lists in trusted_proxies.

install.sh writes .env with random passwords, builds the image, starts the
stack and creates the first admin from a password read on stdin. The Docker
section in docs/server.md covers use, TLS, Postgres, S3, upgrading and what is
not covered. The Dockerfile uses the healthcheck subcommand.

Derived from the deployment in Skyfuller1303/craftcloud (deploy/compose),
written by the same author, MIT OR Apache-2.0, adapted to this repository's
TOML-only configuration.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Migration 0007 gives a device its client kind, platform and app version
and, once revoked, the time and the reason, and gives a user the time of
the last password change (plus a disabled time nothing reads yet). All
of it is additive, BIGINT on Postgres, and the flag stays the check.

Meta gets create_device_with and redeem_with, rename_device,
revoke_device_as, revoke_other_devices, revoke_all_devices, account,
change_password and reset_password. Flag, time and reason are set
together and only by the first revoke. A password change or reset sets
the hash, revokes the devices and deletes the account's invite and
pairings in one transaction.

Tests run on both databases, including the migration from 0006.

Co-Authored-By: Claude <noreply@anthropic.com>
New: GET /account, POST /account/password, PATCH /devices/{id} and
POST /devices/revoke-others. Sign-in and join take an optional client
kind, platform and app version, checked before a try is counted. The
device list shows them with the time and reason of a revoke; all new
fields are optional on the wire.

Changing the password answers a wrong current password with 403
wrong_password, never 401, since the agent reads every 401 on a
signed-in route as a revoked device. It is limited to five tries per
account in 15 minutes, takes a password-check slot, and ends the revoked
devices' live connections at once. Password and name bodies are read up
to the new TINY_JSON_BYTES (4 KiB).

wrong_password and forbidden move into the shared error codes; the local
API refers to them. The agent, app and desktop only change where they
build the extended types.

Co-Authored-By: Claude <noreply@anthropic.com>
user reset-password now revokes every device of the account (reason
reset) and deletes its open invite and pairings, so a stolen token does
not outlive the old password. --keep-devices restores the old behaviour
for the devices. device revoke-all signs out an account's devices
without touching the password. Both print how many devices they ended;
device revoke records the admin as the reason.

Co-Authored-By: Claude <noreply@anthropic.com>
The API table and error codes in the design document, the password and
sign-out commands, the live-connection delay of the command line and the
rollout note (an older server refuses the newer database) in the
server guide.

Co-Authored-By: Claude <noreply@anthropic.com>
…evice

A /changes long-poll that was waiting when its device was signed out
through the API now ends at once, and one that wakes for any other reason
looks the device up again before it answers, so a device signed out by
the command line gets 401 instead of the next change set.

The live endpoint listens for revocations and looks the device up before
the upgrade, so a revoke between the sign-in check and the subscription
is not missed.

Co-Authored-By: Claude <noreply@anthropic.com>
change_password checks, inside its transaction, that the calling device
is still signed in (it may have been revoked while the password was
hashed) and answers 401 if not. A password that changed meanwhile is now
409 conflict instead of 400 invalid. Both are new MetaError variants
mapped in the server.

set_password_hash had no caller outside a test and is removed. A new test
breaks the pairing deletes with a trigger and checks that a password
change, an admin reset and a revoke-others undo everything on SQLite and
Postgres.

Co-Authored-By: Claude <noreply@anthropic.com>
The request used bare line feeds instead of CRLF, and an answer that is not
HTTP at all was reported as a bad status. Send CRLF line endings and report
"not an HTTP answer" unless the first word is an HTTP version.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The long-poll now listens for revocations and looks its device up before
it waits, so a revoke between the sign-in check and the wait is not
missed, and looks it up once more after the change set is fetched, as
the last step before the answer, so a revoke during the fetch is not
answered with changes.

Co-Authored-By: Claude <noreply@anthropic.com>
…changes a password

change_password locked the caller's device row and then the user's row,
while revoke_other_devices from another device locked its own device and
then the user's, and reset_password locked the user before the devices.
On Postgres two of them at once could deadlock and fail with a database
error. All of them now take the user's row first (a no-op update), then
the devices, then the pairings.

A new scenario runs password changes, sign-outs and an admin reset at the
same moment, over and over, on both databases and expects only successes,
conflicts and signed-out callers. The test trigger helper also drops its
Postgres function.

Co-Authored-By: Claude <noreply@anthropic.com>
An answer that stopped inside the status line ("HTTP/1.1 200" and then EOF) or
filled the 64 byte buffer without a line end counted as healthy or as a bad
status. Now only a status line that ends in CRLF is read, and the other cases
are errors. Tests cover a truncated line, a line that does not end and a bare
line feed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When the server is down, Caddy's error log entry for a failed request carried
the full request URI and headers, so an invite code (/join/<code>), a query
or a bearer token could end up in the log. The default log now drops
request>uri and request>headers. Caddy also skips installing its internal CA
into the container, which only produced an error line.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
db-init exported PGPASSWORD, which put the superuser password in the
environment of the shell and of psql. It now writes a mode 0600 password file
on a tmpfs, passes only PGPASSFILE and removes the file on exit. The
db-init service gets a small noexec tmpfs for it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…docs

The server started before Caddy and was given 172.29.215.2, the address
reserved for Caddy, so Caddy failed to start. The frontend network now hands
out dynamic addresses from CC_FRONTEND_POOL (the upper half by default). The
docs name the new setting, the error of a data directory with the wrong owner
and what was tried with S3.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
render-config.sh read only the first line of a secret file, so a database
password with a second line passed the check while db-init, which reads the
whole file, set a different password than the one in config.toml. The whole
file is read now, one trailing line feed is allowed and any other line feed is
refused. The tests cover a second line, an empty second line and the allowed
trailing line feed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Compose gives a container its secrets from .env when it is created. After
`docker compose run --rm secrets-init`, a later `up -d db-init` started the
original secrets-init container again, which wrote the old values back. The
docs and the script comment now say to recreate it (`up -d --force-recreate
secrets-init`) before restarting the services that read the files.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…-in adds a device

redeem consumed the invite and then updated the user, while a password
reset locks the user and then deletes the invite, so the two could
deadlock on Postgres. redeem now finds the invite or pairing with a plain
read, locks the account, and consumes the code again under the lock; a
code that is gone by then is invalid_code. create_pairing locks the
account first too.

A password sign-in verified the password outside any transaction, so a
reset or change that committed meanwhile could be followed by a new
device. sign_in_device locks the account and compares the stored hash
again before it creates the device; if it differs, login answers as for
a wrong password.

New scenarios run redeems, a sign-in, a pairing and a reset at the same
moment on both databases and expect no database error and no live device
once the reset is done.

Co-Authored-By: Claude <noreply@anthropic.com>
The invite delete and the pairing update now name the account the plain
read found, so a code hash removed and made again for another account
between that read and the lock is refused as invalid_code instead of
adding a device to the wrong account.

Co-Authored-By: Claude <noreply@anthropic.com>
The admin API's server overview shows whether blobs are on disk, in S3 or in memory. A backend that does not say is reported as other, so nothing outside the store crate has to change.

Co-Authored-By: Claude <noreply@anthropic.com>
New request and answer types for the routes under /api/v1/admin, with a redacting Debug for everything that holds a password or a code, and the error codes account_disabled and last_admin. The confirmation is a body with an optional password today, so an optional totp_code can join it later without breaking a client.

Co-Authored-By: Claude <noreply@anthropic.com>
Migration 0008 adds the purpose of an invite (invite or reset), a quota per space and a removal mark per user. The new Meta methods for the admin API each re-check the acting admin inside their own transaction and take the locks in one order, with a transaction-wide lock before the last-admin count. Disabled accounts are refused by the token check, sign-in, redeem, pairing and device creation, decided under the account's row lock. A reset code always sets a new password, signs every device out and then signs the redeeming computer in. A space's quota replaces the server-wide cap in the preflight check and in the count that follows.

Co-Authored-By: Claude <noreply@anthropic.com>
The require_admin layer and the routes under /api/v1/admin: counters, a paged and searchable user list, creating accounts, changing role and quota, disabling, enabling and removing, invites, password reset codes and a user's devices. Every change asks for the acting admin's password in its body. Sign-in and join answer account_disabled for a disabled account, and a redeemed reset code ends the live connections of the devices it signed out. Documents the API, the confirmation and the rollout in the design document and the server guide.

Co-Authored-By: Claude <noreply@anthropic.com>
Limiter::count added the attempt before comparing it with the maximum, so tries refused at the limit still raised the count. A refund of an attempt that had been accepted at the same moment then left the key above its limit with fewer wrong passwords than allowed. A refused try now leaves the count alone.

Co-Authored-By: Claude <noreply@anthropic.com>
Disabling and enabling an account leaves the password hash as it was, so a sign-in that verified the password before the disable could still create a usable device after the enable. Users get an auth_gen counter (migration 0008) that a disable, an enable, a removal and every password change or reset bump. The sign-in reads it with the hash and sign_in_device compares both under the account's lock. The server's refunds test covers the limiter change.

Co-Authored-By: Claude <noreply@anthropic.com>
Compaction asks a replica for a snapshot of a live document at an older
version. Loro's SnapshotAt export checks the document out at that version,
and on a long text history that needs many GiB: the live_documents test
an_app_that_misbehaves_is_refused_or_dropped_and_the_rest_go_on, whose app
sends 60 updates of 200,000 characters, ran out of a 16 GiB limit (itrycodinganditsbad#15).

snapshot_at now exports the operations up to the version, imports them into
a fresh document and takes that document's snapshot: the same history and
state, with memory linear in the history. A version whose operations need
others past it is refused instead of silently widened, a shallow document
is refused at once (Loro can't fork one at a version), and a scratch
document that Loro panicked in is let go of without being dropped.

Co-Authored-By: Claude <noreply@anthropic.com>
The typed details of a security event, an event, a page of them and the last line of the NDJSON export. Every
field of the details is a closed list of words or a number, and fields a later release adds are ignored.

Co-Authored-By: Claude <noreply@anthropic.com>
Migration 0009 adds audit_events, with no foreign keys so that an event outlives its account. Every change that the
log names writes its event in its own transaction before the commit, so a change whose event cannot be written is
not made. A change that changes nothing writes none. A failed sign-in is one row per account and minute whose
counter goes up, written in a transaction of its own. The values of an event are typed and come from closed
lists, never from what a client typed. Old events are pruned in batches outside those transactions.

Meta methods that had no way to say who asked get one: revoke_device_by takes the origin, and admin_enable_user and
admin_revoke_invite take the time.

Co-Authored-By: Claude <noreply@anthropic.com>
GET /account/activity, GET /admin/audit and the streamed GET /admin/audit/export; a failed sign-in is written
after the answer is decided and never changes it. [audit] retention_days (1 to 3650, default 365) and
audit list and audit prune on the command line. The export reads fixed pages through a bounded channel up to a
fixed end of the log and ends with a line that says complete or truncated, or breaks the connection.

Co-Authored-By: Claude <noreply@anthropic.com>
Adds [storage] capacity (and init --capacity), cheap filesystem and SQLite
size measurements (fs4), optional fields on AdminStorageOverview, docs and tests.
…ote, recompress and level sampling

The level (1 to 19) and the percentage a compressed blob must save are settings of the store, shared by its clones so
they can change while the server runs. put returns the size of the object it wrote and the level it tried. recompress
rewrites an object at a higher level only when that makes it at least 1% smaller, after checking the stored bytes and
the new object against the ID, with the backend's atomic replace, and never touches an object it cannot read.
Backends can report an object's length without reading it.
…bs, keep small server values

blobs gets physical_size (the stored object's length) and compress_level (the zstd level it was written or tried at),
both NULL for blobs from before they were tracked, so a total says how much of the data it knows. blob_page walks
all blobs in pages, blob_from picks one by ID for sampling, and server_state holds small server-owned values.
…ecompress, estimate and measure commands

Uploads record the physical size next to the logical one. recompress rewrites objects at a higher level where that
saves more, resumable and safe while the server runs; estimate samples stored chunks read only and keeps the result
for the admin overview; measure fills in sizes of blobs from before they were tracked.
The compression stack (zstd blobs, settable level, recompress, estimate) on top of the admin storage overview. Where the
two met: the store keeps get_raw_limited and size_raw for the listing and verify work and gains decompression in
get_sized, which reports the logical size (the plain bytes plus the format byte) whatever the format, so verify still
compares it with blobs.stored_size; stored_size stays the object's length on the backend. The migration is numbered 0015
after the stack's 0014.
…est that sets the level

AdminCompression (algorithm, level in use, the config's level and the admin's, minimum saving, logical and physical
sizes with the share they are known for, the saving, the last estimate) is an optional part of AdminStorageOverview,
left out by an older server. AdminSetCompressionLevel is the body of PUT /admin/storage/compression. The audit log's
details get level_from and level_to, and admin.compression_changed joins the list of event names.
admin_set_compression_level checks the acting admin again inside the transaction like every admin change, keeps the level
in server_state and writes an admin.compression_changed event with both levels in the same transaction, none when the
level does not change.
…et the level

GET /admin/storage gets a compression part (totals over the recorded sizes, kept for 30 seconds; the last estimate).
PUT /admin/storage/compression takes a level from 1 to 19 with the admin's password, applies it to new writes at once, keeps
it in the database, and takes it again at start. compression = "none" in the config still wins.
… route; put compression under [storage]; the PUT answers from fresh totals
…ute table has 13 paths; the PUT's totals are not kept for the overview
…ze, and measure goes on past a failing blob

store recompress records its result only if the row still holds the size the
pass read (Meta::set_physical_if), so an upload that replaced the object in
between keeps its own size and level. store measure counts and lists blobs
it can't measure like recompress does, finishes the pass and exits with an
error.
…evel outside 1 to 19 is ignored at start

Concurrent PUTs could leave the store at another level than the database
holds, because the store was set after the commit, outside the transaction.
A mutex now covers both. At start, a stored value outside 1 to 19 is ignored
with a warning, like the overview does, instead of being clamped.
… measure test fits this branch's Backend trait
The estimate was stored as JSON with the measured floats as decimal text, and serde_json without float_roundtrip
does not promise that a parsed f64 has the bits that were written. A compress speed therefore sometimes came back
one bit off, which made estimating_is_read_only_and_remembered fail now and then on both backends.

The estimate's saving and speed numbers are now rounded to hundredths when the estimate is built and stored as whole
hundredths, so the stored value reads back to the very same f64. An estimate stored as plain decimals by an earlier
server is still read. The CLI and admin output are unchanged apart from the speed in KiB/s being taken from the
rounded value. A unit test sends 10 000 pseudo-random speeds and percentages through write and parse.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant