Repository navigation
Backups: live progress, failure-matrix fixes and MinIO/R2 guide (#1-#6) - #579
Merged
Merged
Conversation
backup.create and backup.restore now send { phase, done, total, pieces }
progress, the store push/pull shape plus pieces. The bucket library already
counted bytes and pieces per thread, but nothing in the API layer observed
them, so --progress showed phase names only. A new backup/progress.rs runs
the transfer on one scoped thread with a recorder that keeps each phase's
exact last count and every piece count; the calling thread sends them at
most every 250 ms and once more when the transfer ends, however it ends.
Updates are deduplicated; a panic is resumed; nothing is retried.
Phases: create check, snapshot, send (worker only, now with bytes), check,
hash, upload; restore check, download, verify, install. A phase at its
total is not success: the result is. A failed transfer ends on its partial
count.
Refs Ephemeral-AI-Lab/layerfs-project#5.
Validation: layerfs-control backup_progress 6/6 (macOS and Linux);
layerfs-api backup_progress 1/1 and backup_family 4/4 against a private
pinned MinIO (Linux, multipart with 5 MiB parts). Real CLI on Linux: a
101,969,920-byte Store showed hash/upload/download updates every ~250 ms,
each ending exactly at the result's bytes; verify/install after download;
nothing on stderr without --progress; killing MinIO mid-upload ended at
67,108,864 bytes with exit 125 and no result, the previous backup intact.
Clippy all targets (layerfs-control, layerfs-api), rustfmt and the product
boundary check passed. Cloudflare R2 (CLI built with --features tls,
16 MiB parts): connect, list, create and restore of a 51,081,216-byte Store
exited 0 with matching sha256; hash/upload/download updates ended exactly
at the result's bytes; keys absent from all output. A network cut while
awaiting a part's reply ended after the 60 s read timeout (exit 125, remote
outcome unknown, abort failed); a cut mid-body had not ended 105 s later
(left for #6); the previous backup stayed current both times. Pieces
(files above 4.995 TiB on R2, 50 TiB on MinIO) covered by the logic test
only. Snapshot bytes are not counted (stage drops its callback). Durable
NOT_RUN: disabled by owner. No tests deleted.
Production LOC: 137478 -> 137634 (delta +156)
Scope: first-party core/engine/connect/API Rust, runtime SQL, published JS,
Python implementation and handwritten stubs. Tests, tools, docs, generated
stubs and third-party code excluded. api 20291 -> 20447 (layerfs-control
17649 -> 17805: progress.rs +142 new, create.rs +10, restore.rs +5,
family.rs -3, mod.rs +2); core 38400, engine 62799, connect 15988
unchanged. Same counter for the exact first-parent and staged trees:
tools/production_loc.py blob 31b3cc2.
Method: git archive HEAD and git checkout-index of the staged tree into
empty directories, then python3 tools/production_loc.py --root <tree> --json.
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fit limits Four bucket-library fixes found while verifying MinIO and Cloudflare R2 backups (layerfs-project #4, #6): - A request body buffer (256 KiB) that takes longer than the I/O timeout to send now fails the request as Uncertain. Each socket write's 60 s timeout restarted whenever a few bytes moved, so a paused MinIO that took in 256 KiB a minute held an upload past 118 s (observed). - A multipart completion whose reply is lost, with its abort failing too, is now Uncertain (the object may have been replaced), not PartsLeft, which the API reports as Failed. A failed part with a failed abort stays PartsLeft: the object was not replaced. - MinIO's largest single object is what one multipart upload carries, 10,000 parts of 5 GiB (48.83 TiB), below its 50 TiB object limit; a larger file goes up as pieces instead of failing in part_size. - A part is one request, so a part size above the provider's request limit (R2: 4.995 GiB) is refused Invalid at connect and before any upload. Refs Ephemeral-AI-Lab/layerfs-project#4, Ephemeral-AI-Lab/layerfs-project#6. Validation: new tests/outcomes.rs 5/5 (fake server: lost completion and abort, lost part and abort, trickle peer, MinIO and R2 limits) on macOS and Linux; 4 of them fail without the fixes. keep_alive 4/4; MinIO tests 9/9 against the pinned MinIO (Linux, --include-ignored); the crate's full macOS suite passes. Clippy all targets with tls, rustfmt passed. Durable NOT_RUN: disabled by owner. No tests deleted. Production LOC: 137634 -> 137670 (delta +36) Scope: first-party core/engine/connect/API Rust, runtime SQL, published JS, Python implementation and handwritten stubs. Tests, tools, docs, generated stubs and third-party code excluded. connect 15988 -> 16024 (layerfs-store-backup: http.rs +16, multipart.rs +14, config.rs +6, minio/provider.rs 0); api 20447, core 38400, engine 62799 unchanged. Same counter for the exact first-parent and staged trees: tools/production_loc.py blob 31b3cc2. Method: git archive HEAD and git checkout-index of the staged tree into empty directories, then python3 tools/production_loc.py --root <tree> --json. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the backup-verify failure matrix (layerfs-project #6) on MinIO: - Two creates of one Store in a home could both hold its (shared) lease, and the `stage` step removes `<store>.backup` before writing a new snapshot there. A second create starting while the first uploads from that path could replace the file under it, so mixed bytes could go over the last good backup. `backup.create` now holds an `flock` on `store/<s>/backup.lock`, taken without waiting, for its whole run; a second create is refused `Busy` before it stages or sends anything. - A restore killed mid-download left `store/.restore-<random>/` with its partial file for good (33 MB in the run). Each restore now holds an `flock` on its folder's `lock` file; making the folder and removing folders whose lock can be taken (their restore has ended) happen under the home lock, so no folder is swept between its creation and its lock. A folder with a held lock or no lock file is left alone. - A failed load's refusal named the temporary file it left, a path the restore's staging folder had already removed, under a "Store pull:" prefix. The refusal now carries the cause alone. Tests: backup_refusals gains the_next_restore_removes_a_killed_restores_folder and a_second_create_of_a_store_is_busy; both fail with the fixes set aside. No tests deleted. Production LOC: 137670 -> 137734 (delta +64) scope: first-party product source (core, engine, api, connect), counted by tools/production_loc.py on the parent tree and the staged tree (nonblank, non-comment lines; tests, docs and tools excluded) api 20447 -> 20511 (layerfs-control 17805 -> 17869: backup/create.rs 198 -> 226, backup/restore.rs 222 -> 258) connect, core, engine unchanged reference absent (retired R9c; last counted 65,417 at tree 498dd19) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the backup-verify permission check (layerfs-project #3) on MinIO: a create with a key that may read but not write was reported `Unknown` ("send: Connection reset by peer"), telling the caller the backup might have landed. MinIO answers a refused PUT with 403 after the head and stops reading the body, so the client's send fails. When a body send fails with a reset or broken pipe, the exchange now reads the reply already on the connection: a 4xx answer is returned as that refusal (nothing was stored); anything else, no answer, or a send timeout stays `Uncertain`. Nothing is resent. Tests: outcomes gains a_refusal_sent_before_the_body_is_read_is_definite; it fails with the fix set aside. No tests deleted. Production LOC: 137734 -> 137760 (delta +26) scope: first-party product source (core, engine, api, connect), counted by tools/production_loc.py on the parent tree and the staged tree (nonblank, non-comment lines; tests, docs and tools excluded) connect 16024 -> 16050 (layerfs-store-backup 3263 -> 3289: bucket/s3_api/http.rs 320 -> 346) api, core, engine unchanged reference absent (retired R9c; last counted 65,417 at tree 498dd19) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tested guide from a clean Linux home to a Store backed up and restored on MinIO or Cloudflare R2: TLS build, bucket and key setup, key references, commands with their real outputs, progress, sizes and parts, replacement and retention, tested minimum MinIO permissions, rotation, troubleshooting by refusal, cleanup of each leftover kind, known limits. Outputs come from the backup-verify receipts at c51d04b. Production LOC: 137760 -> 137760 (delta 0; docs only) scope and method as in the previous commit (tools/production_loc.py) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e base CI `test` on PR #579 failed six tests. - outcomes a_peer_taking_the_body_in_a_trickle_fails_the_request (this branch): on Linux the trickling peer fills the socket buffer, so one write waits out its own timeout and returns EAGAIN before the per-buffer stall check runs. The request was already bounded and `Uncertain`, but was labelled "send: Resource temporarily unavailable". A write that times out (WouldBlock/TimedOut) is now reported "send stalled", like the per-buffer bound. - The other five fail on layerfs_v2_0_0 itself: - sandbox_direct: ssh is permanently disabled (layerfs-project#7), so the ssh forward half could not run. The test now checks that a Sandbox on an ssh machine is refused `Unsupported` before the worker is asked, and keeps its docker and microsandbox halves (renamed ssh_is_refused_and_docker_records_its_published_port; helpers only the ssh half used removed). - machine_host_start (deleted): it drove a Firecracker microVM's host start; Firecracker is permanently disabled (layerfs-project#7) and the policy refusal is covered by layerfs-cli disabled_platforms and the layerfs-machine tests. - workspace_names (2 tests): 6deb7f6 refuses a recorded daemon that is not a file; the placeholder `/nonexistent` became an empty file in the test home, still never run. - sqlite_publication wal_full_settings_and_unavailable_engine_are_explicit: #535 2b removed the Postgres backend selection the first assertion set, leaving it to expect a plain SQLite create to fail. It now checks the surviving contract: a `StoreSelection::Postgres` without the `postgresql` feature is `BackendUnavailable`. Checked on Linux: clippy -D warnings (layerfs-api, -control, -persistence, -store-backup, all targets); sandbox_direct, workspace_names (temp on ext4, as on the CI runner), sqlite_publication and outcomes pass. Deleted tests: api/crates/layerfs-control/tests/machine_host_start.rs (a_microvm_s_stopped_host_is_started_first). Production LOC: 137760 -> 137772 (delta +12) scope: first-party product source (core, engine, api, connect), counted by tools/production_loc.py on the parent tree and the staged tree (nonblank, non-comment lines; tests, docs and tools excluded) connect 16050 -> 16062 (layerfs-store-backup 3289 -> 3301: bucket/s3_api/http.rs 346 -> 358) api, core, engine unchanged reference absent (retired R9c; last counted 65,417 at tree 498dd19) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backup verification for layerfs-project #1–#6: live progress, fixes for the defects the MinIO/R2 failure matrix found, and a tested setup guide.
Commits
6a005f2650backup create/restore(#5)9767c5f335Unknown; MinIO max object and R2 part limit corrected (#4, #6)282ea970e8Busy), so two creates can't replace the staged snapshot under each other; the next restore removes a killed restore's staging; refusals drop the stale temp-file pathc51d04b942Unknown85f36421abdocs/general/backups-minio-r2.md(#3)Each fix has a test that fails without it (
outcomes.rs,backup_refusals.rs). No tests deleted.Verification
282ea970e8; the cases the last fix touches were run again atc51d04b942.backup_refusals,outcomesand clippy (layerfs-control,layerfs-api,layerfs-store-backup --features tls) pass. On Linux against MinIO,backup_family,backup_refusalsandbackup_progresspass.testjob reports 5 failures (sandbox_direct,workspace_names×2,machine_host_start,sqlite_publication). They fail identically on other branches from the same base (e.g.fix/disable-firecracker-ssh), so they don't come from this change.Not run
R2 token scoping, R2 jurisdictions, HTTPS MinIO, pieces and TiB-scale objects. Durable Stores are disabled by the owner.
🤖 Generated with Claude Code