Skip to content

Backups: live progress, failure-matrix fixes and MinIO/R2 guide (#1-#6) - #579

Merged
yifanxuaaa merged 6 commits into
layerfs_v2_0_0from
issue5-backup-progress
Nov 5, 2026
Merged

yifanxuaaa merged 6 commits into
layerfs_v2_0_0from
issue5-backup-progress

Conversation

@yifanxuaaa

@yifanxuaaa yifanxuaaa commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Backup verification for layerfs-project #1–#6: live progress, fixes for the defects the MinIO/R2 failure matrix found, and a tested setup guide.

Commits

Commit What Production LOC
6a005f2650 live byte progress for backup create / restore (#5) see commit
9767c5f335 bound stalled sends; a lost completion with a lost abort stays Unknown; MinIO max object and R2 part limit corrected (#4, #6) 137634 → 137670 (+36)
282ea970e8 one create per Store per home (Busy), so two creates can't replace the staged snapshot under each other; the next restore removes a killed restore's staging; refusals drop the stale temp-file path 137670 → 137734 (+64)
c51d04b942 a 4xx reply sent before the server stops reading the body (a key without write access) is a definite refusal, not Unknown 137734 → 137760 (+26)
85f36421ab guide: docs/general/backups-minio-r2.md (#3) delta 0

Each fix has a test that fails without it (outcomes.rs, backup_refusals.rs). No tests deleted.

Verification

  • Live runs: the full backup-verify matrix (backup-verify: LayerFS backups on MinIO and R2 layerfs-benchmark#1) passed on MinIO and R2 at 282ea970e8; the cases the last fix touches were run again at c51d04b942.
  • Tests: on macOS, backup_refusals, outcomes and clippy (layerfs-control, layerfs-api, layerfs-store-backup --features tls) pass. On Linux against MinIO, backup_family, backup_refusals and backup_progress pass.
  • CI: the test job reports 5 failures (sandbox_direct, workspace_names ×2, machine_host_start, sqlite_publication). They fail identically on other branches from the same base (e.g. fix/disable-firecracker-ssh), so they don't come from this change.

Not run

R2 token scoping, R2 jurisdictions, HTTPS MinIO, pieces and TiB-scale objects. Durable Stores are disabled by the owner.

🤖 Generated with Claude Code

yifanxuaaa and others added 5 commits November 6, 2026 04:42
backup.create and backup.restore now send { phase, done, total, pieces }
progress, the store push/pull shape plus pieces. The bucket library already
counted bytes and pieces per thread, but nothing in the API layer observed
them, so --progress showed phase names only. A new backup/progress.rs runs
the transfer on one scoped thread with a recorder that keeps each phase's
exact last count and every piece count; the calling thread sends them at
most every 250 ms and once more when the transfer ends, however it ends.
Updates are deduplicated; a panic is resumed; nothing is retried.

Phases: create check, snapshot, send (worker only, now with bytes), check,
hash, upload; restore check, download, verify, install. A phase at its
total is not success: the result is. A failed transfer ends on its partial
count.

Refs Ephemeral-AI-Lab/layerfs-project#5.

Validation: layerfs-control backup_progress 6/6 (macOS and Linux);
layerfs-api backup_progress 1/1 and backup_family 4/4 against a private
pinned MinIO (Linux, multipart with 5 MiB parts). Real CLI on Linux: a
101,969,920-byte Store showed hash/upload/download updates every ~250 ms,
each ending exactly at the result's bytes; verify/install after download;
nothing on stderr without --progress; killing MinIO mid-upload ended at
67,108,864 bytes with exit 125 and no result, the previous backup intact.
Clippy all targets (layerfs-control, layerfs-api), rustfmt and the product
boundary check passed. Cloudflare R2 (CLI built with --features tls,
16 MiB parts): connect, list, create and restore of a 51,081,216-byte Store
exited 0 with matching sha256; hash/upload/download updates ended exactly
at the result's bytes; keys absent from all output. A network cut while
awaiting a part's reply ended after the 60 s read timeout (exit 125, remote
outcome unknown, abort failed); a cut mid-body had not ended 105 s later
(left for #6); the previous backup stayed current both times. Pieces
(files above 4.995 TiB on R2, 50 TiB on MinIO) covered by the logic test
only. Snapshot bytes are not counted (stage drops its callback). Durable
NOT_RUN: disabled by owner. No tests deleted.

Production LOC: 137478 -> 137634 (delta +156)
Scope: first-party core/engine/connect/API Rust, runtime SQL, published JS,
Python implementation and handwritten stubs. Tests, tools, docs, generated
stubs and third-party code excluded. api 20291 -> 20447 (layerfs-control
17649 -> 17805: progress.rs +142 new, create.rs +10, restore.rs +5,
family.rs -3, mod.rs +2); core 38400, engine 62799, connect 15988
unchanged. Same counter for the exact first-parent and staged trees:
tools/production_loc.py blob 31b3cc2.
Method: git archive HEAD and git checkout-index of the staged tree into
empty directories, then python3 tools/production_loc.py --root <tree> --json.
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fit limits

Four bucket-library fixes found while verifying MinIO and Cloudflare R2
backups (layerfs-project #4, #6):

- A request body buffer (256 KiB) that takes longer than the I/O timeout
  to send now fails the request as Uncertain. Each socket write's 60 s
  timeout restarted whenever a few bytes moved, so a paused MinIO that
  took in 256 KiB a minute held an upload past 118 s (observed).
- A multipart completion whose reply is lost, with its abort failing too,
  is now Uncertain (the object may have been replaced), not PartsLeft,
  which the API reports as Failed. A failed part with a failed abort stays
  PartsLeft: the object was not replaced.
- MinIO's largest single object is what one multipart upload carries,
  10,000 parts of 5 GiB (48.83 TiB), below its 50 TiB object limit; a
  larger file goes up as pieces instead of failing in part_size.
- A part is one request, so a part size above the provider's request limit
  (R2: 4.995 GiB) is refused Invalid at connect and before any upload.

Refs Ephemeral-AI-Lab/layerfs-project#4, Ephemeral-AI-Lab/layerfs-project#6.

Validation: new tests/outcomes.rs 5/5 (fake server: lost completion and
abort, lost part and abort, trickle peer, MinIO and R2 limits) on macOS
and Linux; 4 of them fail without the fixes. keep_alive 4/4; MinIO tests
9/9 against the pinned MinIO (Linux, --include-ignored); the crate's full
macOS suite passes. Clippy all targets with tls, rustfmt passed. Durable
NOT_RUN: disabled by owner. No tests deleted.

Production LOC: 137634 -> 137670 (delta +36)
Scope: first-party core/engine/connect/API Rust, runtime SQL, published JS,
Python implementation and handwritten stubs. Tests, tools, docs, generated
stubs and third-party code excluded. connect 15988 -> 16024
(layerfs-store-backup: http.rs +16, multipart.rs +14, config.rs +6,
minio/provider.rs 0); api 20447, core 38400, engine 62799 unchanged.
Same counter for the exact first-parent and staged trees:
tools/production_loc.py blob 31b3cc2.
Method: git archive HEAD and git checkout-index of the staged tree into
empty directories, then python3 tools/production_loc.py --root <tree> --json.
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the backup-verify failure matrix (layerfs-project #6) on MinIO:

- Two creates of one Store in a home could both hold its (shared) lease,
  and the `stage` step removes `<store>.backup` before writing a new
  snapshot there. A second create starting while the first uploads from
  that path could replace the file under it, so mixed bytes could go over
  the last good backup. `backup.create` now holds an `flock` on
  `store/<s>/backup.lock`, taken without waiting, for its whole run; a
  second create is refused `Busy` before it stages or sends anything.
- A restore killed mid-download left `store/.restore-<random>/` with its
  partial file for good (33 MB in the run). Each restore now holds an
  `flock` on its folder's `lock` file; making the folder and removing
  folders whose lock can be taken (their restore has ended) happen under
  the home lock, so no folder is swept between its creation and its lock.
  A folder with a held lock or no lock file is left alone.
- A failed load's refusal named the temporary file it left, a path the
  restore's staging folder had already removed, under a "Store pull:"
  prefix. The refusal now carries the cause alone.

Tests: backup_refusals gains the_next_restore_removes_a_killed_restores_folder
and a_second_create_of_a_store_is_busy; both fail with the fixes set aside.
No tests deleted.

Production LOC: 137670 -> 137734 (delta +64)
  scope: first-party product source (core, engine, api, connect), counted by
  tools/production_loc.py on the parent tree and the staged tree (nonblank,
  non-comment lines; tests, docs and tools excluded)
  api 20447 -> 20511 (layerfs-control 17805 -> 17869:
    backup/create.rs 198 -> 226, backup/restore.rs 222 -> 258)
  connect, core, engine unchanged
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the backup-verify permission check (layerfs-project #3) on MinIO:
a create with a key that may read but not write was reported `Unknown`
("send: Connection reset by peer"), telling the caller the backup might
have landed. MinIO answers a refused PUT with 403 after the head and stops
reading the body, so the client's send fails. When a body send fails with
a reset or broken pipe, the exchange now reads the reply already on the
connection: a 4xx answer is returned as that refusal (nothing was stored);
anything else, no answer, or a send timeout stays `Uncertain`. Nothing is
resent.

Tests: outcomes gains a_refusal_sent_before_the_body_is_read_is_definite;
it fails with the fix set aside. No tests deleted.

Production LOC: 137734 -> 137760 (delta +26)
  scope: first-party product source (core, engine, api, connect), counted by
  tools/production_loc.py on the parent tree and the staged tree (nonblank,
  non-comment lines; tests, docs and tools excluded)
  connect 16024 -> 16050 (layerfs-store-backup 3263 -> 3289:
    bucket/s3_api/http.rs 320 -> 346)
  api, core, engine unchanged
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tested guide from a clean Linux home to a Store backed up and restored
on MinIO or Cloudflare R2: TLS build, bucket and key setup, key
references, commands with their real outputs, progress, sizes and parts,
replacement and retention, tested minimum MinIO permissions, rotation,
troubleshooting by refusal, cleanup of each leftover kind, known limits.
Outputs come from the backup-verify receipts at c51d04b.

Production LOC: 137760 -> 137760 (delta 0; docs only)
  scope and method as in the previous commit (tools/production_loc.py)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa yifanxuaaa changed the title feat(backup): live byte progress for backup create and restore Backups: live progress, failure-matrix fixes and MinIO/R2 guide (#1-#6) Nov 5, 2026
…e base

CI `test` on PR #579 failed six tests.

- outcomes a_peer_taking_the_body_in_a_trickle_fails_the_request (this
  branch): on Linux the trickling peer fills the socket buffer, so one
  write waits out its own timeout and returns EAGAIN before the per-buffer
  stall check runs. The request was already bounded and `Uncertain`, but
  was labelled "send: Resource temporarily unavailable". A write that
  times out (WouldBlock/TimedOut) is now reported "send stalled", like the
  per-buffer bound.
- The other five fail on layerfs_v2_0_0 itself:
  - sandbox_direct: ssh is permanently disabled (layerfs-project#7), so the
    ssh forward half could not run. The test now checks that a Sandbox on
    an ssh machine is refused `Unsupported` before the worker is asked, and
    keeps its docker and microsandbox halves (renamed
    ssh_is_refused_and_docker_records_its_published_port; helpers only the
    ssh half used removed).
  - machine_host_start (deleted): it drove a Firecracker microVM's host
    start; Firecracker is permanently disabled (layerfs-project#7) and the
    policy refusal is covered by layerfs-cli disabled_platforms and the
    layerfs-machine tests.
  - workspace_names (2 tests): 6deb7f6 refuses a recorded daemon that is
    not a file; the placeholder `/nonexistent` became an empty file in the
    test home, still never run.
  - sqlite_publication wal_full_settings_and_unavailable_engine_are_explicit:
    #535 2b removed the Postgres backend selection the first assertion set,
    leaving it to expect a plain SQLite create to fail. It now checks the
    surviving contract: a `StoreSelection::Postgres` without the
    `postgresql` feature is `BackendUnavailable`.

Checked on Linux: clippy -D warnings (layerfs-api, -control,
-persistence, -store-backup, all targets); sandbox_direct, workspace_names
(temp on ext4, as on the CI runner), sqlite_publication and outcomes pass.

Deleted tests: api/crates/layerfs-control/tests/machine_host_start.rs
(a_microvm_s_stopped_host_is_started_first).

Production LOC: 137760 -> 137772 (delta +12)
  scope: first-party product source (core, engine, api, connect), counted by
  tools/production_loc.py on the parent tree and the staged tree (nonblank,
  non-comment lines; tests, docs and tools excluded)
  connect 16050 -> 16062 (layerfs-store-backup 3289 -> 3301:
    bucket/s3_api/http.rs 346 -> 358)
  api, core, engine unchanged
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit 6d820c7 into layerfs_v2_0_0 Nov 5, 2026
4 checks passed
@yifanxuaaa
yifanxuaaa deleted the issue5-backup-progress branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant