Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/layerfs-js-sdk/references/methods/store.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ store.update(name, { machine, url, password, synchronous_commit, clear })
store.push(name, machine, { on_progress })
store.pull(name, { on_progress })
store.remove(name)
store.compact(name)
store.compact(name, { on_progress })
store.disconnect(name)
```

Expand Down Expand Up @@ -84,7 +84,7 @@ Store on another machine is read through its worker; a stopped worker is
| `push(name, machine, …)` | Sends a sqlite Store's file from `native` to `machine` | `InvalidInput` unless the Store is on `native` (pull it first). Progress `{ phase, done, total }` on `on_progress` / `--progress`: `done` and `total` are bytes during `send`, null in the other phases (`check`, `snapshot`, `verify`, `install`, `record`, `retire`). `InUse` while the Store is held (Sandbox records, processes with it open; `detail.related` lists them as `{ kind, name }`); `Capacity` when the target has less free disk than the Store plus a headroom of 1 GiB (or a tenth of the Store), checked before anything changes; `Unavailable` naming `layerfs machine start <m>` for a stopped machine; while it runs, Store calls and operations on it are `Busy`, from the caller's process too. An uncertain push whose process is gone (it ended during install or record, or a step's answer was lost) leaves every later call `Unresolved`, naming `store status` and the push that clears it: a new `push` (or `pull`) first deletes the copy on the machine the record does not name, then runs. A push whose `retire` did not finish leaves the Store in use on `machine` and `last_transfer.retired: false`; the next `pull` deletes the old copy first |
| `pull(name, …)` | Brings a sqlite Store's file from its machine to `native` | `InvalidInput` when it is already on `native`; otherwise as `push`. Between two machines: `pull`, then `push` |
| `remove(name)` | Deletes a sqlite Store with every project in it | Returns `{ removed }`. `NotFound` for an unknown name (live); `Unavailable` for a Store whose machine is unknown; `Busy` while a live process pushes, pulls or removes it. It clears an `Unresolved` Store. Backups stay |
| `compact(name)` | Rebuilds the Store from its live history, reclaiming dead bytes | The only byte-reclaiming call: no confirm, force or dry run. Returns `{ kind: "Compacted", path, generation, … }` with `dead_bytes_bound`, `dead_bytes_refined`, `logical_discarded`, `backend_reusable`, `physical_shrink`, `kept_waste`, `peak_temp_bytes`, `residue`, `estimate` and `zero_reasons` (a list of `{field, reason}`). A Store with under 10% dead packs takes the `none` path and only marks. A sqlite Store on another machine is `Unsupported` (local-only route). `MountsActive`, `Busy`, `Unresolved`, `StaleStore`, `Conflict`, `Capacity`, `Unknown` |
| `compact(name, { on_progress })` | Rebuilds the Store from its live history, reclaiming dead bytes | The only byte-reclaiming call: no confirm, force or dry run. `on_progress` gets `{ phase, done, total }` as each phase starts (`mark`, `build`, `verify`, `install` on sqlite; `mark`, `build`, `swap`, `cleanup` on PostgreSQL), with null counts. Returns `{ kind: "Compacted", path, generation, … }` with `dead_bytes_bound`, `dead_bytes_refined`, `logical_discarded`, `backend_reusable`, `physical_shrink`, `kept_waste`, `peak_temp_bytes`, `residue`, `estimate` and `zero_reasons` (a list of `{field, reason}`). A Store with under 10% dead packs takes the `none` path and only marks. A sqlite Store on another machine is `Unsupported` (local-only route). `MountsActive`, `Busy`, `Unresolved`, `StaleStore`, `Conflict`, `Capacity`, `Unknown` |
| `disconnect(name)` | Forgets a PostgreSQL Store; its data stays | `InUse` while another process holds the Store (its calls hold it while it stays open) or a Sandbox of it runs; `detail.related` lists them as `{ kind: "lease", name: <pid> }` and `{ kind: "sandbox", name }` |

Pushing or pulling a Store moves where its Sandboxes can run: a Sandbox of a sqlite Store
Expand Down
4 changes: 2 additions & 2 deletions api/crates/layerfs-api/src/store/compact.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
//! local-only (spec S6 deviation D5, `core/docs/issues/project-41/gc-42-decisions.md`);
//! this entry parses the name and hands the call over.
use crate::calls::table::Kind::Name;
use crate::calls::table::{a, call, Call};
use crate::calls::table::{a, call, Call, PROGRESS};
use crate::calls::Context;
use layerfs_bridge::control::SdkValue;
use layerfs_home::names::StoreName;
Expand All @@ -15,7 +15,7 @@ pub(crate) const CALL: Call = call(
"store",
"compact",
&[a("name", Name)],
&[],
&[PROGRESS],
"reclaim dead bytes from one Store",
run,
);
Expand Down
10 changes: 8 additions & 2 deletions api/crates/layerfs-api/tests/calls_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -567,7 +567,8 @@ fn sandbox_images_are_named_by_the_image_kind() {

#[test]
fn the_four_gc_calls_take_only_ids() {
// T11.1: the four GC calls, each one resource identifier and no option.
// T11.1: the four GC calls, each one resource identifier and no data
// option; `store.compact` alone declares `on_progress` (#42 item 45).
use layerfs_api::Kind;
let expected: &[(&str, &[(&str, Kind)])] = &[
(
Expand All @@ -586,6 +587,11 @@ fn the_four_gc_calls_take_only_ids() {
let given: Vec<(&str, Kind)> = spec.args.iter().map(|arg| (arg.name, arg.kind)).collect();
assert_eq!(given.as_slice(), args, "{name}");
assert!(spec.args.iter().all(|arg| !arg.optional), "{name}");
assert!(spec.options.is_empty(), "{name} takes no option");
let options: Vec<(&str, Kind)> = spec.options.iter().map(|o| (o.name, o.kind)).collect();
if name == "store.compact" {
assert_eq!(options, [("on_progress", Kind::Function)], "{name}");
} else {
assert!(options.is_empty(), "{name} takes no option");
}
}
}
45 changes: 44 additions & 1 deletion api/crates/layerfs-api/tests/gc_calls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,14 @@ fn the_four_gc_calls_take_ids_only() {
let given: Vec<(&str, Kind)> = spec.args.iter().map(|arg| (arg.name, arg.kind)).collect();
assert_eq!(given.as_slice(), args, "{name} arguments");
assert!(spec.args.iter().all(|arg| !arg.optional), "{name} required");
assert!(spec.options.is_empty(), "{name} has no options");
// ids only: no call takes a data option. `store.compact` alone
// declares the shared `on_progress` function (#42 item 45).
let options: Vec<(&str, Kind)> = spec.options.iter().map(|o| (o.name, o.kind)).collect();
if name == "store.compact" {
assert_eq!(options, [("on_progress", Kind::Function)], "{name}");
} else {
assert!(options.is_empty(), "{name} has no options");
}
}
}

Expand Down Expand Up @@ -273,6 +280,42 @@ fn project_init_after_delete_is_nametaken_deleted() {
let _ = std::fs::remove_dir_all(folder);
}

/// `store.compact` sends each phase as a progress event before it answers:
/// `{ progress: { phase, done, total } }` with null counts.
#[test]
fn store_compact_sends_each_phase_as_a_progress_event() {
let (home, folder) = project();
let compact = || {
let mut phases = Vec::new();
let report = layerfs_api::call(
Some(&home),
"store.compact",
map(&[("name", t("shop"))]),
&mut |event| {
let progress = event.get("progress").expect("a progress event").clone();
assert_eq!(progress.get("done"), Some(&SdkValue::Null), "{progress:?}");
assert_eq!(progress.get("total"), Some(&SdkValue::Null), "{progress:?}");
phases.push(progress.get_text("phase").expect("phase").to_owned());
},
)
.unwrap_or_else(|refusal| panic!("store.compact: {refusal:?}"));
(report.get_text("path").expect("path").to_owned(), phases)
};
// nothing is dead yet: the run stops after its mark
let (path, phases) = compact();
assert_eq!(
(path.as_str(), phases.as_slice()),
("none", ["mark".to_owned()].as_slice())
);
// a deleted project makes the next run a rebuild, through all four phases
ok(&home, "project.delete", &[p()]);
let (path, phases) = compact();
assert_eq!(path, "rebuild");
assert_eq!(phases, ["mark", "build", "verify", "install"]);
std::fs::remove_dir_all(home).unwrap();
let _ = std::fs::remove_dir_all(folder);
}

#[test]
fn store_compact_refuses_an_unknown_store_and_reports_a_run() {
let home = tempdir("compact-none");
Expand Down
5 changes: 3 additions & 2 deletions api/crates/layerfs-api/tests/progress_options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ fn declaring(option: &str) -> Vec<String> {
}

#[test]
fn eleven_calls_declare_on_progress_and_one_declares_the_exec_sinks() {
fn twelve_calls_declare_on_progress_and_one_declares_the_exec_sinks() {
assert_eq!(
declaring("on_progress"),
[
Expand All @@ -28,6 +28,7 @@ fn eleven_calls_declare_on_progress_and_one_declares_the_exec_sinks() {
"machine.create",
"store.push",
"store.pull",
"store.compact",
"backup.create",
"backup.restore"
]
Expand All @@ -44,7 +45,7 @@ fn eleven_calls_declare_on_progress_and_one_declares_the_exec_sinks() {
.count()
})
.sum();
assert_eq!(functions, 11 + 2);
assert_eq!(functions, 12 + 2);
}

#[test]
Expand Down
1 change: 1 addition & 0 deletions api/crates/layerfs-cli/tests/progress.rs
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ fn only_calls_declaring_on_progress_accept_the_cli_switch() {
["project", "init", "--help"],
["store", "push", "--help"],
["store", "pull", "--help"],
["store", "compact", "--help"],
["backup", "create", "--help"],
["backup", "restore", "--help"],
["workspace", "exec", "--help"],
Expand Down
7 changes: 6 additions & 1 deletion api/node/src/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,11 @@ export interface StorePullOptions extends CallOptions {
on_progress?: ProgressHandler;
}

/** Options of `store.compact`; an unknown option is refused. */
export interface StoreCompactOptions extends CallOptions {
on_progress?: ProgressHandler;
}

/** Options of `backup.connect`; an unknown option is refused. */
export interface BackupConnectOptions extends CallOptions {
key?: string;
Expand Down Expand Up @@ -647,7 +652,7 @@ export interface StoreFamily {
/** delete a sqlite Store with every project in it */
remove(name: string, options?: CallOptions): Promise<Result>;
/** reclaim dead bytes from one Store */
compact(name: string, options?: CallOptions): Promise<Result>;
compact(name: string, options?: StoreCompactOptions): Promise<Result>;
/** forget a PostgreSQL Store; its data stays */
disconnect(name: string, options?: CallOptions): Promise<Result>;
}
Expand Down
6 changes: 3 additions & 3 deletions api/node/test/progress.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,11 @@ function folderOf(files) {

const calling = (name) => table.filter((c) => c.options.some((o) => o.name === name && o.kind === "function")).map((c) => `${c.family}.${c.call}`);

test("the table: eleven calls take on_progress and one takes the exec sinks", () => {
assert.deepEqual(calling("on_progress"), ["project.init", "project.export", "sandbox.create", "workspace.exec", "machine.load", "machine.start", "machine.create", "store.push", "store.pull", "backup.create", "backup.restore"]);
test("the table: twelve calls take on_progress and one takes the exec sinks", () => {
assert.deepEqual(calling("on_progress"), ["project.init", "project.export", "sandbox.create", "workspace.exec", "machine.load", "machine.start", "machine.create", "store.push", "store.pull", "store.compact", "backup.create", "backup.restore"]);
assert.deepEqual(calling("stdout"), ["workspace.exec"]);
assert.deepEqual(calling("stderr"), ["workspace.exec"]);
assert.equal(table.flatMap((c) => c.options).filter((o) => o.kind === "function").length, 11 + 2);
assert.equal(table.flatMap((c) => c.options).filter((o) => o.kind === "function").length, 12 + 2);
});

test("project.init in this process: check, install, record with null counts, while it runs, before it settles", { timeout: 90_000 }, async () => {
Expand Down
8 changes: 7 additions & 1 deletion api/python/src/layerfs/_client.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -574,7 +574,13 @@ class StoreFamily:
"""bring a sqlite Store from its machine to native"""
def remove(self, name: str, *, log_tag: str | None = None) -> Result:
"""delete a sqlite Store with every project in it"""
def compact(self, name: str, *, log_tag: str | None = None) -> Result:
def compact(
self,
name: str,
*,
on_progress: Callable[[Progress], object] = ...,
log_tag: str | None = None,
) -> Result:
"""reclaim dead bytes from one Store"""
def disconnect(self, name: str, *, log_tag: str | None = None) -> Result:
"""forget a PostgreSQL Store; its data stays"""
Expand Down
8 changes: 7 additions & 1 deletion api/python/src/layerfs/aio.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -585,7 +585,13 @@ class AsyncStoreFamily:
"""bring a sqlite Store from its machine to native"""
async def remove(self, name: str, *, log_tag: str | None = None) -> Result:
"""delete a sqlite Store with every project in it"""
async def compact(self, name: str, *, log_tag: str | None = None) -> Result:
async def compact(
self,
name: str,
*,
on_progress: Callable[[Progress], object] = ...,
log_tag: str | None = None,
) -> Result:
"""reclaim dead bytes from one Store"""
async def disconnect(self, name: str, *, log_tag: str | None = None) -> Result:
"""forget a PostgreSQL Store; its data stays"""
Expand Down
3 changes: 2 additions & 1 deletion api/python/test/test_progress.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ def setUp(self):
self.addCleanup(shutil.rmtree, self.home, True)
self.addCleanup(shutil.rmtree, self.folder, True)

def test_the_table_eleven_calls_take_on_progress(self):
def test_the_table_twelve_calls_take_on_progress(self):
self.assertEqual(
declaring("on_progress"),
[
Expand All @@ -76,6 +76,7 @@ def test_the_table_eleven_calls_take_on_progress(self):
"machine.create",
"store.push",
"store.pull",
"store.compact",
"backup.create",
"backup.restore",
],
Expand Down
71 changes: 41 additions & 30 deletions control/crates/layerfs-store/src/compact.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ use crate::open_args::OpenArgs;
use crate::place::{self, Executor};
use crate::postgresql;
use crate::record::{StoreRecord, StoreRef};
use crate::sqlite::family::progress;
use layerfs_bridge::control::{SdkRefusal, SdkValue};
use layerfs_history::GcScopeKind;
use layerfs_home::events::Events;
Expand All @@ -28,8 +29,7 @@ use layerfs_home::outcome::{failed, refused, Outcome};
use layerfs_persistence::compact_metrics::{self, CompactRefusal};
use layerfs_persistence::PostgresHandles;
use layerfs_persistence::{
CompactFailpoint, CompactReport, CompactRequest, PersistenceConfig, SqliteHandles,
SqlitePersistenceProfile,
CompactFailpoint, CompactRequest, PersistenceConfig, SqliteHandles, SqlitePersistenceProfile,
};
use layerfs_store_service::history::{admission, lookup};
use std::path::Path;
Expand Down Expand Up @@ -72,8 +72,10 @@ fn crash_failpoint() -> Option<CompactFailpoint> {
}

/// `store.compact(name)`: reclaim one Store's dead bytes, answering
/// the §11 report.
pub fn compact(home: &Path, _events: &mut Events<'_>, store: &str) -> Outcome {
/// the §11 report. Each phase is sent to `events` as it starts, as
/// `{ phase, done, total }` with null counts (the shape `store.push` sends);
/// the op record `store.status` reads is written by the run, not from here.
pub fn compact(home: &Path, events: &mut Events<'_>, store: &str) -> Outcome {
let home = home.to_path_buf();
// A live operation is `Busy`, a dead owner's record `Unresolved`; the
// refusal is a bounded metric label and never changes what is returned.
Expand All @@ -86,7 +88,7 @@ pub fn compact(home: &Path, _events: &mut Events<'_>, store: &str) -> Outcome {
};
let store_ref = record.store_ref()?.clone();
if matches!(store_ref, StoreRef::Postgres { .. }) {
return compact_postgresql(&record, &home, store);
return compact_postgresql(&record, &home, events, store);
}
match place::plan(&home, &store_ref)? {
Executor::Local => {}
Expand Down Expand Up @@ -134,31 +136,37 @@ pub fn compact(home: &Path, _events: &mut Events<'_>, store: &str) -> Outcome {
layerfs_bridge::sdk_values::hex(&op_id)
));
let receipt = handles
.compact(&CompactRequest {
op_id,
owner_pid: Some(std::process::id()),
owner_pid_start,
owner_pid_boot,
owner_runtime: Some(owner_runtime),
owner_boot: Some(owner_boot),
owner_live: Some(layerfs_home::process::recorded_owner_alive),
failpoint: crash_failpoint(),
..CompactRequest::default()
})
.compact_observed(
&CompactRequest {
op_id,
owner_pid: Some(std::process::id()),
owner_pid_start,
owner_pid_boot,
owner_runtime: Some(owner_runtime),
owner_boot: Some(owner_boot),
owner_live: Some(layerfs_home::process::recorded_owner_alive),
failpoint: crash_failpoint(),
..CompactRequest::default()
},
&mut |phase| progress(events, phase, None),
)
.map_err(|error| {
lookup::history_in(error, GcScopeKind::Store, store, None, Some(&op_id))
})?;
// One serializer for the shared §11 report: the SQLite receipt projects
// into `CompactReport`, and T9's PostgreSQL receipt will do the same.
Ok(crate::report::compact_report(&CompactReport::from(receipt)))
Ok(crate::report::compact_report(&receipt))
}

/// The §8 PostgreSQL path: dial the Store's recorded URL, open its schema as
/// this home's authority, and run Phase M/A/B/C. The Store is usable at
/// G + 1 from the swap on; a lost reply is judged by the next call (§9).
///
/// The PostgreSQL engine is always built beside SQLite (#726).
fn compact_postgresql(record: &StoreRecord, home: &Path, store: &str) -> Outcome {
fn compact_postgresql(
record: &StoreRecord,
home: &Path,
events: &mut Events<'_>,
store: &str,
) -> Outcome {
let url = crate::record::url(record)?;
let id = identity::identity(&OpenArgs {
home: Some(home.to_path_buf()),
Expand Down Expand Up @@ -189,19 +197,22 @@ fn compact_postgresql(record: &StoreRecord, home: &Path, store: &str) -> Outcome
layerfs_bridge::sdk_values::hex(&op_id)
));
let receipt = handles
.compact(&CompactRequest {
op_id,
owner_pid: Some(std::process::id()),
owner_pid_start,
owner_pid_boot,
owner_runtime: Some(owner_runtime),
owner_boot: Some(owner_boot),
..CompactRequest::default()
})
.compact_observed(
&CompactRequest {
op_id,
owner_pid: Some(std::process::id()),
owner_pid_start,
owner_pid_boot,
owner_runtime: Some(owner_runtime),
owner_boot: Some(owner_boot),
..CompactRequest::default()
},
&mut |phase| progress(events, phase, None),
)
.map_err(|error| {
lookup::history_in(error, GcScopeKind::Store, store, None, Some(&op_id))
})?;
Ok(crate::report::compact_report(&CompactReport::from(receipt)))
Ok(crate::report::compact_report(&receipt))
}

/// An open refusal before the run: `Busy` is definite, everything else a
Expand Down
Loading