From d19d98f1385ed38f1ee284dc1c76f2edc0797aa8 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Tue, 17 Nov 2026 23:45:04 +0800 Subject: [PATCH 01/31] perf(gc): mark reads record headers, not pack bodies (layerfs-project#41) Mark needs one fact about a file payload: its delta base. It found it by reading and hashing each live pack whole. It now reads the group's record directory and each record's first 41 bytes. Changes - core delta/record.rs: header_base parses a payload record's base from its header alone and accepts and refuses exactly what stored_base does. - core delta/edge.rs (new): RecordDirectory validates and parses a raw group's directory once; sparse_directory and sparse_edge answer from the bytes a caller holds and name the next range they lack. - core delta/read.rs: stored_edge is ChainBases::edge over caller-owned caches, so the walk's decoded-group cache is shared and a compressed group is decompressed once, not once per record. - engine history/objects/edges.rs (new): per pack and per wave, bounded acquisitions of directories and headers; group_directories for the waste accounting. - engine history/objects.rs: the walk takes header edges first and keeps the decoding reader for compressed groups and the Ordinary lane; kept waste reads group directories instead of group bodies; inode values use a map; the wave sort computes each key once. - engine backend/sqlite/objects/read.rs: the pack prefix read uses incremental BLOB I/O. `substr(body,1,?)` made SQLite load every pack body whole to return its first bytes. Quadratic steps removed: a record lookup rescanned its group's directory (records squared per group, in kept waste and in the first header reader); inode values were scanned once per ordinal. Trade-off: header reads are not authenticated, so the mark no longer hashes uncompressed payload packs as a side effect. The spec's verify step never required it (no sha256 of either file); spec section 6.2 now says so. Measured (gc_bench, Disposable profile, warm cache, macOS, one invocation per binary, load 5 to 8; receipts not committed) - small, debug, 3 runs per arm, 704 live objects: mark 564.0 -> 50.9 ms (none), 578.4 -> 61.2 ms (rebuild); same 3,133,440 bytes freed. - medium, release, 1 run per arm, 9.789 GB live in 83,559 packs: mark 45.17 -> 4.79 s (none), 24.68 -> 1.53 s (rebuild); rebuild wall 66.58 -> 46.16 s; same 1,095,524,352 bytes freed. Tests - New core tests/sparse_edges.rs (5): header reader equivalence with stored_base including truncations, sparse walk bounds, broken directories, and a full 8,191-record group answered in one pass. - Pass: layerfs-storage (122); persistence kept_objects, kept_node, compact_metrics, sqlite_compact, sqlite_compact_crash, sqlite_compact_matrix, sqlite_compact_units, space_reclamation, acquisition_reads, acquisition_batching, sqlite_group_rows; PostgreSQL gc_mark, compact, compact_fixtures, compact_units, identical_ids, gc_fence against a local PostgreSQL 17. - No test deleted. Production LOC: 187394 -> 187909 (delta +515) Scope and method: tools/production_loc.py at this commit, first parent 6bca86167f against the committed tree; core 40408 -> 40589, engine 89386 -> 89720, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd1917812). Co-Authored-By: Claude Fable 5.2 --- .../src/encoding/delta/edge.rs | 161 ++++++++ .../layerfs-storage/src/encoding/delta/mod.rs | 1 + .../src/encoding/delta/read.rs | 42 +- .../src/encoding/delta/record.rs | 60 +++ .../layerfs-storage/tests/sparse_edges.rs | 334 ++++++++++++++++ .../docs/issues/project-41/LayerFS-GC-spec.md | 10 +- .../src/backend/sqlite/objects/read.rs | 47 ++- .../src/history/objects.rs | 91 +++-- .../src/history/objects/edges.rs | 367 ++++++++++++++++++ 9 files changed, 1053 insertions(+), 60 deletions(-) create mode 100644 core/crates/layerfs-storage/src/encoding/delta/edge.rs create mode 100644 core/crates/layerfs-storage/tests/sparse_edges.rs create mode 100644 engine/crates/layerfs-persistence/src/history/objects/edges.rs diff --git a/core/crates/layerfs-storage/src/encoding/delta/edge.rs b/core/crates/layerfs-storage/src/encoding/delta/edge.rs new file mode 100644 index 0000000000..663d84f9ea --- /dev/null +++ b/core/crates/layerfs-storage/src/encoding/delta/edge.rs @@ -0,0 +1,161 @@ +//! A payload record's dependency edge from bounded reads of its pack. +//! +//! [`super::read::ChainBases::edge`] borrows the record's complete encoded +//! group, so a walk that wants only edges pays for every payload byte it +//! passes. An uncompressed group keeps everything an edge needs at offsets its +//! own framing names: the record count and directory at the group's head, then +//! the record's first bytes. [`sparse_directory`] and [`sparse_edge`] read +//! those from whatever the caller already holds and name the next range it +//! lacks, so the caller decides how to batch the reads. No payload byte is +//! read, and so none is authenticated here: a consumer that reconstructs the +//! record still does that. +//! +//! A group's directory is validated and parsed once into a +//! [`RecordDirectory`]; each record's range is then one step. Asking the +//! framing reader for every record of a group instead scans the directory once +//! per record. +use super::read::ChainEdge; +use super::record; +use crate::error::{StorageError, StorageResult}; +use crate::location::ObjectLocation; +use crate::pack::layout::{record_range, GroupCodec, GroupView, PackLane}; +use crate::policy::RECORD_COUNT_LIMIT; +use crate::port::PackRange; + +/// Bytes at a group's head a walk asks for first: the record count, the +/// directory of a group of up to 100 records, and usually its first header. +pub const HEAD_BYTES: usize = 512; + +/// One uncompressed group's record directory, validated whole exactly as +/// [`crate::encoding::framed_record`] validates it. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct RecordDirectory { + /// Record `n` spans `bounds[n]..bounds[n + 1]` of the group. + bounds: Vec, +} + +impl RecordDirectory { + /// The directory of a complete uncompressed group body. + pub fn of(group: &[u8]) -> StorageResult { + let count = count_of(group.get(..4), group.len())?; + let ends = group + .get(4..4 + 4 * count) + .ok_or(StorageError::Integrity("group record directory"))?; + Self::parse(count, ends, group.len()) + } + + fn parse(count: usize, ends: &[u8], group_length: usize) -> StorageResult { + // The one validator of a directory: counts, ordering and the exact + // record area. What follows only copies offsets it accepted. + record_range(count, ends, group_length, 0)?; + let area = 4 + ends.len(); + let mut bounds = Vec::with_capacity(count + 1); + bounds.push(area); + for end in ends.chunks_exact(4) { + let mut field = [0; 4]; + field.copy_from_slice(end); + bounds.push(area + u32::from_le_bytes(field) as usize); + } + Ok(Self { bounds }) + } + + /// The records the group holds. + pub fn count(&self) -> usize { + self.bounds.len() - 1 + } + + /// The group-relative range of record `number`. + pub fn range(&self, number: usize) -> StorageResult<(usize, usize)> { + match (self.bounds.get(number), self.bounds.get(number + 1)) { + (Some(start), Some(end)) => Ok((*start, *end)), + _ => Err(StorageError::Integrity("group record directory")), + } + } +} + +fn count_of(field: Option<&[u8]>, group_length: usize) -> StorageResult { + let mut count = [0; 4]; + count.copy_from_slice(field.ok_or(StorageError::Integrity("group framing"))?); + let count = u32::from_le_bytes(count) as usize; + if !(1..=RECORD_COUNT_LIMIT).contains(&count) || 4 + 4 * count > group_length { + return Err(StorageError::Integrity("group record directory")); + } + Ok(count) +} + +/// What the bytes a caller holds answer. +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum Sparse { + /// The answer. + Ready(T), + /// The pack range to acquire before asking again. + Need(PackRange), +} + +/// Whether a record of the group `view` of a `lane` pack has an edge these +/// readers can report. A compressed group and the role-dependent `Ordinary` +/// lane take the decoding reader. +pub fn has_sparse_edges(lane: PackLane, view: GroupView) -> bool { + lane != PackLane::Ordinary && view.codec == GroupCodec::Raw +} + +/// The record directory of the uncompressed group `view`, from the pack bytes +/// `held` returns. +/// +/// `held(offset, length)` returns exactly `length` bytes at pack offset +/// `offset`, or `None` when the caller does not hold all of them. +pub fn sparse_directory<'a>( + view: GroupView, + held: impl Fn(usize, usize) -> Option<&'a [u8]>, +) -> StorageResult> { + let length = view + .end + .checked_sub(view.start) + .filter(|length| *length >= 4) + .ok_or(StorageError::Integrity("group framing"))?; + let need = |length: usize| { + Sparse::Need(PackRange { + offset: view.start, + length, + }) + }; + let Some(count) = held(view.start, 4) else { + return Ok(need(length.min(HEAD_BYTES))); + }; + let count = count_of(Some(count), length)?; + let Some(ends) = held(view.start + 4, 4 * count) else { + return Ok(need((4 + 4 * count + record::HEADER_BYTES).min(length))); + }; + RecordDirectory::parse(count, ends, length).map(Sparse::Ready) +} + +/// The edge of the record `location` names in the group `view` of a `lane` +/// pack, exactly as [`super::read::ChainBases::edge`] reports it, from the +/// group's `directory` and the pack bytes `held` returns. +pub fn sparse_edge<'a>( + lane: PackLane, + view: GroupView, + directory: &RecordDirectory, + location: &ObjectLocation, + held: impl Fn(usize, usize) -> Option<&'a [u8]>, +) -> StorageResult> { + if !has_sparse_edges(lane, view) { + return Err(StorageError::Integrity("encoded record edge")); + } + if lane == PackLane::Singleton && location.record_number != 0 { + return Err(StorageError::Integrity("singleton record ordinal")); + } + let (start, end) = directory.range(location.record_number)?; + let width = end - start; + let header = width.min(record::HEADER_BYTES); + let Some(bytes) = held(view.start + start, header) else { + return Ok(Sparse::Need(PackRange { + offset: view.start + start, + length: header, + })); + }; + Ok(Sparse::Ready(ChainEdge { + base: record::header_base(lane, bytes, width, location.canonical_length)?, + width: width as u64, + })) +} diff --git a/core/crates/layerfs-storage/src/encoding/delta/mod.rs b/core/crates/layerfs-storage/src/encoding/delta/mod.rs index 0cf400c9e1..4514a764ef 100644 --- a/core/crates/layerfs-storage/src/encoding/delta/mod.rs +++ b/core/crates/layerfs-storage/src/encoding/delta/mod.rs @@ -4,6 +4,7 @@ pub mod branch; pub mod candidates; +pub mod edge; pub mod read; pub mod record; pub mod select; diff --git a/core/crates/layerfs-storage/src/encoding/delta/read.rs b/core/crates/layerfs-storage/src/encoding/delta/read.rs index 2850201b81..071b31ec3e 100644 --- a/core/crates/layerfs-storage/src/encoding/delta/read.rs +++ b/core/crates/layerfs-storage/src/encoding/delta/read.rs @@ -440,15 +440,45 @@ impl<'a> ChainBases<'a> { workspace: &mut DecompressionWorkspace, location: &ObjectLocation, ) -> StorageResult { - let base = self.base_of(connection, workspace, location)?; - let (selected, _) = group_of(self.packs, connection, location)?; - Ok(ChainEdge { - base, - width: selected_record_width(selected, location)?, - }) + stored_edge( + connection, + self.packs, + &mut self.groups, + workspace, + &mut self.group_decodes, + location, + ) } } +/// [`ChainBases::edge`] over caches the caller owns. +/// +/// A walk that also decodes passes its own decoded-group cache, so a +/// compressed group it already decoded, or whose other records it asks about, +/// is decompressed once and not once per record. +pub fn stored_edge( + connection: &dyn Source, + packs: &mut crate::encoding::PackCache, + groups: &mut GroupCache, + workspace: &mut DecompressionWorkspace, + group_decodes: &mut u64, + location: &ObjectLocation, +) -> StorageResult { + let base = stored_base( + connection, + packs, + groups, + workspace, + group_decodes, + location, + )?; + let (selected, _) = group_of(packs, connection, location)?; + Ok(ChainEdge { + base, + width: selected_record_width(selected, location)?, + }) +} + /// One stored record of a chain, as a cost walk meets it. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub struct ChainEdge { diff --git a/core/crates/layerfs-storage/src/encoding/delta/record.rs b/core/crates/layerfs-storage/src/encoding/delta/record.rs index f9255077e6..b7989bff2e 100644 --- a/core/crates/layerfs-storage/src/encoding/delta/record.rs +++ b/core/crates/layerfs-storage/src/encoding/delta/record.rs @@ -36,6 +36,10 @@ pub const fn is_stored(tag: u8) -> bool { /// Width of one canonical identity. const ID_BYTES: usize = 32; +/// The widest record prefix that carries a payload lane's tag, length fields +/// and base identity: the framed lane's nine bytes and one identity. +pub const HEADER_BYTES: usize = 9 + ID_BYTES; + /// One parsed lane record. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub struct ParsedRecord<'a> { @@ -358,3 +362,59 @@ pub fn stored_base( } Ok(parse(lane, record, canonical_length)?.base) } + +/// The direct base a payload-lane record names, read from its first bytes. +/// +/// `header` is the record's first `min(record_length, HEADER_BYTES)` bytes and +/// `record_length` its complete stored width, which the group directory gives. +/// This accepts and refuses exactly the records [`stored_base`] does on the +/// complete record: the tag, the base identity, the length fields and the +/// non-empty frame are all decided by the header and the width, so only the +/// frame's own bytes go unread. The `Ordinary` lane has no fixed header (its +/// groups are compressed and its grammar depends on the role) and is refused. +pub fn header_base( + lane: PackLane, + header: &[u8], + record_length: usize, + canonical_length: usize, +) -> StorageResult> { + let fixed = match lane { + PackLane::WholeFile => 1, + PackLane::Chunk => 5, + PackLane::Singleton => 9, + PackLane::Ordinary => return Err(StorageError::Integrity("ordinary record header")), + }; + if header.len() != record_length.min(HEADER_BYTES) || header.len() < fixed { + return Err(StorageError::Integrity("record header width")); + } + let field = |at: usize| { + let mut bytes = [0; 4]; + bytes.copy_from_slice(&header[at..at + 4]); + u32::from_le_bytes(bytes) as usize + }; + let (base, start) = match header[0] { + FULL_TAG | STORED_TAG => (None, fixed), + PREFIX_TAG => ( + Some(ObjectId::from_bytes( + header + .get(fixed..fixed + ID_BYTES) + .ok_or(StorageError::Integrity("record base identity"))?, + )?), + fixed + ID_BYTES, + ), + _ => return Err(StorageError::Integrity("record tag")), + }; + let frame = record_length + .checked_sub(start) + .filter(|frame| *frame > 0) + .ok_or(StorageError::Integrity("record frame"))?; + let raw_known = match lane { + // The compact lane drops its length fields; the locator carries them. + PackLane::WholeFile => canonical_length > crate::policy::WHOLE_FILE_CANONICAL_OVERHEAD, + _ => field(1) != 0, + }; + if !raw_known || (lane == PackLane::Singleton && field(5) != frame) { + return Err(StorageError::Integrity("record framing")); + } + Ok(base) +} diff --git a/core/crates/layerfs-storage/tests/sparse_edges.rs b/core/crates/layerfs-storage/tests/sparse_edges.rs new file mode 100644 index 0000000000..270b5d1927 --- /dev/null +++ b/core/crates/layerfs-storage/tests/sparse_edges.rs @@ -0,0 +1,334 @@ +//! A payload record's edge read from bounded ranges agrees with the reader +//! that borrows the complete record. +//! +//! `header_base` must accept and refuse exactly the records `stored_base` +//! does, and `sparse_edge` must report the base and width the complete group +//! gives, whatever pieces of the pack the caller happens to hold. A group's +//! directory is parsed once, so asking for every record of a group costs one +//! pass over it and not one pass per record. +use layerfs_content::{ObjectId, ObjectRole}; +use layerfs_storage::encoding::delta::edge::{ + has_sparse_edges, sparse_directory, sparse_edge, RecordDirectory, Sparse, HEAD_BYTES, +}; +use layerfs_storage::encoding::delta::read::ChainEdge; +use layerfs_storage::encoding::delta::record::{ + header_base, stored_base, FULL_TAG, HEADER_BYTES, PREFIX_TAG, STORED_TAG, +}; +use layerfs_storage::encoding::framed_record; +use layerfs_storage::location::ObjectLocation; +use layerfs_storage::pack::layout::{GroupCodec, GroupView, PackLane}; +use layerfs_storage::policy::WHOLE_FILE_CANONICAL_OVERHEAD; + +const PAYLOAD_LANES: [PackLane; 3] = [PackLane::WholeFile, PackLane::Chunk, PackLane::Singleton]; + +fn base() -> ObjectId { + ObjectId::for_bytes(b"sparse-edges/base") +} + +/// One record of `lane` as the encoder frames it, with every field free so a +/// case can break exactly one of them. +fn record(lane: PackLane, tag: u8, raw: u32, declared_frame: u32, frame: &[u8]) -> Vec { + let mut out = vec![tag]; + if lane != PackLane::WholeFile { + out.extend_from_slice(&raw.to_le_bytes()); + } + if lane == PackLane::Singleton { + out.extend_from_slice(&declared_frame.to_le_bytes()); + } + if tag == PREFIX_TAG { + out.extend_from_slice(&base().to_bytes()); + } + out.extend_from_slice(frame); + out +} + +fn header_of(record: &[u8]) -> &[u8] { + &record[..record.len().min(HEADER_BYTES)] +} + +#[test] +fn the_header_reader_accepts_and_refuses_what_the_record_reader_does() { + let mut accepted = 0; + let mut refused = 0; + for lane in PAYLOAD_LANES { + for tag in [FULL_TAG, PREFIX_TAG, STORED_TAG, 3, 255] { + for raw in [0u32, 1, 70_000] { + for frame_length in [0usize, 1, 7, 64] { + for declared in [frame_length as u32, frame_length as u32 + 1] { + for canonical in [ + 0, + WHOLE_FILE_CANONICAL_OVERHEAD, + WHOLE_FILE_CANONICAL_OVERHEAD + 1, + WHOLE_FILE_CANONICAL_OVERHEAD + 70_000, + ] { + let frame = vec![0xA5; frame_length]; + let whole = record(lane, tag, raw, declared, &frame); + // Every truncation too: a short record must be + // refused by both, never read past by either. + for cut in 0..=whole.len().min(HEADER_BYTES + 2) { + let record = if cut == whole.len().min(HEADER_BYTES + 2) { + &whole[..] + } else { + &whole[..cut] + }; + let complete = + stored_base(lane, record, canonical, ObjectRole::WholeFile); + let header = + header_base(lane, header_of(record), record.len(), canonical); + match (&complete, &header) { + (Ok(left), Ok(right)) => { + assert_eq!(left, right); + accepted += 1; + } + (Err(_), Err(_)) => refused += 1, + _ => panic!( + "{lane:?} tag {tag} raw {raw} frame {frame_length} \ + declared {declared} canonical {canonical} cut {cut}: \ + complete {complete:?}, header {header:?}" + ), + } + } + } + } + } + } + } + } + assert!(accepted > 100 && refused > 100, "{accepted} {refused}"); +} + +#[test] +fn the_header_reader_refuses_a_header_of_the_wrong_width_and_the_ordinary_lane() { + let whole = record(PackLane::Chunk, PREFIX_TAG, 9, 0, &[1; 200]); + assert_eq!( + header_base(PackLane::Chunk, header_of(&whole), whole.len(), 0).unwrap(), + Some(base()) + ); + // One byte short or long of the declared prefix is not the record's header. + assert!(header_base(PackLane::Chunk, &whole[..HEADER_BYTES - 1], whole.len(), 0).is_err()); + assert!(header_base(PackLane::Chunk, &whole[..HEADER_BYTES + 1], whole.len(), 0).is_err()); + assert!(header_base(PackLane::Ordinary, &[FULL_TAG, 1], 2, 0).is_err()); +} + +/// A Raw group body: the record count, each record's end offset, the records. +fn group(records: &[Vec]) -> Vec { + let mut out = (records.len() as u32).to_le_bytes().to_vec(); + let mut end = 0u32; + for record in records { + end += record.len() as u32; + out.extend_from_slice(&end.to_le_bytes()); + } + for record in records { + out.extend_from_slice(record); + } + out +} + +fn location(record_number: usize, canonical_length: usize) -> ObjectLocation { + ObjectLocation { + object_id: ObjectId::for_bytes(&record_number.to_le_bytes()), + role: ObjectRole::Chunk, + canonical_length, + pack_id: 7, + group_number: 0, + record_number, + } +} + +/// Runs the sparse readers the way a walk does: start with nothing held, +/// acquire each range they name, and count the acquisitions. +fn walk( + lane: PackLane, + view: GroupView, + pack: &[u8], + location: &ObjectLocation, +) -> (ChainEdge, usize, usize) { + let mut held: Vec<(usize, Vec)> = Vec::new(); + let mut acquired = 0; + for round in 0..8 { + let have = |offset: usize, length: usize| { + held.iter().find_map(|(at, bytes)| { + let from = offset.checked_sub(*at)?; + bytes.get(from..from + length) + }) + }; + let range = match sparse_directory(view, have).unwrap() { + Sparse::Need(range) => range, + Sparse::Ready(directory) => { + match sparse_edge(lane, view, &directory, location, have).unwrap() { + Sparse::Ready(edge) => return (edge, round, acquired), + Sparse::Need(range) => range, + } + } + }; + assert!(range.offset >= view.start && range.offset + range.length <= view.end); + acquired += range.length; + held.push(( + range.offset, + pack[range.offset..range.offset + range.length].to_vec(), + )); + } + panic!("the walk never settled"); +} + +#[test] +fn a_sparse_walk_reports_the_complete_groups_edge_without_its_payload() { + for lane in [PackLane::WholeFile, PackLane::Chunk] { + for count in [1usize, 2, 5, 130, 400] { + let canonical = WHOLE_FILE_CANONICAL_OVERHEAD + 9_000; + let records: Vec> = (0..count) + .map(|index| { + let tag = [FULL_TAG, PREFIX_TAG, STORED_TAG][index % 3]; + // Widths from one byte of frame to several kilobytes. + let frame = vec![index as u8; 1 + (index * 37) % 9_000]; + record(lane, tag, 9_000, 0, &frame) + }) + .collect(); + let body = group(&records); + let start = 40; + let mut pack = vec![0xEE; start]; + pack.extend_from_slice(&body); + pack.extend_from_slice(&[0xEE; 13]); + let view = GroupView { + start, + end: start + body.len(), + decoded_length: body.len(), + codec: GroupCodec::Raw, + }; + let directory = 4 + 4 * count; + for number in [0, count / 2, count - 1] { + let location = location(number, canonical); + let complete = framed_record(&body, number).unwrap(); + let expected = stored_base(lane, complete, canonical, location.role).unwrap(); + let (edge, rounds, acquired) = walk(lane, view, &pack, &location); + assert_eq!(edge.base, expected, "{lane:?} {count} {number}"); + assert_eq!(edge.width, complete.len() as u64); + assert!(rounds <= 3, "{rounds} rounds"); + // The head, at most a wider directory, at most one header: + // never the payload behind them. + assert!( + acquired <= HEAD_BYTES + directory + 2 * HEADER_BYTES, + "{lane:?} {count} {number}: {acquired} of {}", + body.len() + ); + } + // An ordinal the directory does not hold is refused, as it is by + // the complete reader. + let held = |offset: usize, length: usize| pack.get(offset..offset + length); + let Sparse::Ready(directory) = sparse_directory(view, held).unwrap() else { + panic!("the whole pack is held"); + }; + assert_eq!(directory, RecordDirectory::of(&body).unwrap()); + assert_eq!(directory.count(), count); + // Every record's range is the framing reader's. + for number in 0..count { + let (from, to) = directory.range(number).unwrap(); + assert_eq!(&body[from..to], framed_record(&body, number).unwrap()); + } + let past = location(count, canonical); + assert!(sparse_edge(lane, view, &directory, &past, held).is_err()); + assert!(directory.range(count).is_err()); + assert!(framed_record(&body, count).is_err()); + } + } +} + +#[test] +fn a_broken_directory_is_refused_and_an_encoded_group_is_left_to_the_decoder() { + let records = vec![ + record(PackLane::Chunk, FULL_TAG, 5, 0, &[1; 30]), + record(PackLane::Chunk, FULL_TAG, 5, 0, &[2; 30]), + ]; + let body = group(&records); + let view = |codec| GroupView { + start: 0, + end: body.len(), + decoded_length: body.len(), + codec, + }; + let location = location(1, 0); + let raw = view(GroupCodec::Raw); + let held = |offset: usize, length: usize| body.get(offset..offset + length); + let directory = RecordDirectory::of(&body).unwrap(); + assert!(matches!( + sparse_edge(PackLane::Chunk, raw, &directory, &location, held).unwrap(), + Sparse::Ready(_) + )); + // A compressed group and the Ordinary lane have no edge to read this way: + // the caller asks first, and the reader refuses rather than guess. + assert!(has_sparse_edges(PackLane::Chunk, raw)); + assert!(!has_sparse_edges( + PackLane::Chunk, + view(GroupCodec::Zstandard) + )); + assert!(!has_sparse_edges(PackLane::Ordinary, raw)); + let compressed = view(GroupCodec::Zstandard); + assert!(sparse_edge(PackLane::Chunk, compressed, &directory, &location, held).is_err()); + assert!(sparse_edge(PackLane::Ordinary, raw, &directory, &location, held).is_err()); + // A record count of zero, one past the body, and an end that runs backwards. + for (at, value) in [(0usize, 0u32), (0, 9_999), (4, 0), (8, 3)] { + let mut broken = body.clone(); + broken[at..at + 4].copy_from_slice(&value.to_le_bytes()); + let held = |offset: usize, length: usize| broken.get(offset..offset + length); + assert!( + sparse_directory(raw, held).is_err(), + "field at {at} = {value}" + ); + assert!( + RecordDirectory::of(&broken).is_err(), + "field at {at} = {value}" + ); + assert!( + framed_record(&broken, 1).is_err(), + "field at {at} = {value}" + ); + } + // A group shorter than its count field, and one truncated inside its + // directory, are refused by both readers. + for cut in [0, 3, 7, 11] { + assert!(RecordDirectory::of(&body[..cut]).is_err(), "cut {cut}"); + assert!(framed_record(&body[..cut], 0).is_err(), "cut {cut}"); + } + // A singleton pack holds one record: any other ordinal is refused. + assert!(sparse_edge(PackLane::Singleton, raw, &directory, &location, held).is_err()); +} + +#[test] +fn every_record_of_a_full_group_is_answered_in_one_pass_over_its_directory() { + // The widest group the format admits. One directory parse answers all of + // its records; asking the framing reader for each one scans the directory + // once per record, which is what this reader exists to avoid. + let count = layerfs_storage::policy::RECORD_COUNT_LIMIT; + let records: Vec> = (0..count) + .map(|index| { + let tag = [FULL_TAG, PREFIX_TAG][index % 2]; + record(PackLane::Chunk, tag, 7, 0, &[index as u8; 3]) + }) + .collect(); + let body = group(&records); + let view = GroupView { + start: 0, + end: body.len(), + decoded_length: body.len(), + codec: GroupCodec::Raw, + }; + let held = |offset: usize, length: usize| body.get(offset..offset + length); + let Sparse::Ready(directory) = sparse_directory(view, held).unwrap() else { + panic!("the whole group is held"); + }; + assert_eq!(directory.count(), count); + let mut bases = 0; + let mut width = 0; + for number in 0..count { + let location = location(number, 0); + let Sparse::Ready(edge) = + sparse_edge(PackLane::Chunk, view, &directory, &location, held).unwrap() + else { + panic!("record {number}"); + }; + bases += usize::from(edge.base == Some(base())); + width += edge.width as usize; + } + assert_eq!(bases, count / 2); + assert_eq!(width, body.len() - 4 - 4 * count); +} diff --git a/core/docs/issues/project-41/LayerFS-GC-spec.md b/core/docs/issues/project-41/LayerFS-GC-spec.md index 65abed9252..7be922c696 100644 --- a/core/docs/issues/project-41/LayerFS-GC-spec.md +++ b/core/docs/issues/project-41/LayerFS-GC-spec.md @@ -602,9 +602,13 @@ finds (`engine/crates/layerfs-persistence/src/diagnostic/history.rs:155-198`). (#42 item 36). - Window by window, sort lookups by object ID before locating them. - Whole files and chunks have no logical references; only their delta base - is needed. Reading it still loads and authenticates the whole group - (`delta/read.rs:561-571`); on SQLite the payload part of the walk runs - inside the copy (§7 step 4), and the PostgreSQL mark reads it in Phase M. + is needed. In an uncompressed payload group the mark reads the group's + record directory and each record's first bytes (`delta/edge.rs`, + `history/objects/edges.rs`), never the payload, so those bytes are not + authenticated by the mark; a compressed group and the `Ordinary` lane still + load the whole group (`delta/read.rs`). On SQLite the payload part of the + walk runs inside the copy (§7 step 4), and the PostgreSQL mark reads it in + Phase M. ### 6.3 Visited set diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs index 1b683bba9f..e4b52bd057 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs @@ -1,4 +1,5 @@ //! Read budgets are checked before any BLOB is acquired into the result buffer. +use crate::backend::sqlite::session::rows; use crate::{ backend::metadata_locations::info, backend::{ @@ -8,6 +9,7 @@ use crate::{ }; use layerfs_storage::{policy, port::*}; use std::collections::BTreeSet; +use std::time::Instant; pub(crate) fn read( tx: &Transaction<'_>, ids: &[i64], @@ -99,21 +101,48 @@ pub(crate) fn prefix( } return Ok(Some((length, control))); } - let limit = i64::try_from(limit).map_err(|_| PersistenceError::Malformed)?; + // Incremental BLOB I/O, not `substr(body,1,?)`: SQLite loads a column's + // whole overflow chain before a function sees it, so that statement read + // every pack body to return its first bytes. let mut rows = tx.query( - "SELECT length,substr(body,1,?) FROM pack WHERE pack_id=?", - vec![Param::I64(limit), Param::I64(id)], + "SELECT length FROM pack WHERE pack_id=?", + vec![Param::I64(id)], )?; let Some(row) = rows.pop() else { return Ok(None); }; let length = row.get::(0).map_err(|_| PersistenceError::Malformed)?; - let body = row - .get::>(1) - .map_err(|_| PersistenceError::Malformed)?; let length = usize::try_from(length).map_err(|_| PersistenceError::Malformed)?; - if body.len() != length.min(limit as usize) { - return Err(PersistenceError::Malformed); + tx.work.borrow_mut().blob_open_calls += 1; + let blob = tx + .connection + .blob_open("main", "pack", "body", id, true) + .map_err(rows::error)?; + let mut body = vec![0; length.min(limit)]; + let read = if blob.len() != length { + Err(PersistenceError::Malformed) + } else { + { + let mut work = tx.work.borrow_mut(); + work.blob_read_calls += 1; + work.blob_requested_bytes += body.len() as u64; + } + let start = Instant::now(); + let read = blob.read_at_exact(&mut body, 0).map_err(rows::error); + let mut work = tx.work.borrow_mut(); + work.blob_read_ns += start.elapsed().as_nanos() as u64; + if read.is_ok() { + work.blob_read_bytes += body.len() as u64; + } + read.map_err(Into::into) + }; + tx.work.borrow_mut().blob_close_calls += 1; + let close: Result<(), PersistenceError> = blob.close().map_err(rows::error).map_err(Into::into); + match (read, close) { + (Err(PersistenceError::Uncertain), _) | (_, Err(PersistenceError::Uncertain)) => { + Err(PersistenceError::Uncertain) + } + (Err(error), _) | (_, Err(error)) => Err(error), + (Ok(()), Ok(())) => Ok(Some((length, body))), } - Ok(Some((length, body))) } diff --git a/engine/crates/layerfs-persistence/src/history/objects.rs b/engine/crates/layerfs-persistence/src/history/objects.rs index 2acbaeb3cf..edce4449cb 100644 --- a/engine/crates/layerfs-persistence/src/history/objects.rs +++ b/engine/crates/layerfs-persistence/src/history/objects.rs @@ -26,9 +26,11 @@ use layerfs_history::records::GcIntent; use layerfs_storage::{ encoding::pool::PoolReader, encoding::{ - delta::read::{locator_key, BodyCaches, ChainBases, ChainCounters, Resolver}, - framed_record, framed_record_count, DecompressionWorkspace, GroupCache, PackAcquisition, - PackCache, PackUnit, + delta::{ + edge::RecordDirectory, + read::{locator_key, stored_edge, BodyCaches, ChainCounters, Resolver}, + }, + DecompressionWorkspace, GroupCache, PackAcquisition, PackCache, PackUnit, }, location::{ObjectLocation, PackInfo, SignatureRow}, pack::layout::{self, GroupCodec, PackLane}, @@ -42,6 +44,7 @@ use layerfs_storage::{ use std::collections::{BTreeMap, BTreeSet}; use std::path::Path; +mod edges; mod impls; const ROOTS: &str = "gc_kept_roots"; const LOCATIONS: &str = "gc_object_locations"; @@ -541,12 +544,15 @@ impl Source for ClosureSource<'_> { self.mechanics .inode_values(ordinals, &mut rows) .map_err(StorageError::from)?; + // One ordered map, so a page of many ordinals is not scanned per ordinal. + let values: BTreeMap = + rows.iter().map(|row| (row.ordinal, row.value)).collect(); ordinals .iter() .map(|ordinal| { - rows.iter() - .find(|row| row.ordinal == *ordinal) - .map(|row| row.value) + values + .get(ordinal) + .copied() .ok_or(StorageError::Integrity("mark inode value")) }) .collect() @@ -585,6 +591,7 @@ fn walk( let mut groups = GroupCache::new(); let mut workspace = DecompressionWorkspace::new().map_err(storage)?; let mut counters = ChainCounters::default(); + let mut edge_decodes = 0u64; let mut closure = Closure { visited: BTreeMap::new(), widths: BTreeMap::new(), @@ -596,12 +603,16 @@ fn walk( while !frontier.is_empty() { // Pack order: one body answers every object of its group, so a wave // sorted by locator reads each group once instead of once per identity. - frontier.sort_unstable_by_key(|id| { + // The key is one locator lookup per object, not one per comparison. + frontier.sort_by_cached_key(|id| { source .locations .get(id) .map_or((i64::MAX, usize::MAX, usize::MAX), locator_key) }); + // A file payload's edge is its record header: read just that, a pack + // at a time, and leave the payload unread. + let mut headers = edges::wave_edges(source, &frontier)?; let mut next = Vec::new(); for id in std::mem::take(&mut frontier) { let location = source @@ -645,13 +656,21 @@ fn walk( next.push(reference); } } - // The base identity lives in the record header, so the edge is read - // from the header and never by decoding the chain payload. - let edge = { - let mut chain = ChainBases::new(&mut packs); - chain - .edge(source, &mut workspace, &location) - .map_err(storage)? + // The base identity lives in the record header. An encoded group + // has no header to read on its own, so its edge borrows the group. + let edge = match headers.remove(&id) { + Some(edge) => edge, + // The walk's own decoded-group cache: a compressed group is + // decompressed once, not once more for each record's edge. + None => stored_edge( + source, + &mut packs, + &mut groups, + &mut workspace, + &mut edge_decodes, + &location, + ) + .map_err(storage)?, }; closure.widths.insert(id, edge.width); if let Some(base) = edge.base { @@ -724,8 +743,8 @@ fn waste( let views = layout::directory_group_views(&prefix, header).map_err(storage)?; // A wholly dead group's stored extent is known from the directory. A // retained Raw group whose inventory no longer accounts for every - // record is read on its own, so its exact per-record split never costs - // the whole pack body. + // record has its own record directory read, so its exact per-record + // split costs neither the pack body nor the group's. let mut mixed_raw = Vec::new(); for (number, view) in views.iter().enumerate() { let records = groups.and_then(|groups| groups.get(&number)); @@ -743,22 +762,14 @@ fn waste( if mixed_raw.is_empty() { continue; } - let numbers: Vec = mixed_raw.iter().map(|(number, _)| *number).collect(); - let mut cache = PackCache::new(); - cache - .acquire_groups(source, pack_id, &numbers) - .map_err(storage)?; - for (number, view) in mixed_raw { - let slice = cache - .group(source, pack_id, number) - .map_err(storage)? - .ok_or(HistoryError::Integrity("mark group acquisition"))?; - if slice.view.start != view.start || slice.view.end != view.end { - return Err(HistoryError::Integrity("mark group acquisition")); - } - let records = groups.and_then(|groups| groups.get(&number)); + // Each record's extent is in the group's directory, so the records + // themselves stay unread. + let views: Vec = mixed_raw.iter().map(|(_, view)| *view).collect(); + let directories = edges::group_directories(source, pack_id, &views)?; + for ((number, _), directory) in mixed_raw.iter().zip(&directories) { + let records = groups.and_then(|groups| groups.get(number)); dead_in_retained_packs_bytes = dead_in_retained_packs_bytes - .saturating_add(dead_raw_record_bytes(slice.bytes, records, closure)?); + .saturating_add(dead_raw_record_bytes(directory, records, closure)?); } } let deltas: Vec = closure @@ -816,23 +827,22 @@ fn group_may_hide_dead_records( /// The framed bytes of every record in a Raw group that no reached object owns, /// including a record whose `object_location` row a rebuild did not retain. The -/// caller passes a group body it already read. +/// group's directory names every record's extent, so this is one pass over it. fn dead_raw_record_bytes( - group: &[u8], + directory: &RecordDirectory, records: Option<&BTreeMap>, closure: &Closure, ) -> HistoryResult { - let count = framed_record_count(group).map_err(storage)?; let mut dead = 0u64; - for record in 0..count { + for record in 0..directory.count() { let live = records .and_then(|records| records.get(&record)) .is_some_and(|id| closure.visited.contains_key(id)); if live { continue; } - let framed = framed_record(group, record).map_err(storage)?; - dead = dead.saturating_add(u64::try_from(framed.len()).unwrap_or(u64::MAX)); + let (start, end) = directory.range(record).map_err(storage)?; + dead = dead.saturating_add(u64::try_from(end - start).unwrap_or(u64::MAX)); } Ok(dead) } @@ -861,11 +871,8 @@ fn pack_waste( // stored uncompressed. A compressed group's bytes cannot be // attributed to one record, so a retained group contributes // nothing and the error stays conservative. - dead = dead.saturating_add(dead_raw_record_bytes( - &body[view.start..view.end], - records, - closure, - )?); + let directory = RecordDirectory::of(&body[view.start..view.end]).map_err(storage)?; + dead = dead.saturating_add(dead_raw_record_bytes(&directory, records, closure)?); } } Ok(dead) diff --git a/engine/crates/layerfs-persistence/src/history/objects/edges.rs b/engine/crates/layerfs-persistence/src/history/objects/edges.rs new file mode 100644 index 0000000000..8d64dbcaab --- /dev/null +++ b/engine/crates/layerfs-persistence/src/history/objects/edges.rs @@ -0,0 +1,367 @@ +//! The delta edges of one mark wave's payload records, from bounded pack reads +//! (§6.2). +//! +//! A file payload has no logical references, so the mark needs only its +//! record's base and stored width. Both sit in the record's first bytes and in +//! its group's directory, so this reads those ranges and never the payload: a +//! mark costs a few hundred bytes per pack instead of the pack. Compressed +//! groups and the `Ordinary` lane answer `None` and keep the decoding reader. +use super::{storage, ClosureSource}; +use layerfs_content::ObjectId; +use layerfs_history::error::{HistoryError, HistoryResult}; +use layerfs_storage::{ + encoding::delta::edge::{ + has_sparse_edges, sparse_directory, sparse_edge, RecordDirectory, Sparse, + }, + encoding::delta::read::ChainEdge, + location::{ObjectLocation, PackInfo}, + pack::layout::{self, GroupView, PackHeader}, + port::{AcquiredPackRead, PackRange, PackReadChoice, PackReadPlan, PersistenceError}, + source::Source, + StorageError, +}; +use std::collections::BTreeMap; + +/// Count, then a wider directory, then record headers: no pack needs more. +const ROUNDS: usize = 4; +/// Ranges one acquisition selects; far below the port's own limits. +const BATCH_RANGES: usize = 1024; +/// Bytes one acquisition selects. +const BATCH_BYTES: usize = 512 * 1024; + +/// The pack bytes held so far, by pack offset. +/// +/// Pieces never overlap or touch: `add` joins a new piece to its neighbours. +/// So the one piece that can hold a range is the last one starting at or +/// before it, and a lookup is one ordered-map step, whatever the pack holds. +#[derive(Default)] +struct Pieces(BTreeMap>); + +impl Pieces { + fn held(&self, offset: usize, length: usize) -> Option<&[u8]> { + let (at, bytes) = self.0.range(..=offset).next_back()?; + let from = offset - at; + bytes.get(from..from.checked_add(length)?) + } + + /// Adds the bytes at `offset`. A pack is immutable, so where a new piece + /// overlaps a held one both carry the same bytes and either copy serves. + fn add(&mut self, offset: usize, bytes: Vec) { + let (mut start, mut joined) = (offset, bytes); + let before = self + .0 + .range(..=start) + .next_back() + .map(|(at, held)| (*at, held.len())) + .filter(|(at, length)| at + length >= start); + if let Some((at, length)) = before { + let mut held = self.0.remove(&at).unwrap_or_default(); + held.extend_from_slice(joined.get(at + length - start..).unwrap_or(&[])); + (start, joined) = (at, held); + } + while let Some(at) = self + .0 + .range(start..=start + joined.len()) + .next() + .map(|(at, _)| *at) + { + let held = self.0.remove(&at).unwrap_or_default(); + joined.extend_from_slice(held.get(start + joined.len() - at..).unwrap_or(&[])); + } + self.0.insert(start, joined); + } +} + +/// One record's state after a round. +#[derive(Clone, Copy)] +enum Answer { + Edge(ChainEdge), + Encoded, + Pending, +} + +/// Sorts `needs` and joins the ranges that overlap or touch. +fn merged(mut needs: Vec) -> Vec { + needs.sort_unstable_by_key(|range| (range.offset, range.length)); + let mut merged: Vec = Vec::with_capacity(needs.len()); + for range in needs { + match merged.last_mut() { + Some(last) if range.offset <= last.offset + last.length => { + last.length = last.length.max(range.offset + range.length - last.offset); + } + _ => merged.push(range), + } + } + merged +} + +/// Answers every item the held bytes can, and lists the ranges still needed, +/// sorted and disjoint. +/// +/// `items` are in locator order, so a group's records are adjacent and its +/// directory is parsed once for all of them, not once per record. +fn resolve( + header: PackHeader, + views: &[GroupView], + items: &[ObjectLocation], + held: &Pieces, + answers: &mut [Answer], +) -> Result, StorageError> { + let mut needs: Vec = Vec::new(); + let mut group: Option<(usize, Option)> = None; + for (item, answer) in items.iter().zip(answers.iter_mut()) { + if !matches!(answer, Answer::Pending) { + continue; + } + let view = *views + .get(item.group_number) + .ok_or(StorageError::Integrity("group ordinal"))?; + if !has_sparse_edges(header.lane, view) { + *answer = Answer::Encoded; + continue; + } + if group.as_ref().map(|(number, _)| *number) != Some(item.group_number) { + let directory = + match sparse_directory(view, |offset, length| held.held(offset, length))? { + Sparse::Ready(directory) => Some(directory), + Sparse::Need(range) => { + needs.push(range); + None + } + }; + group = Some((item.group_number, directory)); + } + let Some((_, Some(directory))) = &group else { + continue; + }; + match sparse_edge(header.lane, view, directory, item, |offset, length| { + held.held(offset, length) + })? { + Sparse::Ready(edge) => *answer = Answer::Edge(edge), + Sparse::Need(range) => needs.push(range), + } + } + Ok(merged(needs)) +} + +/// Selects the ranges a round needs. The first round has no directory yet, so +/// it derives its ranges from the prefix the acquisition hands it. +struct EdgeRanges<'a> { + pack_id: i64, + items: &'a [ObjectLocation], + held: &'a Pieces, + answers: &'a mut [Answer], + layout: Option<(PackHeader, Vec)>, + ranges: Option>, +} + +impl PackReadPlan for EdgeRanges<'_> { + fn select( + &mut self, + info: PackInfo, + prefix: &[u8], + ) -> Result { + if info.pack_id != self.pack_id { + return Err(PersistenceError::Malformed); + } + if let Some(ranges) = self.ranges.take() { + return Ok(PackReadChoice::Ranges(ranges)); + } + let header = layout::parse_directory_header(prefix, info.length) + .map_err(|_| PersistenceError::Malformed)?; + let views = layout::directory_group_views(prefix, header) + .map_err(|_| PersistenceError::Malformed)?; + let mut ranges = resolve(header, &views, self.items, self.held, self.answers) + .map_err(|_| PersistenceError::Malformed)?; + if ranges.is_empty() { + // Every group of this pack is encoded. An acquisition must select + // something, and the first body byte always exists. + ranges.push(PackRange { + offset: header.body_offset, + length: 1, + }); + } + ranges.truncate(BATCH_RANGES); + self.layout = Some((header, views)); + Ok(PackReadChoice::Ranges(ranges)) + } +} + +/// One acquisition of `plan`'s ranges, added to `held` and to the mark's bytes. +fn acquire( + source: &ClosureSource<'_>, + pack_id: i64, + plan: &mut EdgeRanges<'_>, +) -> HistoryResult)>> { + let read = source + .mechanics + .read_scoped_pack(pack_id, plan) + .map_err(StorageError::from) + .map_err(storage)?; + let AcquiredPackRead::Units(units) = read else { + return Err(HistoryError::Integrity("mark edge acquisition")); + }; + source.note_scoped_body( + pack_id, + u64::try_from(units.acquired_bytes()).unwrap_or(u64::MAX), + u64::try_from(units.info().length).unwrap_or(u64::MAX), + ); + Ok(units.into_parts().2) +} + +/// Acquires `needs`, sorted and disjoint ranges of pack `pack_id`, in bounded +/// batches, and adds them to `held`. +fn acquire_all( + source: &ClosureSource<'_>, + pack_id: i64, + needs: Vec, + held: &mut Pieces, +) -> HistoryResult<()> { + let mut acquired = Vec::new(); + let mut flush = |batch: &mut Vec| -> HistoryResult<()> { + if batch.is_empty() { + return Ok(()); + } + let mut plan = EdgeRanges { + pack_id, + items: &[], + held: &Pieces::default(), + answers: &mut [], + layout: None, + ranges: Some(std::mem::take(batch)), + }; + acquired.extend(acquire(source, pack_id, &mut plan)?); + Ok(()) + }; + let mut batch = Vec::new(); + let mut bytes = 0; + for range in needs { + if batch.len() == BATCH_RANGES || bytes + range.length > BATCH_BYTES { + flush(&mut batch)?; + bytes = 0; + } + bytes += range.length; + batch.push(range); + } + flush(&mut batch)?; + for (range, bytes) in acquired { + held.add(range.offset, bytes); + } + Ok(()) +} + +/// The record directories of `groups`, uncompressed groups of pack `pack_id` +/// in pack order, from bounded reads: each record's stored extent without the +/// records themselves. +pub(super) fn group_directories( + source: &ClosureSource<'_>, + pack_id: i64, + groups: &[GroupView], +) -> HistoryResult> { + let mut held = Pieces::default(); + // The head, then a directory wider than the head: no group needs more. + for _ in 0..3 { + let mut needs = Vec::new(); + let mut directories = Vec::with_capacity(groups.len()); + for view in groups { + match sparse_directory(*view, |offset, length| held.held(offset, length)) + .map_err(storage)? + { + Sparse::Ready(directory) => directories.push(directory), + Sparse::Need(range) => needs.push(range), + } + } + if needs.is_empty() { + return Ok(directories); + } + acquire_all(source, pack_id, merged(needs), &mut held)?; + } + Err(HistoryError::Integrity("mark group directory")) +} + +/// The edges of `items`, all records of pack `pack_id`: `None` where the +/// record's group is encoded and must take the decoding reader. +fn pack_edges( + source: &ClosureSource<'_>, + pack_id: i64, + items: &[ObjectLocation], +) -> HistoryResult>> { + let mut held = Pieces::default(); + let mut answers = vec![Answer::Pending; items.len()]; + let pieces = { + let mut plan = EdgeRanges { + pack_id, + items, + held: &held, + answers: &mut answers, + layout: None, + ranges: None, + }; + let pieces = acquire(source, pack_id, &mut plan)?; + (plan.layout, pieces) + }; + let (Some((header, views)), first) = pieces else { + return Err(HistoryError::Integrity("mark edge directory")); + }; + for (range, bytes) in first { + held.add(range.offset, bytes); + } + for _ in 0..ROUNDS { + let needs = resolve(header, &views, items, &held, &mut answers).map_err(storage)?; + if needs.is_empty() { + return answers + .into_iter() + .map(|answer| match answer { + Answer::Edge(edge) => Ok(Some(edge)), + Answer::Encoded => Ok(None), + Answer::Pending => Err(HistoryError::Integrity("mark edge read")), + }) + .collect(); + } + acquire_all(source, pack_id, needs, &mut held)?; + } + Err(HistoryError::Integrity("mark edge read")) +} + +/// The header-read edges of one wave: every object of `wave` whose role names +/// no other object and whose record sits in an uncompressed payload group. +/// +/// `wave` is in locator order, so each pack's records are adjacent and one +/// pack is read at a time. An object absent from the result takes the decoding +/// reader, exactly as before. +pub(super) fn wave_edges( + source: &ClosureSource<'_>, + wave: &[ObjectId], +) -> HistoryResult> { + let mut edges = BTreeMap::new(); + let mut items: Vec = Vec::new(); + let mut flush = |items: &mut Vec| -> HistoryResult<()> { + let Some(first) = items.first() else { + return Ok(()); + }; + for (item, edge) in items.iter().zip(pack_edges(source, first.pack_id, items)?) { + if let Some(edge) = edge { + edges.insert(item.object_id, edge); + } + } + items.clear(); + Ok(()) + }; + for id in wave { + let Some(location) = source.location(*id, i64::MAX).map_err(storage)? else { + continue; + }; + if location.role.has_references() { + continue; + } + if items + .first() + .is_some_and(|first| first.pack_id != location.pack_id) + { + flush(&mut items)?; + } + items.push(location); + } + flush(&mut items)?; + Ok(edges) +} From f33b44a5e2eea92ca8cc810f67feb1a0e80bb2c8 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:00:04 +0800 Subject: [PATCH 02/31] feat(gc): per-statement SQL profile and gc_profile example (layerfs-project#41) Finding where a mark or a compaction spends its time needed more than the aggregate counters: which statement, how often, how long. - session/thread_work.rs: profile_statements() starts a per-thread profile keyed by statement text (whitespace collapsed, first 110 characters); take_statement_profile() ends it and returns it costliest first. Off, a statement pays one thread-local read. - session/prepared.rs: a statement lease keeps its text only while its thread profiles, and notes each preparation and execution. - examples/gc_profile.rs: on a Store gc_bench kept, runs N marks and optionally one compaction, printing wall time, this thread's SQL work counters and the costliest statements. - tools/ci/gc-bench.md: how to use it and what it has shown. No test added: the example is a diagnostic with no receipt. No test deleted. Production LOC: 187909 -> 187965 (delta +56) Scope and method: tools/production_loc.py at this commit, first parent d19d98f138 against the committed tree; core 40589 -> 40589, engine 89720 -> 89776, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd1917812). Co-Authored-By: Claude Fable 5.2 --- .../examples/gc_profile.rs | 125 ++++++++++++++++++ .../src/backend/sqlite/session/prepared.rs | 11 ++ .../src/backend/sqlite/session/thread_work.rs | 66 ++++++++- engine/crates/layerfs-persistence/src/lib.rs | 4 +- tools/ci/gc-bench.md | 28 ++++ 5 files changed, 232 insertions(+), 2 deletions(-) create mode 100644 engine/crates/layerfs-persistence/examples/gc_profile.rs diff --git a/engine/crates/layerfs-persistence/examples/gc_profile.rs b/engine/crates/layerfs-persistence/examples/gc_profile.rs new file mode 100644 index 0000000000..fe95321e92 --- /dev/null +++ b/engine/crates/layerfs-persistence/examples/gc_profile.rs @@ -0,0 +1,125 @@ +//! Where one GC mark, or one whole compaction, spends its time on a kept +//! `gc_bench` Store: wall time, this thread's SQL work counters, and the +//! statements that cost the most. +//! +//! ```text +//! cargo run -p layerfs-persistence --release --example gc_bench -- --keep --tmpdir DIR +//! cargo run -p layerfs-persistence --release --example gc_profile -- DIR//store.sqlite +//! ``` +//! +//! `--marks N` runs N marks (default 2: the second shows a warm cache). +//! `--compact` then runs one compaction; it changes the Store. `--top N` +//! prints the N costliest statements (default 12). The rebuild's copy runs on +//! its own connection, so its statements are not in the profile; its time is +//! the compaction's wall time less the profiled statements. +#![allow(dead_code)] +#[path = "gc_bench/gc_fixture.rs"] +mod gc_fixture; + +use layerfs_persistence::{ + profile_statements, take_statement_profile, thread_work, CompactPath, CompactRequest, +}; +use std::time::Instant; + +fn counters(label: &str, wall_ms: f64, work: &layerfs_persistence::SqlWork) { + println!( + "{label}: wall_ms={wall_ms:.1} statements={} statement_ms={:.1} vm_steps={} \ + fullscan_steps={} sorts={} autoindex_rows={} returned_rows={} blob_opens={} \ + blob_reads={} blob_read_bytes={} blob_read_ms={:.1} transactions={} commit_ms={:.1}", + work.statements, + work.statement_ns as f64 / 1e6, + work.vm_steps, + work.fullscan_steps, + work.sorts, + work.autoindex_rows, + work.returned_rows, + work.blob_open_calls, + work.blob_read_calls, + work.blob_read_bytes, + work.blob_read_ns as f64 / 1e6, + work.transactions, + work.commit_ns as f64 / 1e6 + ); +} + +fn statements(top: usize) { + let profile = take_statement_profile(); + let total: u64 = profile.iter().map(|(_, cost)| cost.wall_ns).sum(); + println!( + " {} distinct statements, {:.1} ms; costliest first (ms, calls, rows, text):", + profile.len(), + total as f64 / 1e6 + ); + for (sql, cost) in profile.iter().take(top) { + println!( + " {:>9.2} {:>8} {:>9} {sql}", + cost.wall_ns as f64 / 1e6, + cost.calls, + cost.rows + ); + } +} + +fn main() { + let mut store = None; + let (mut marks, mut top, mut compact) = (2usize, 12usize, false); + let mut args = std::env::args().skip(1); + while let Some(arg) = args.next() { + let mut number = |name: &str| -> usize { + args.next() + .and_then(|value| value.parse().ok()) + .unwrap_or_else(|| panic!("{name} needs a number")) + }; + match arg.as_str() { + "--marks" => marks = number("--marks"), + "--top" => top = number("--top"), + "--compact" => compact = true, + _ => store = Some(std::path::PathBuf::from(arg)), + } + } + let store = store.expect("usage: gc_profile STORE [--marks N] [--top N] [--compact]"); + let handles = gc_fixture::reopen(&store); + for index in 1..=marks { + let before = thread_work(); + profile_statements(); + let started = Instant::now(); + let mark = handles.history.mark().expect("mark"); + let wall = started.elapsed().as_secs_f64() * 1e3; + counters( + &format!("mark {index}"), + wall, + &thread_work().since(&before), + ); + println!( + " objects={} packs={} history_nodes={} mark_bytes={}", + mark.objects().len(), + mark.objects().packs().len(), + mark.history().len(), + mark.mark_bytes() + ); + statements(top); + } + if compact { + let before = thread_work(); + profile_statements(); + let started = Instant::now(); + let receipt = handles + .compact(&CompactRequest { + op_id: [0xC7; 16], + ..CompactRequest::default() + }) + .expect("compact"); + let wall = started.elapsed().as_secs_f64() * 1e3; + counters("compact", wall, &thread_work().since(&before)); + println!( + " path={} physical_shrink={} peak_temp_bytes={}", + match receipt.path { + CompactPath::None => "none", + CompactPath::Rebuild => "rebuild", + }, + receipt.physical_shrink, + receipt.peak_temp_bytes + ); + statements(top); + } +} diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/session/prepared.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/session/prepared.rs index cea1bdb3ad..6a64dd6e41 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/session/prepared.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/session/prepared.rs @@ -15,6 +15,8 @@ pub(crate) struct Prepared<'a> { work: &'a RefCell, commit: bool, uncertain: Option<&'a Cell>, + /// The statement text, kept only while this thread profiles statements. + profiled: Option>, } impl<'a> Prepared<'a> { pub(crate) fn new( @@ -30,11 +32,16 @@ impl<'a> Prepared<'a> { connection.prepare_cached(sql).map_err(rows::error) }; Self::elapsed(work, commit, start); + let profiled = super::thread_work::profiling_statements().then(|| { + super::thread_work::note_statement(sql, 0, start.elapsed().as_nanos() as u64, 0); + Box::from(sql) + }); Ok(Self { statement: Some(statement?), work, commit, uncertain, + profiled, }) } fn elapsed(work: &RefCell, commit: bool, start: Instant) { @@ -158,6 +165,10 @@ impl<'a> Prepared<'a> { work.bound_bytes += bytes; } Self::elapsed(work, commit, start); + if let Some(sql) = &self.profiled { + let wall = start.elapsed().as_nanos() as u64; + super::thread_work::note_statement(sql, 1, wall, returned); + } if result.as_ref().err().is_some_and(OutcomeError::uncertain) { if let Some(uncertain) = self.uncertain { uncertain.set(true); diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/session/thread_work.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/session/thread_work.rs index c7f57220eb..1eb11d791e 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/session/thread_work.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/session/thread_work.rs @@ -4,10 +4,74 @@ //! connection opened, also adds its work to its own thread's totals: a call's //! own work is its thread's totals after it minus before it. use super::connection::{SqlWork, State}; -use std::cell::Cell; +use std::cell::{Cell, RefCell}; +use std::collections::BTreeMap; thread_local! { static THREAD: Cell = Cell::new(SqlWork::default()); + /// This thread's per-statement profile; `None` unless one was started. + static STATEMENTS: RefCell>> = + const { RefCell::new(None) }; +} + +/// What one statement text cost this thread while a profile was running. +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +pub struct StatementCost { + /// Executions. + pub calls: u64, + /// Wall time preparing and executing it, cursor and decode included. + pub wall_ns: u64, + /// Rows it returned. + pub rows: u64, +} + +/// The statement text a profile groups by: the first characters, with +/// whitespace collapsed, so a list of bound parameters does not split a group. +const STATEMENT_KEY_CHARS: usize = 110; + +/// Starts this thread's per-statement profile, discarding any earlier one. +/// +/// A diagnostic for finding which statements an operation spends its time +/// in. Off, it costs one thread-local read per statement. +pub fn profile_statements() { + STATEMENTS.with(|profile| *profile.borrow_mut() = Some(BTreeMap::new())); +} + +/// Ends this thread's per-statement profile and returns it, costliest first. +pub fn take_statement_profile() -> Vec<(String, StatementCost)> { + let mut rows: Vec<_> = STATEMENTS + .with(|profile| profile.borrow_mut().take()) + .unwrap_or_default() + .into_iter() + .collect(); + rows.sort_by(|left, right| right.1.wall_ns.cmp(&left.1.wall_ns)); + rows +} + +/// Adds one preparation or execution of `sql` to a running profile. +pub(crate) fn note_statement(sql: &str, calls: u64, wall_ns: u64, rows: u64) { + STATEMENTS.with(|profile| { + let mut profile = profile.borrow_mut(); + let Some(profile) = profile.as_mut() else { + return; + }; + let key: String = sql + .split_whitespace() + .collect::>() + .join(" ") + .chars() + .take(STATEMENT_KEY_CHARS) + .collect(); + let cost = profile.entry(key).or_default(); + cost.calls += calls; + cost.wall_ns += wall_ns; + cost.rows += rows; + }); +} + +/// Whether this thread is profiling statements now. +pub(crate) fn profiling_statements() -> bool { + STATEMENTS.with(|profile| profile.borrow().is_some()) } /// The SQL work this thread has run on every session of this process. diff --git a/engine/crates/layerfs-persistence/src/lib.rs b/engine/crates/layerfs-persistence/src/lib.rs index 579d32134b..5cfa4930a1 100644 --- a/engine/crates/layerfs-persistence/src/lib.rs +++ b/engine/crates/layerfs-persistence/src/lib.rs @@ -30,7 +30,9 @@ pub use backend::sqlite::lifecycle::headroom::STORE_HEADROOM_BYTES; pub use backend::sqlite::lifecycle::reclamation::{SpaceReclamation, RECLAMATION_PAGE_LIMIT}; pub use backend::sqlite::session::connection::{ConnectionProfile, SqlWork}; pub use backend::sqlite::session::statement_work::StatementPhaseWork; -pub use backend::sqlite::session::thread_work::thread_work; +pub use backend::sqlite::session::thread_work::{ + profile_statements, take_statement_profile, thread_work, StatementCost, +}; pub use backend::sqlite::storage::merge_records::{ MergeRecords, MergeRecordsWork, MergeScratch, BUFFER_BYTES, MERGE_LEASE, RESIDENT_BYTES, }; diff --git a/tools/ci/gc-bench.md b/tools/ci/gc-bench.md index 7848a9e6e7..10e4d2b1fc 100644 --- a/tools/ci/gc-bench.md +++ b/tools/ci/gc-bench.md @@ -333,3 +333,31 @@ This is a measurement harness for the GC section 13 arms, not a new registered model asks for, including the E3 median/range for the medium arm. The 50 GB arm is a single run per arm, as #42 item 44 specifies, and its result stays owner-only until the owner records it. + +## Profiling a mark or a compaction + +`gc_profile` answers "where does the time go" on a Store `gc_bench` kept. It is +a diagnostic, not a measurement harness: it has no receipt and claims nothing. + +```text +cargo run -p layerfs-persistence --release --example gc_bench -- --keep --tmpdir DIR +cargo run -p layerfs-persistence --release --example gc_profile -- DIR//store.sqlite +``` + +- Each mark prints its wall time, this thread's SQL work counters (statements, + engine steps, full-scan steps, sorts, rows, BLOB opens, reads and bytes) and + the statements that cost the most, by text. `--marks N` repeats it; the + second mark shows the warm-cache cost, and the difference from the first is + what a cold cache adds. +- `--compact` then runs one compaction and prints the same for it. The + rebuild's copy runs on its own connection, so its statements are not listed. +- The per-statement list comes from `layerfs_persistence::profile_statements` + and `take_statement_profile`: a per-thread profile that is off unless + started. +- For CPU stacks on macOS, run with `--marks 40` and `sample 2`. + +What it has shown so far (2026-11-17): a function over a BLOB column, such as +`substr(body,1,?)`, loads the whole overflow chain, so it reads the pack to +return a prefix; `length(body)` does not. In the default `Monolithic` layout a +read at offset X of a pack follows X / 4 KiB overflow pages, so a read deep in +a large pack costs page loads for everything before it. From 962acd4f09baf0475ec810075b175f9d10d74077 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:00:34 +0800 Subject: [PATCH 03/31] perf(gc): mark reads pack heads in one call, not per-pack statements (layerfs-project#41) The profile of a mark over 8,816 packs showed two statements and two BLOB handles per pack: one acquisition for the record headers and one prefix read for the kept-waste pass. - backend/seam.rs: Mechanics::pack_prefixes reads the bounded prefixes of several packs in one call (default: one pack_prefix each). - backend/sqlite: one BLOB handle moved from row to row, so a pack costs no statement and no handle of its own. The single prefix read shares the same head reader. - history/objects/edges.rs: a wave reads the first 1,024 bytes of each of its packs, 256 packs per call. A pack whose directory and record headers sit in that head is answered from it; a wider directory is read again, and only a pack that needs more ranges takes a scoped acquisition. - history/objects.rs: the walk keeps each pack's group extents and the kept-waste pass uses them instead of reading the directory again; the scope hash used for metadata objects is computed once per walk. PostgreSQL keeps the default pack_prefixes (one statement per pack); a batched statement there is a follow-up. Measured (gc_profile, release, warm cache, macOS, load about 4, one run; a 1.13 GB gc_bench Store of 8,913 objects in 8,816 packs): mark wall 88.8 -> 36.3 ms; statements 17,740 -> 298; BLOB handles 17,633 -> 229; BLOB reads 35,267 -> 9,104; bytes read 40.4 -> 9.0 MB. The same binary on the 28 MB small Store (737 objects in 158 packs of up to six groups): mark 6.8 ms, 172 statements, 150 handles; packs of several groups still take one acquisition each. Tests: kept_objects, kept_node, compact_metrics, sqlite_compact, sqlite_compact_crash, sqlite_compact_matrix, space_reclamation, postgresql_gc_mark, postgresql_compact, postgresql_compact_fixtures pass. No test added or deleted. Production LOC: 187965 -> 188037 (delta +72) Scope and method: tools/production_loc.py at this commit, first parent against the committed tree; core 40589 -> 40589, engine 89776 -> 89848, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd1917812). Co-Authored-By: Claude Fable 5.2 --- .../layerfs-persistence/src/backend/seam.rs | 17 ++ .../src/backend/sqlite/mechanics.rs | 8 + .../src/backend/sqlite/objects/read.rs | 85 +++++++-- .../src/history/objects.rs | 81 ++++---- .../src/history/objects/edges.rs | 178 +++++++++--------- 5 files changed, 233 insertions(+), 136 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/seam.rs b/engine/crates/layerfs-persistence/src/backend/seam.rs index 93d568526a..272fdc567e 100644 --- a/engine/crates/layerfs-persistence/src/backend/seam.rs +++ b/engine/crates/layerfs-persistence/src/backend/seam.rs @@ -88,6 +88,23 @@ pub(crate) trait Mechanics { ) -> Result)>, PersistenceError> { Ok(None) } + /// [`Self::pack_prefix`] for several packs in one call, in `ids` order. + /// An engine answers from one cursor or one statement, not one per pack; + /// a pack that is absent is an error. + fn pack_prefixes( + &self, + ids: &[i64], + limit: usize, + out: &mut Vec<(usize, Vec)>, + ) -> Result<(), PersistenceError> { + for id in ids { + out.push( + self.pack_prefix(*id, limit)? + .ok_or(PersistenceError::BackendUnavailable)?, + ); + } + Ok(()) + } /// One pack through the caller's bounded range plan: only the control /// prefix and the selected complete groups are read, never the body behind /// them. The storage port owns the plan and the returned units. diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs index 9e8b58a97d..64ddc3d5c6 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs @@ -61,6 +61,14 @@ impl Mechanics for Transaction<'_> { ) -> Result)>, PersistenceError> { super::objects::read::prefix(self, id, limit) } + fn pack_prefixes( + &self, + ids: &[i64], + limit: usize, + out: &mut Vec<(usize, Vec)>, + ) -> Result<(), PersistenceError> { + super::objects::read::prefixes(self, ids, limit, out) + } fn read_scoped_pack( &self, id: i64, diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs index e4b52bd057..053fb6937b 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs @@ -118,24 +118,41 @@ pub(crate) fn prefix( .connection .blob_open("main", "pack", "body", id, true) .map_err(rows::error)?; - let mut body = vec![0; length.min(limit)]; let read = if blob.len() != length { Err(PersistenceError::Malformed) } else { - { - let mut work = tx.work.borrow_mut(); - work.blob_read_calls += 1; - work.blob_requested_bytes += body.len() as u64; - } - let start = Instant::now(); - let read = blob.read_at_exact(&mut body, 0).map_err(rows::error); - let mut work = tx.work.borrow_mut(); - work.blob_read_ns += start.elapsed().as_nanos() as u64; - if read.is_ok() { - work.blob_read_bytes += body.len() as u64; - } - read.map_err(Into::into) + head(tx, &blob, limit) }; + closed(tx, blob, read).map(|body| Some((length, body))) +} + +/// The first `limit` bytes of the body `blob` is open on. +fn head( + tx: &Transaction<'_>, + blob: &rusqlite::blob::Blob<'_>, + limit: usize, +) -> Result, PersistenceError> { + let mut body = vec![0; blob.len().min(limit)]; + { + let mut work = tx.work.borrow_mut(); + work.blob_read_calls += 1; + work.blob_requested_bytes += body.len() as u64; + } + let start = Instant::now(); + let read = blob.read_at_exact(&mut body, 0).map_err(rows::error); + let mut work = tx.work.borrow_mut(); + work.blob_read_ns += start.elapsed().as_nanos() as u64; + read?; + work.blob_read_bytes += body.len() as u64; + Ok(body) +} + +/// Closes `blob` and keeps an unproven outcome ahead of any other failure. +fn closed( + tx: &Transaction<'_>, + blob: rusqlite::blob::Blob<'_>, + read: Result, +) -> Result { tx.work.borrow_mut().blob_close_calls += 1; let close: Result<(), PersistenceError> = blob.close().map_err(rows::error).map_err(Into::into); match (read, close) { @@ -143,6 +160,44 @@ pub(crate) fn prefix( Err(PersistenceError::Uncertain) } (Err(error), _) | (_, Err(error)) => Err(error), - (Ok(()), Ok(())) => Ok(Some((length, body))), + (Ok(value), Ok(())) => Ok(value), } } + +/// The bounded prefixes of several packs, in `ids` order, through one BLOB +/// handle moved from row to row: no statement and no handle per pack. +pub(crate) fn prefixes( + tx: &Transaction<'_>, + ids: &[i64], + limit: usize, + out: &mut Vec<(usize, Vec)>, +) -> Result<(), PersistenceError> { + if limit == 0 || ids.iter().any(|id| *id <= 0) { + return Err(PersistenceError::Malformed); + } + let Some(first) = ids.first() else { + return Ok(()); + }; + if tx.layout().uses_units() { + for id in ids { + out.push(prefix(tx, *id, limit)?.ok_or(PersistenceError::Missing)?); + } + return Ok(()); + } + tx.work.borrow_mut().blob_open_calls += 1; + let mut blob = tx + .connection + .blob_open("main", "pack", "body", *first, true) + .map_err(rows::error)?; + let read = (|| { + for (index, id) in ids.iter().enumerate() { + if index > 0 { + tx.work.borrow_mut().blob_reopen_calls += 1; + blob.reopen(*id).map_err(rows::error)?; + } + out.push((blob.len(), head(tx, &blob, limit)?)); + } + Ok(()) + })(); + closed(tx, blob, read) +} diff --git a/engine/crates/layerfs-persistence/src/history/objects.rs b/engine/crates/layerfs-persistence/src/history/objects.rs index edce4449cb..97fde644e1 100644 --- a/engine/crates/layerfs-persistence/src/history/objects.rs +++ b/engine/crates/layerfs-persistence/src/history/objects.rs @@ -260,6 +260,7 @@ fn mark_recorded( mechanics: tx.mechanics(), locations: &located.locations, read_bytes: std::cell::RefCell::new(BTreeMap::new()), + layouts: std::cell::RefCell::new(BTreeMap::new()), }; let roots = roots(tx)?; let closure = walk(&source, &capacities, policy.construction(), roots, visit)?; @@ -375,9 +376,17 @@ struct ClosureSource<'a> { /// [`MarkResult::mark_bytes`]: a pack read in several groups counts each /// extent once, and a later whole-body read replaces the partial sum. read_bytes: std::cell::RefCell>, + /// pack -> its group extents, kept from the walk's own read of the pack's + /// directory so the kept-waste pass does not read it a second time. + layouts: std::cell::RefCell>>, } impl ClosureSource<'_> { + /// Keeps the group extents the walk read for `pack_id`. + fn keep_layout(&self, pack_id: i64, views: Vec) { + self.layouts.borrow_mut().entry(pack_id).or_insert(views); + } + /// Records a whole-body read of `pack_id`. The full extent is the pack's /// body, so it replaces any group extents already counted for the pack. fn note_whole_body(&self, pack_id: i64, bytes: u64) { @@ -598,6 +607,8 @@ fn walk( bases: BTreeSet::new(), logical: roots.iter().copied().collect(), }; + // One hash for the whole walk, not one per metadata object. + let unscoped = InodeScope::from_object(ObjectId::for_bytes(b"gc.mark/object-closure")); let mut queued: BTreeSet = roots.iter().copied().collect(); let mut frontier: Vec = roots; while !frontier.is_empty() { @@ -643,7 +654,7 @@ fn walk( } else { // No other admitted role reads the scope; the leaf encoders // that derive references ignore it. - InodeScope::from_object(ObjectId::for_bytes(b"gc.mark/object-closure")) + unscoped }; decode_references(role, &canonical, construction, scope) .map_err(|_| HistoryError::Integrity("mark object references"))? @@ -708,39 +719,43 @@ fn waste( let mut dead_in_retained_packs_bytes = 0u64; for pack_id in closure.packs() { let groups = located.inventory.get(&pack_id); - // The pack row may reserve more capacity than it uses; the header names - // the used length. Everything below parses against `used`, never the - // reserved capacity. A prefix that already covers `used` is the body. - let Some((capacity, prefix)) = source - .pack_prefix(pack_id, PACK_READ_PREFIX_BYTES) - .map_err(storage)? - else { - return Err(HistoryError::Integrity("mark pack directory")); - }; - if prefix.len() < layout::USED_OFFSET + 4 { - return Err(HistoryError::Integrity("mark pack length")); - } - let used = { - let field = &prefix[layout::USED_OFFSET..layout::USED_OFFSET + 4]; - u32::from_le_bytes( - field - .try_into() - .map_err(|_| HistoryError::Integrity("mark pack length"))?, - ) as usize + // The walk already read this pack's directory when it read its + // record headers; only a pack it reached another way is read here. + let kept = source.layouts.borrow().get(&pack_id).cloned(); + let views = if let Some(views) = kept { + views + } else { + // The pack row may reserve more capacity than it uses; the header names + // the used length. Everything below parses against `used`, never the + // reserved capacity. A prefix that already covers `used` is the body. + let Some((capacity, prefix)) = source + .pack_prefix(pack_id, PACK_READ_PREFIX_BYTES) + .map_err(storage)? + else { + return Err(HistoryError::Integrity("mark pack directory")); + }; + if prefix.len() < layout::USED_OFFSET + 4 { + return Err(HistoryError::Integrity("mark pack length")); + } + let used = { + let field = &prefix[layout::USED_OFFSET..layout::USED_OFFSET + 4]; + u32::from_le_bytes( + field + .try_into() + .map_err(|_| HistoryError::Integrity("mark pack length"))?, + ) as usize + }; + if used < layout::HEADER_LEN || used > capacity { + return Err(HistoryError::Integrity("mark pack length")); + } + if prefix.len() >= used { + dead_in_retained_packs_bytes = dead_in_retained_packs_bytes + .saturating_add(pack_waste(&prefix[..used], groups, closure)?); + continue; + } + let header = layout::parse_directory_header(&prefix, used).map_err(storage)?; + layout::directory_group_views(&prefix, header).map_err(storage)? }; - if used < layout::HEADER_LEN || used > capacity { - return Err(HistoryError::Integrity("mark pack length")); - } - if prefix.len() >= used { - dead_in_retained_packs_bytes = dead_in_retained_packs_bytes.saturating_add(pack_waste( - &prefix[..used], - groups, - closure, - )?); - continue; - } - let header = layout::parse_directory_header(&prefix, used).map_err(storage)?; - let views = layout::directory_group_views(&prefix, header).map_err(storage)?; // A wholly dead group's stored extent is known from the directory. A // retained Raw group whose inventory no longer accounts for every // record has its own record directory read, so its exact per-record diff --git a/engine/crates/layerfs-persistence/src/history/objects/edges.rs b/engine/crates/layerfs-persistence/src/history/objects/edges.rs index 8d64dbcaab..b3fd1ea5fa 100644 --- a/engine/crates/layerfs-persistence/src/history/objects/edges.rs +++ b/engine/crates/layerfs-persistence/src/history/objects/edges.rs @@ -6,6 +6,11 @@ //! its group's directory, so this reads those ranges and never the payload: a //! mark costs a few hundred bytes per pack instead of the pack. Compressed //! groups and the `Ordinary` lane answer `None` and keep the decoding reader. +//! +//! The first read of a wave is one call for the heads of many packs. A pack +//! whose directory and record headers sit in its head, the common shape of a +//! pack of large records, costs no statement of its own; only a pack that +//! needs more is acquired on its own. use super::{storage, ClosureSource}; use layerfs_content::ObjectId; use layerfs_history::error::{HistoryError, HistoryResult}; @@ -16,14 +21,24 @@ use layerfs_storage::{ encoding::delta::read::ChainEdge, location::{ObjectLocation, PackInfo}, pack::layout::{self, GroupView, PackHeader}, - port::{AcquiredPackRead, PackRange, PackReadChoice, PackReadPlan, PersistenceError}, + port::{ + AcquiredPackRead, PackRange, PackReadChoice, PackReadPlan, PersistenceError, + PACK_READ_PREFIX_BYTES, + }, source::Source, StorageError, }; use std::collections::BTreeMap; -/// Count, then a wider directory, then record headers: no pack needs more. +/// After the head: a wider directory, then record headers. No pack needs more. const ROUNDS: usize = 4; +/// Bytes of each pack the first read asks for: the directory of a pack of up +/// to about sixty groups, its first group's record directory and first record +/// header. Small on purpose: on a cold cache each further page of a pack is +/// one more random read. A pack with a wider directory is read again. +const HEAD_BYTES: usize = 1024; +/// Packs whose heads one call reads. +const HEAD_PACKS: usize = 256; /// Ranges one acquisition selects; far below the port's own limits. const BATCH_RANGES: usize = 1024; /// Bytes one acquisition selects. @@ -144,58 +159,41 @@ fn resolve( Ok(merged(needs)) } -/// Selects the ranges a round needs. The first round has no directory yet, so -/// it derives its ranges from the prefix the acquisition hands it. -struct EdgeRanges<'a> { +/// One acquisition's ranges, chosen before it starts. +struct Ranges { pack_id: i64, - items: &'a [ObjectLocation], - held: &'a Pieces, - answers: &'a mut [Answer], - layout: Option<(PackHeader, Vec)>, ranges: Option>, } -impl PackReadPlan for EdgeRanges<'_> { +impl PackReadPlan for Ranges { fn select( &mut self, info: PackInfo, - prefix: &[u8], + _prefix: &[u8], ) -> Result { if info.pack_id != self.pack_id { return Err(PersistenceError::Malformed); } - if let Some(ranges) = self.ranges.take() { - return Ok(PackReadChoice::Ranges(ranges)); - } - let header = layout::parse_directory_header(prefix, info.length) - .map_err(|_| PersistenceError::Malformed)?; - let views = layout::directory_group_views(prefix, header) - .map_err(|_| PersistenceError::Malformed)?; - let mut ranges = resolve(header, &views, self.items, self.held, self.answers) - .map_err(|_| PersistenceError::Malformed)?; - if ranges.is_empty() { - // Every group of this pack is encoded. An acquisition must select - // something, and the first body byte always exists. - ranges.push(PackRange { - offset: header.body_offset, - length: 1, - }); - } - ranges.truncate(BATCH_RANGES); - self.layout = Some((header, views)); - Ok(PackReadChoice::Ranges(ranges)) + self.ranges + .take() + .map(PackReadChoice::Ranges) + .ok_or(PersistenceError::Malformed) } } -/// One acquisition of `plan`'s ranges, added to `held` and to the mark's bytes. +/// One acquisition of `ranges`, counted in the mark's bytes. fn acquire( source: &ClosureSource<'_>, pack_id: i64, - plan: &mut EdgeRanges<'_>, + ranges: Vec, ) -> HistoryResult)>> { + let mut plan = Ranges { + pack_id, + ranges: Some(ranges), + }; let read = source .mechanics - .read_scoped_pack(pack_id, plan) + .read_scoped_pack(pack_id, &mut plan) .map_err(StorageError::from) .map_err(storage)?; let AcquiredPackRead::Units(units) = read else { @@ -222,15 +220,7 @@ fn acquire_all( if batch.is_empty() { return Ok(()); } - let mut plan = EdgeRanges { - pack_id, - items: &[], - held: &Pieces::default(), - answers: &mut [], - layout: None, - ranges: Some(std::mem::take(batch)), - }; - acquired.extend(acquire(source, pack_id, &mut plan)?); + acquired.extend(acquire(source, pack_id, std::mem::take(batch))?); Ok(()) }; let mut batch = Vec::new(); @@ -279,36 +269,45 @@ pub(super) fn group_directories( Err(HistoryError::Integrity("mark group directory")) } -/// The edges of `items`, all records of pack `pack_id`: `None` where the -/// record's group is encoded and must take the decoding reader. +/// The edges of `items`, all records of pack `pack_id`, given the pack's +/// declared length and head: `None` where the record's group is encoded and +/// must take the decoding reader. fn pack_edges( source: &ClosureSource<'_>, pack_id: i64, items: &[ObjectLocation], + length: usize, + head: Vec, ) -> HistoryResult>> { + let control = layout::parse_control_header(&head, length).map_err(storage)?; + let head = if head.len() < control.body_offset { + source + .mechanics + .pack_prefix(pack_id, PACK_READ_PREFIX_BYTES) + .map_err(StorageError::from) + .map_err(storage)? + .filter(|(wide, _)| *wide == length) + .ok_or(HistoryError::Integrity("mark pack directory"))? + .1 + } else { + head + }; + let header = layout::parse_directory_header(&head, length).map_err(storage)?; + let views = + layout::directory_group_views(&head[..header.body_offset], header).map_err(storage)?; + // The head's bytes past the directory are body bytes this mark read. + source.note_scoped_body( + pack_id, + u64::try_from(head.len() - header.body_offset).unwrap_or(u64::MAX), + u64::try_from(length).unwrap_or(u64::MAX), + ); let mut held = Pieces::default(); + held.add(0, head); let mut answers = vec![Answer::Pending; items.len()]; - let pieces = { - let mut plan = EdgeRanges { - pack_id, - items, - held: &held, - answers: &mut answers, - layout: None, - ranges: None, - }; - let pieces = acquire(source, pack_id, &mut plan)?; - (plan.layout, pieces) - }; - let (Some((header, views)), first) = pieces else { - return Err(HistoryError::Integrity("mark edge directory")); - }; - for (range, bytes) in first { - held.add(range.offset, bytes); - } - for _ in 0..ROUNDS { + for _ in 0..=ROUNDS { let needs = resolve(header, &views, items, &held, &mut answers).map_err(storage)?; if needs.is_empty() { + source.keep_layout(pack_id, views); return answers .into_iter() .map(|answer| match answer { @@ -326,27 +325,13 @@ fn pack_edges( /// The header-read edges of one wave: every object of `wave` whose role names /// no other object and whose record sits in an uncompressed payload group. /// -/// `wave` is in locator order, so each pack's records are adjacent and one -/// pack is read at a time. An object absent from the result takes the decoding -/// reader, exactly as before. +/// `wave` is in locator order, so each pack's records are adjacent. An object +/// absent from the result takes the decoding reader, exactly as before. pub(super) fn wave_edges( source: &ClosureSource<'_>, wave: &[ObjectId], ) -> HistoryResult> { - let mut edges = BTreeMap::new(); - let mut items: Vec = Vec::new(); - let mut flush = |items: &mut Vec| -> HistoryResult<()> { - let Some(first) = items.first() else { - return Ok(()); - }; - for (item, edge) in items.iter().zip(pack_edges(source, first.pack_id, items)?) { - if let Some(edge) = edge { - edges.insert(item.object_id, edge); - } - } - items.clear(); - Ok(()) - }; + let mut packs: Vec<(i64, Vec)> = Vec::new(); for id in wave { let Some(location) = source.location(*id, i64::MAX).map_err(storage)? else { continue; @@ -354,14 +339,31 @@ pub(super) fn wave_edges( if location.role.has_references() { continue; } - if items - .first() - .is_some_and(|first| first.pack_id != location.pack_id) - { - flush(&mut items)?; + match packs.last_mut() { + Some((pack_id, items)) if *pack_id == location.pack_id => items.push(location), + _ => packs.push((location.pack_id, vec![location])), + } + } + let mut edges = BTreeMap::new(); + for batch in packs.chunks(HEAD_PACKS) { + let ids: Vec = batch.iter().map(|(pack_id, _)| *pack_id).collect(); + let mut heads = Vec::with_capacity(ids.len()); + source + .mechanics + .pack_prefixes(&ids, HEAD_BYTES, &mut heads) + .map_err(StorageError::from) + .map_err(storage)?; + if heads.len() != batch.len() { + return Err(HistoryError::Integrity("mark pack heads")); + } + for ((pack_id, items), (length, head)) in batch.iter().zip(heads) { + let answers = pack_edges(source, *pack_id, items, length, head)?; + for (item, edge) in items.iter().zip(answers) { + if let Some(edge) = edge { + edges.insert(item.object_id, edge); + } + } } - items.push(location); } - flush(&mut items)?; Ok(edges) } From 7b2f9acd5372cde38cb1242155332271ba068cba Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:20:25 +0800 Subject: [PATCH 04/31] perf(gc): size the SQLite rebuild's fresh file before the bulk load (layerfs-project#41) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The §7 build writes the fresh Store one page per pwrite. A CPU sample of the 1.1 GB rebuild put 70% of the copy in pwrite: on APFS a page write that also moves the end of the file costs about four times one that lands inside a sized, allocated file (1 GiB of 4 KiB writes: 1.05 s extending, 0.58 s into a sized hole, 0.28 s sized and preallocated; ext4 in the Docker VM showed no difference between the three). The run already knows the retained payload (the mark's packs and their lengths) before it copies a byte, so the build now sizes the fresh file for it once, after the schema exists and before the load: - compact/space.rs `reserve`: macOS only. One advisory F_PREALLOCATE and one set_len, rounded down to whole pages, never shrinking. A refused allocation leaves a sparse file of the same size; a load that is really out of space still meets its own Capacity refusal. - compact/space.rs `trim`: after the checked close the file is cut to the pages its header counts. The size is a lower bound of the load today (a pack body needs at least its length in pages), so nothing is cut; a build that writes less than the mark's packs (a repack of partly dead packs) would otherwise install zero pages past the database's end. - build.rs: `BuildInput.payload_bytes` carries the retained pack bytes from run.rs; the page count is read once the schema is complete. No statement is added to the copy; one PRAGMA page_count is added to the build. The installed file is byte-for-byte the size it was (1,134,886,912 bytes on the 1.1 GB shape, before and after). Measured, release, macOS APFS, one sample each, gc_bench --runs 1: - small (28 MB): rebuild 122.7 -> 106.3 ms, build 51.9 -> 36.9 ms (1-minute load 3.3 before, 4.8 after). - 1.1 GB (GC_BENCH_GROUPS=96 ... OBJECT_BYTES=131071): rebuild 1749.9 -> 1039.2 ms, build 1550.7 -> 833.2 ms; the pack copy statement alone 1305 -> 548 ms (load 3.4 before, 4.8 after). A cold source is not helped by this: its copy is bound by reads. Tests: new sqlite_compact_units `a_sized_fresh_file_loads_and_is_cut_back_to_its_page_count`. Pass: sqlite_compact (33), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (8), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13); sqlite_compact_capacity has no macOS test. No test deleted. Production LOC: 187,909 -> 187,965 (delta +56) core 40,589 -> 40,589 (0) engine 89,720 -> 89,776 (+56) layerfs-persistence +56 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent d19d98f138 and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/backend/sqlite/compact/build.rs | 16 +++- .../src/backend/sqlite/compact/mod.rs | 7 +- .../src/backend/sqlite/compact/run.rs | 1 + .../src/backend/sqlite/compact/space.rs | 75 ++++++++++++++++++ .../layerfs-persistence/src/internals.rs | 2 +- .../tests/sqlite_compact_units.rs | 77 +++++++++++++++++++ 6 files changed, 174 insertions(+), 4 deletions(-) create mode 100644 engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs index 530a6bfb61..df3a0a79d4 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs @@ -13,6 +13,7 @@ use super::op::hex; use super::op::stop_at; use super::op::visited_path; use super::op::RunOp; +use super::space; use super::types::db_error; use super::types::io_error; use super::types::CompactFailpoint; @@ -132,6 +133,9 @@ pub(crate) struct BuildInput<'a> { pub(crate) facts: &'a SealedFacts, pub(crate) tuple: &'a MarkTuple, pub(crate) synchronous: i64, + /// The stored bytes of the packs the mark retains: what the bulk load is + /// about to write, known before the fresh file holds a row. + pub(crate) payload_bytes: u64, } /// How one table's rows enter the fresh file. @@ -421,6 +425,7 @@ pub(crate) fn build( facts, tuple, synchronous, + payload_bytes, } = input; let store = &facts.path; let target = compact_path(store); @@ -509,6 +514,10 @@ PRAGMA user_version={user_version};PRAGMA application_id={application_id};" // inside the bulk load leaves (the in-process arm tears this file, and the // next run removes the whole recorded set instead of trusting it). stop_at(request, CompactFailpoint::DuringCopy, op, store)?; + // The load appends the retained payload: size the file for it once, so + // its page writes land inside the file instead of each moving its end. + let page_bytes = u64::try_from(page_size).unwrap_or(0); + space::reserve(&target, payload_bytes, page_bytes)?; let target_tables = main_tables(&connection)?; copy_rows(&connection, op, tuple.generation + 1, &target_tables)?; @@ -562,6 +571,11 @@ WHERE sql IS NOT NULL AND substr(name,1,7)!='sqlite_' ORDER BY type,name", |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), ) .map_err(db_error)?; + // The schema is complete, so the header's page count is final: the file + // ends there, whatever the load was sized for. + let pages: i64 = connection + .query_row("PRAGMA main.page_count", params![], |row| row.get(0)) + .map_err(db_error)?; // §7 step 5: the same raw connection switches the fresh file to WAL, so the // mode switch runs under `journal_mode=OFF` and writes no rollback journal // beside the staged name; then checkpoint and leave no sidecar. @@ -599,7 +613,7 @@ WHERE sql IS NOT NULL AND substr(name,1,7)!='sqlite_' ORDER BY type,name", return Err(HistoryError::Integrity("staged Store sidecar")); } } - let target_bytes = std::fs::metadata(&target).map_err(io_error)?.len(); + let target_bytes = space::trim(&target, u64::try_from(pages).unwrap_or(0), page_bytes)?; Ok(Built { target, target_bytes, diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs index 45c1f261b3..1c0d336313 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs @@ -23,20 +23,23 @@ //! sweep to refine the bound with (#42 A10). //! //! The module is split by phase: [`types`] (the §7/§11 shapes), [`op`] (the -//! `.compact-op` sidecar), [`build`], [`verify`], [`install`], [`receipt`] and -//! [`run`] (the orchestration on `SqliteHandles`). +//! `.compact-op` sidecar), [`build`] with [`space`] (the fresh file's size +//! during the load), [`verify`], [`install`], [`receipt`] and [`run`] (the +//! orchestration on `SqliteHandles`). mod build; mod install; mod op; mod receipt; mod run; +mod space; mod types; mod verify; pub use build::{peak_temp_bytes, source_mmap, SRC_MMAP_BYTES}; pub use op::{compact_op_status, CompactOpStatus}; pub use receipt::receipt; +pub use space::{reserve, trim}; pub use types::io_error; pub use types::{cap_delta_references, DELTA_BASE_REFERENCES_CAP}; pub use types::{ diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs index 1d96ec7d45..344e394f26 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs @@ -374,6 +374,7 @@ impl SqliteHandles { facts: &facts, tuple: &tuple, synchronous, + payload_bytes: live_stored, }, capacity, request, diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs new file mode 100644 index 0000000000..6801651760 --- /dev/null +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs @@ -0,0 +1,75 @@ +//! The fresh file's size during the §7 bulk load. +//! +//! SQLite writes the load one page per `pwrite`. On APFS a page write that +//! also moves the end of the file costs about four times one that lands inside +//! an already sized, already allocated file (measured: 1 GiB of 4 KiB writes +//! in about 1.05 s extending, 0.58 s into a sized hole, 0.28 s into a sized and +//! preallocated file). The rebuild knows the retained payload before it copies +//! a byte, so it sizes the file once instead of 250,000 times per gigabyte. +//! +//! Ext4 showed no difference between the three shapes, so only macOS sizes the +//! file. Nothing here changes a byte SQLite reads back: the sized region is a +//! hole the load then fills, and [`trim`] cuts whatever the load did not reach. + +use super::types::io_error; +use layerfs_history::error::HistoryResult; +use std::path::Path; + +/// Sizes the fresh file to `bytes` before the bulk load, and on macOS asks +/// APFS for that much space in one allocation. +/// +/// `bytes` is rounded down to whole pages and never shrinks the file. The +/// allocation request is advisory: a volume that refuses it leaves a sparse +/// file of the same size, and a load that really is out of space still meets +/// its own `Capacity` refusal at the write. +#[cfg(target_os = "macos")] +pub fn reserve(target: &Path, bytes: u64, page_size: u64) -> HistoryResult<()> { + use nix::fcntl::fcntl; + use nix::fcntl::FcntlArg; + + let bytes = bytes - bytes % page_size.max(1); + let file = std::fs::OpenOptions::new() + .write(true) + .open(target) + .map_err(io_error)?; + let present = file.metadata().map_err(io_error)?.len(); + if bytes <= present { + return Ok(()); + } + if let Ok(length) = i64::try_from(bytes - present) { + let mut request = nix::libc::fstore_t { + fst_flags: nix::libc::F_ALLOCATEALL, + fst_posmode: nix::libc::F_PEOFPOSMODE, + fst_offset: 0, + fst_length: length, + fst_bytesalloc: 0, + }; + let _ = fcntl(&file, FcntlArg::F_PREALLOCATE(&mut request)); + } + file.set_len(bytes).map_err(io_error) +} + +/// Other platforms leave the file to grow with the load. +#[cfg(not(target_os = "macos"))] +pub fn reserve(_target: &Path, _bytes: u64, _page_size: u64) -> HistoryResult<()> { + Ok(()) +} + +/// Cuts the closed fresh file back to the `pages` its header counts. +/// +/// A load that wrote less than [`reserve`] sized leaves zero pages past the +/// database's own end; SQLite ignores them, but they would be installed and +/// counted as Store bytes. Returns the file's final size. +pub fn trim(target: &Path, pages: u64, page_size: u64) -> HistoryResult { + let length = std::fs::metadata(target).map_err(io_error)?.len(); + let counted = pages.saturating_mul(page_size); + if counted == 0 || length <= counted { + return Ok(length); + } + std::fs::OpenOptions::new() + .write(true) + .open(target) + .and_then(|file| file.set_len(counted)) + .map_err(io_error)?; + Ok(counted) +} diff --git a/engine/crates/layerfs-persistence/src/internals.rs b/engine/crates/layerfs-persistence/src/internals.rs index a5f29c82b5..f24a9d06cc 100644 --- a/engine/crates/layerfs-persistence/src/internals.rs +++ b/engine/crates/layerfs-persistence/src/internals.rs @@ -10,6 +10,6 @@ pub use crate::backend::postgresql::compact::{ pub use crate::backend::postgresql::session::rows::sqlstate; pub use crate::backend::records::BackendError; pub use crate::backend::sqlite::compact::{ - cap_delta_references, io_error, peak_temp_bytes, receipt, source_mmap, + cap_delta_references, io_error, peak_temp_bytes, receipt, reserve, source_mmap, trim, DELTA_BASE_REFERENCES_CAP, SRC_MMAP_BYTES, }; diff --git a/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs b/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs index c7dc7b9ac7..1389ac5fb2 100644 --- a/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs +++ b/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs @@ -184,3 +184,80 @@ fn the_attached_source_gets_a_large_mmap_window() { "the applied window reads back" ); } + +/// The rebuild sizes the fresh file for the retained payload before the bulk +/// load (macOS; other platforms leave it to grow) and cuts the closed file +/// back to the pages its header counts, so a load that writes less than it +/// was sized for installs no page past the database's own end. +#[test] +fn a_sized_fresh_file_loads_and_is_cut_back_to_its_page_count() { + use layerfs_persistence::internals::{reserve, trim}; + use rusqlite::{params, Connection}; + + const PAGE: u64 = 4096; + let temp = support::Temp::new("compact-space"); + let staged = temp.join("staged.sqlite"); + let connection = Connection::open(&staged).unwrap(); + connection + .execute_batch( + "PRAGMA page_size=4096; PRAGMA journal_mode=OFF; PRAGMA synchronous=OFF; \ + CREATE TABLE pack(pack_id INTEGER PRIMARY KEY, body BLOB NOT NULL);", + ) + .unwrap(); + let schema_only = std::fs::metadata(&staged).unwrap().len(); + + // Sized for 1 MiB and a part of a page; the load below writes far less. + reserve(&staged, 1024 * 1024 + 100, PAGE).unwrap(); + let sized = std::fs::metadata(&staged).unwrap().len(); + if cfg!(target_os = "macos") { + assert_eq!(sized, 1024 * 1024, "whole pages only"); + } else { + assert_eq!(sized, schema_only, "only macOS sizes the file"); + } + // A size the file already has is never cut. + reserve(&staged, PAGE, PAGE).unwrap(); + assert_eq!(std::fs::metadata(&staged).unwrap().len(), sized); + + connection + .execute_batch( + "BEGIN IMMEDIATE; \ + INSERT INTO pack VALUES (1, zeroblob(100000)); \ + INSERT INTO pack VALUES (2, randomblob(200000)); \ + COMMIT;", + ) + .unwrap(); + let pages: i64 = connection + .query_row("PRAGMA page_count", params![], |row| row.get(0)) + .unwrap(); + let pages = u64::try_from(pages).unwrap(); + connection.close().unwrap(); + assert!( + pages * PAGE < 1024 * 1024, + "the load is smaller than its size" + ); + + assert_eq!(trim(&staged, pages, PAGE).unwrap(), pages * PAGE); + assert_eq!(std::fs::metadata(&staged).unwrap().len(), pages * PAGE); + // Nothing to cut the second time, and a count past the file cuts nothing. + assert_eq!(trim(&staged, pages, PAGE).unwrap(), pages * PAGE); + assert_eq!(trim(&staged, pages + 8, PAGE).unwrap(), pages * PAGE); + + let reopened = Connection::open(&staged).unwrap(); + let check: String = reopened + .query_row("PRAGMA integrity_check", params![], |row| row.get(0)) + .unwrap(); + assert_eq!(check, "ok"); + let lengths: (i64, i64) = reopened + .query_row( + "SELECT (SELECT length(body) FROM pack WHERE pack_id=1),\ + (SELECT length(body) FROM pack WHERE pack_id=2)", + params![], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .unwrap(); + assert_eq!(lengths, (100_000, 200_000)); + let reread: i64 = reopened + .query_row("PRAGMA page_count", params![], |row| row.get(0)) + .unwrap(); + assert_eq!(u64::try_from(reread).unwrap(), pages); +} From d46387933601271a34ee673101f646de0f9f5323 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:29:36 +0800 Subject: [PATCH 05/31] perf(gc): copy packs in windows and read the sealed Store ahead (layerfs-project#41) One INSERT ... SELECT over every retained pack reads the sealed file in the order SQLite stored it: a pack's row, then its payload pages a few dozen pages behind that row, then the next row. No kernel read-ahead follows that pattern, so a source that is not in memory is read one pack per device round trip. On a 1.26 GB Store written past the buffer cache (F_NOCACHE) the pack copy took 3.4 to 3.5 s, against 0.55 s for the same Store in memory; a plain sequential read of that file runs at 4.4 GB/s cold. The 9.8 GB bench shape was in this state (its first mark ran 3x slower than its second), which is where its 225 MB/s build came from. - compact/window.rs `windows`: cuts the retained packs into runs of 64 MiB of retained payload, ended early by a dead run over 8 MiB. Packs are appended in id order, so a pack's payload starts no earlier than the summed lengths of the packs before it and no later than that plus every byte of the file that is not pack payload; a window's file range is that bound over its packs. Both inputs (every pack's length, the mark's packs) were already in memory in run.rs. - compact/window.rs `ReadAhead`: before each window, asks the kernel for the window's range through a second read-only descriptor, each byte once. macOS: F_RDADVISE, which returns when the range is in memory (about 5 GB/s cold, 4 ms per GB warm). Linux: POSIX_FADV_WILLNEED in 128 KiB steps (the kernel caps one request at the device read-ahead size). A hint only: no byte and no result changes, and a file laid out in another order only loses the benefit. The sealed file is attached immutable=1, so SQLite holds no lock on it that closing the descriptor could drop. - build.rs `copy_packs`: the payload table (`pack`, or `pack_unit` in the group-row layouts) is copied one window per execution of one prepared statement, `WHERE pack_id BETWEEN ?1 AND ?2 AND `. The retained filter and every other table's copy are unchanged. Statements: the pack copy is one prepared statement run once per 64 MiB of retained payload (17 on the 1.1 GB shape, 1 on the small one) instead of one statement run once. Rows, bytes and the installed file are identical. Measured, release, macOS APFS, one sample each. Cold = the Store copy was written with F_NOCACHE just before the run (the cache state is declared by method, not verified); rebuild_probe is a local diagnostic, the same compaction call gc_bench makes. - 1.1 GB cold: pack copy 3483 -> 690 ms, build 3710 -> 865 ms, rebuild wall 5054 -> 2071 ms (load 6.1 and 6.8). The mark is 0.8 to 0.9 s of what remains: it is bound by the same one-pack-per-round-trip reads. - small (28 MB) cold: pack copy 5.9 -> 6.2 ms, no change (one window). - 1.1 GB warm, gc_bench --runs 1: rebuild 1039 -> 1638 ms in a sample taken at load 5.9 to 7.2 in which the untouched mark also ran 96 -> 175 ms; the instrumented warm run beside it (load 6.6) gave pack copy 583 ms in 17 windows against 548 ms as one statement, build 734 ms, rebuild 1082 ms. Small warm, gc_bench: 106 -> 146 ms in the same loaded sample (mark 17 -> 22 ms, none path 25 -> 51 ms). Linux was type-checked for aarch64-unknown-linux-musl and x86_64-unknown-linux-gnu in a scratch crate (the package itself does not cross-check from this host); it was not run or measured. Tests: new sqlite_compact `retained_packs_on_both_sides_of_a_long_dead_run_are_all_copied` (a two window rebuild) and sqlite_compact_units `the_pack_copy_is_cut_into_windows_in_pack_order`. The generated matrix report counts the new sqlite_compact test (34 -> 35). Pass: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). No test deleted. Production LOC: 187,965 -> 188,160 (delta +195) core 40,589 -> 40,589 (0) engine 89,776 -> 89,971 (+195) layerfs-persistence +195 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent 7b2f9acd53 and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../gc-t12.3a-sqlite-matrix-report.md | 2 +- .../src/backend/sqlite/compact/build.rs | 69 +++++- .../src/backend/sqlite/compact/mod.rs | 5 +- .../src/backend/sqlite/compact/run.rs | 2 + .../src/backend/sqlite/compact/window.rs | 199 ++++++++++++++++++ .../layerfs-persistence/src/internals.rs | 4 +- .../tests/sqlite_compact.rs | 99 +++++++++ .../tests/sqlite_compact_units.rs | 98 +++++++++ 8 files changed, 469 insertions(+), 9 deletions(-) create mode 100644 engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs diff --git a/core/docs/issues/project-41/gc-t12.3a-sqlite-matrix-report.md b/core/docs/issues/project-41/gc-t12.3a-sqlite-matrix-report.md index 34f634fa6a..f9c47e4544 100644 --- a/core/docs/issues/project-41/gc-t12.3a-sqlite-matrix-report.md +++ b/core/docs/issues/project-41/gc-t12.3a-sqlite-matrix-report.md @@ -263,7 +263,7 @@ AC-51 does not depend on the projection alone. `/var/tmp`), rustc 1.85.1, `SqlitePersistenceProfile::Disposable` (WAL/OFF), `CARGO_TARGET_DIR=/tmp/target-t123c`. - Verified here: `sqlite_compact_matrix` (1 test); the persistence suites - `sqlite_compact` (34), `sqlite_compact_crash` (24), + `sqlite_compact` (35), `sqlite_compact_crash` (24), `sqlite_compact_fixtures` (1), `admission` (10), `path_removal` (4), `branch_removal` (4), `project_delete` (4), `epoch_fence` (7), `gc_backfill` (5), `kept_objects` (7); the Store-service path suites diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs index df3a0a79d4..c7afc41a30 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs @@ -20,6 +20,9 @@ use super::types::CompactFailpoint; use super::types::CompactRequest; use super::types::MarkTuple; use super::verify::immutable_uri; +use super::window::windows; +use super::window::ReadAhead; +use super::window::Window; use crate::backend::records::BackendError; use crate::backend::sqlite::lifecycle::seal::SealedFacts; use crate::backend::sqlite::session::connection::Session; @@ -136,6 +139,10 @@ pub(crate) struct BuildInput<'a> { /// The stored bytes of the packs the mark retains: what the bulk load is /// about to write, known before the fresh file holds a row. pub(crate) payload_bytes: u64, + /// Every pack of the sealed Store and its stored length. + pub(crate) lengths: &'a BTreeMap, + /// The packs the mark retains, ascending. + pub(crate) retained: &'a [i64], } /// How one table's rows enter the fresh file. @@ -159,10 +166,8 @@ enum Copy { /// admission intent, so the installed Store carries the row the settle clears. fn copy_rule(table: &str) -> Copy { match table { - "pack" | "pack_unit" => { - // A pack is one unit: a pack with any live object is copied whole. - Copy::Where("pack_id IN (SELECT pack_id FROM scratch.live_pack)") - } + // A pack is one unit: a pack with any live object is copied whole. + "pack" | "pack_unit" => Copy::Where(RETAINED_PACK), "object_location" | "content_signature" => { Copy::Where("object_id IN (SELECT object_id FROM scratch.visited)") } @@ -209,6 +214,35 @@ UNION SELECT run FROM src.commit_lineage WHERE commit_id IN \ } } +/// The filter that keeps a retained pack's rows. +const RETAINED_PACK: &str = "pack_id IN (SELECT pack_id FROM scratch.live_pack)"; + +/// The pack copy: the rows of the table that holds the pack payload, one +/// window of retained packs per statement. Before each window the sealed file +/// is asked for the part the window reads, so a source that is not in memory +/// arrives in sequential requests instead of one pack per round trip. +fn copy_packs( + connection: &Connection, + table: &str, + plan: &[Window], + sealed: &mut ReadAhead, +) -> HistoryResult<()> { + let name = identifier(table); + let mut statement = connection + .prepare(&format!( + "INSERT INTO main.{name} SELECT * FROM src.{name} \ +WHERE pack_id BETWEEN ?1 AND ?2 AND {RETAINED_PACK}" + )) + .map_err(db_error)?; + for window in plan { + sealed.ask(window); + statement + .execute(params![window.first, window.last]) + .map_err(db_error)?; + } + Ok(()) +} + /// The `store_policy` columns the rebuild copies, named explicitly: the fresh /// file's schema is a copy of the source's `sqlite_master`, so `SELECT *` would /// follow whatever column order that schema happens to have. The named list @@ -294,7 +328,16 @@ fn copy_rows( op: &RunOp, generation: i64, target_tables: &BTreeSet, + plan: &[Window], + sealed: &mut ReadAhead, ) -> HistoryResult<()> { + // The pack payload is `pack.body` in a Monolithic Store and the + // `pack_unit` rows otherwise; that table is copied window by window. + let payload = if target_tables.contains("pack_unit") { + "pack_unit" + } else { + "pack" + }; connection .execute_batch("BEGIN IMMEDIATE") .map_err(db_error)?; @@ -319,6 +362,10 @@ fn copy_rows( if !target_tables.contains(table) { continue; } + if table == payload { + copy_packs(connection, table, plan, sealed)?; + continue; + } let name = identifier(table); let statement = if table == "gc_intent" { format!( @@ -426,6 +473,8 @@ pub(crate) fn build( tuple, synchronous, payload_bytes, + lengths, + retained, } = input; let store = &facts.path; let target = compact_path(store); @@ -519,7 +568,17 @@ PRAGMA user_version={user_version};PRAGMA application_id={application_id};" let page_bytes = u64::try_from(page_size).unwrap_or(0); space::reserve(&target, payload_bytes, page_bytes)?; let target_tables = main_tables(&connection)?; - copy_rows(&connection, op, tuple.generation + 1, &target_tables)?; + let plan = windows(lengths, retained, facts.bytes)?; + let mut sealed = ReadAhead::open(store)?; + copy_rows( + &connection, + op, + tuple.generation + 1, + &target_tables, + &plan, + &mut sealed, + )?; + drop(sealed); // §7 step 4: the secondary indexes and the immutability/logical triggers // are built after the bulk load, so the load writes no index page and fires diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs index 1c0d336313..f44d4845c6 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/mod.rs @@ -24,7 +24,8 @@ //! //! The module is split by phase: [`types`] (the §7/§11 shapes), [`op`] (the //! `.compact-op` sidecar), [`build`] with [`space`] (the fresh file's size -//! during the load), [`verify`], [`install`], [`receipt`] and [`run`] (the +//! during the load) and [`window`] (the pack copy's windows and the sealed +//! file's read-ahead), [`verify`], [`install`], [`receipt`] and [`run`] (the //! orchestration on `SqliteHandles`). mod build; @@ -35,6 +36,7 @@ mod run; mod space; mod types; mod verify; +mod window; pub use build::{peak_temp_bytes, source_mmap, SRC_MMAP_BYTES}; pub use op::{compact_op_status, CompactOpStatus}; @@ -46,3 +48,4 @@ pub use types::{ CompactFailpoint, CompactKeptWaste, CompactPath, CompactReceipt, CompactRequest, CompactResidue, CompactZeroReason, OwnerLiveness, }; +pub use window::{windows, Window, GAP_BYTES, WINDOW_BYTES}; diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs index 344e394f26..2482d4a991 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs @@ -375,6 +375,8 @@ impl SqliteHandles { tuple: &tuple, synchronous, payload_bytes: live_stored, + lengths: &lengths, + retained: mark.objects().packs(), }, capacity, request, diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs new file mode 100644 index 0000000000..74028d92a3 --- /dev/null +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs @@ -0,0 +1,199 @@ +//! The pack copy's windows and the sealed file's read-ahead. +//! +//! One `INSERT ... SELECT` over every retained pack reads the sealed file in +//! the order SQLite stored it: a pack's row, then its payload pages a few +//! dozen pages *behind* that row, then the next row. No kernel reads ahead +//! across that pattern, so a source that is not already in memory is read one +//! pack per device round trip (measured on a 1.26 GB Store written past the +//! cache: 3.4 s for the copy, against 0.55 s once the file is in memory). +//! +//! The copy therefore runs one bounded window of packs at a time, and before +//! each window asks the kernel for the part of the file the window will read, +//! in one sequential request. The request is a hint: it changes no byte and no +//! result, and a file laid out in another order only loses the benefit. + +use super::types::io_error; +use layerfs_history::error::HistoryError; +use layerfs_history::error::HistoryResult; +use std::collections::BTreeMap; +use std::fs::File; +use std::path::Path; + +/// Retained payload bytes one window copies before the next read-ahead. +pub const WINDOW_BYTES: u64 = 64 * 1024 * 1024; + +/// A run of dead packs this long ends the window, so it is not read ahead. +pub const GAP_BYTES: u64 = 8 * 1024 * 1024; + +/// One run of retained packs and the part of the sealed file it reads. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct Window { + /// The first retained pack of the run. + pub first: i64, + /// The last retained pack of the run. + pub last: i64, + /// Where the run's payload cannot start before, in file bytes. + pub offset: u64, + /// How far past `offset` the run's payload can reach. + pub bytes: u64, +} + +/// Cuts the retained packs into windows, in pack order. +/// +/// `lengths` is every pack of the sealed Store and `retained` the ascending +/// packs the mark keeps. A Store's packs are appended in id order, so pack +/// `p`'s payload starts no earlier than the summed lengths of the packs before +/// it and no later than that plus every byte of the file that is not pack +/// payload; a window's range is that bound over its packs. +pub fn windows( + lengths: &BTreeMap, + retained: &[i64], + file_bytes: u64, +) -> HistoryResult> { + let total = lengths.values().copied().fold(0u64, u64::saturating_add); + let slack = file_bytes.saturating_sub(total); + let mut planned = Vec::new(); + // The open run: its window so far, the payload offset it ends at and the + // retained bytes it holds. + let mut open: Option<(Window, u64, u64)> = None; + let mut wanted = retained.iter().copied().peekable(); + let mut before = 0u64; + for (&pack, &length) in lengths { + if wanted.peek() == Some(&pack) { + wanted.next(); + if let Some((window, end, held)) = open { + if held >= WINDOW_BYTES || before.saturating_sub(end) > GAP_BYTES { + planned.push(closed(window, end, slack)); + open = None; + } + } + let end = before.saturating_add(length); + open = Some(match open { + Some((window, _, held)) => ( + Window { + last: pack, + ..window + }, + end, + held.saturating_add(length), + ), + None => ( + Window { + first: pack, + last: pack, + offset: before, + bytes: 0, + }, + end, + length, + ), + }); + } + before = before.saturating_add(length); + } + if wanted.next().is_some() { + // A retained pack the Store does not hold, or an unordered mark. + return Err(HistoryError::Integrity("mark pack")); + } + if let Some((window, end, _)) = open { + planned.push(closed(window, end, slack)); + } + Ok(planned) +} + +fn closed(window: Window, end: u64, slack: u64) -> Window { + Window { + bytes: end.saturating_sub(window.offset).saturating_add(slack), + ..window + } +} + +/// The sealed file's read-ahead: a second read-only descriptor the kernel is +/// asked through. The file is attached `immutable=1`, so SQLite holds no lock +/// on it that closing this descriptor could drop. +pub(crate) struct ReadAhead { + file: File, + length: u64, + /// Everything before this offset has been asked for already. + asked: u64, +} + +impl ReadAhead { + pub(crate) fn open(sealed: &Path) -> HistoryResult { + let file = File::open(sealed).map_err(io_error)?; + let length = file.metadata().map_err(io_error)?.len(); + Ok(Self { + file, + length, + asked: 0, + }) + } + + /// Asks the kernel to bring the window's part of the file into memory. + /// Windows arrive in file order, so a byte is asked for once. + pub(crate) fn ask(&mut self, window: &Window) { + let start = window.offset.max(self.asked); + let end = window.offset.saturating_add(window.bytes).min(self.length); + if start < end { + advise(&self.file, start, end); + self.asked = end; + } + } +} + +/// macOS: one advisory read per gigabyte of the range. The call returns once +/// the range is in memory (about 5 GB/s cold here, about 4 ms per gigabyte +/// when it already is). +#[cfg(target_os = "macos")] +fn advise(file: &File, start: u64, end: u64) { + use nix::fcntl::fcntl; + use nix::fcntl::FcntlArg; + + const STEP: u64 = 1 << 30; + let mut offset = start; + while offset < end { + let count = (end - offset).min(STEP); + let (Ok(ra_offset), Ok(ra_count)) = ( + nix::libc::off_t::try_from(offset), + nix::libc::c_int::try_from(count), + ) else { + return; + }; + let request = nix::libc::radvisory { + ra_offset, + ra_count, + }; + if fcntl(file, FcntlArg::F_RDADVISE(request)).is_err() { + return; + } + offset += count; + } +} + +/// Linux: `POSIX_FADV_WILLNEED` starts the reads and returns. The kernel caps +/// one request at the device's read-ahead size, which is at least 128 KiB by +/// default, so the range is asked for in steps of that size. +#[cfg(target_os = "linux")] +fn advise(file: &File, start: u64, end: u64) { + use nix::fcntl::posix_fadvise; + use nix::fcntl::PosixFadviseAdvice; + + const STEP: u64 = 128 * 1024; + let mut offset = start; + while offset < end { + let count = (end - offset).min(STEP); + let (Ok(at), Ok(len)) = ( + nix::libc::off_t::try_from(offset), + nix::libc::off_t::try_from(count), + ) else { + return; + }; + if posix_fadvise(file, at, len, PosixFadviseAdvice::POSIX_FADV_WILLNEED).is_err() { + return; + } + offset += count; + } +} + +#[cfg(not(any(target_os = "macos", target_os = "linux")))] +fn advise(_file: &File, _start: u64, _end: u64) {} diff --git a/engine/crates/layerfs-persistence/src/internals.rs b/engine/crates/layerfs-persistence/src/internals.rs index f24a9d06cc..180eb47435 100644 --- a/engine/crates/layerfs-persistence/src/internals.rs +++ b/engine/crates/layerfs-persistence/src/internals.rs @@ -10,6 +10,6 @@ pub use crate::backend::postgresql::compact::{ pub use crate::backend::postgresql::session::rows::sqlstate; pub use crate::backend::records::BackendError; pub use crate::backend::sqlite::compact::{ - cap_delta_references, io_error, peak_temp_bytes, receipt, reserve, source_mmap, trim, - DELTA_BASE_REFERENCES_CAP, SRC_MMAP_BYTES, + cap_delta_references, io_error, peak_temp_bytes, receipt, reserve, source_mmap, trim, windows, + Window, DELTA_BASE_REFERENCES_CAP, GAP_BYTES, SRC_MMAP_BYTES, WINDOW_BYTES, }; diff --git a/engine/crates/layerfs-persistence/tests/sqlite_compact.rs b/engine/crates/layerfs-persistence/tests/sqlite_compact.rs index 437262c0f6..99dff2f892 100644 --- a/engine/crates/layerfs-persistence/tests/sqlite_compact.rs +++ b/engine/crates/layerfs-persistence/tests/sqlite_compact.rs @@ -434,6 +434,105 @@ fn a_wholly_dead_pack_is_rebuilt_away() { clean(&reopened); } +/// The pack copy runs one window of retained packs per statement, and a long +/// run of dead packs ends a window. Retained packs on both sides of such a run +/// are two windows, and both arrive in the rebuilt Store. +#[test] +fn retained_packs_on_both_sides_of_a_long_dead_run_are_all_copied() { + use layerfs_persistence::internals::{windows, GAP_BYTES}; + use std::collections::{BTreeMap, BTreeSet}; + + let (_temp, handles, path) = create_store(); + let early = filesystem_root(&handles, 1); + initialize(&handles, 1, early); + // More than `GAP_BYTES` of wholly dead packs. The bytes do not compress, + // so the stored run is as long as the payload. + let mut state = 0x9e37_79b9_7f4a_7c15_u64; + let mut dead = Vec::new(); + for _ in 0..120 { + let bytes: Vec = (0..100_000) + .map(|_| { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + state.to_le_bytes()[0] + }) + .collect(); + dead.push(whole_file(&handles, &bytes)); + } + let late = filesystem_root(&handles, 2); + initialize(&handles, 2, late); + let before = generation(&handles); + + // The plan the build will make: the retained packs are every pack that + // holds no dead object, and the dead run splits them into two windows. + let (lengths, retained, dead_bytes) = { + let connection = rusqlite::Connection::open_with_flags( + &path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, + ) + .unwrap(); + let mut dead_packs = BTreeSet::new(); + for id in &dead { + let pack: i64 = connection + .query_row( + "SELECT pack_id FROM object_location WHERE object_id=?1", + rusqlite::params![id.as_bytes().as_slice()], + |row| row.get(0), + ) + .unwrap(); + dead_packs.insert(pack); + } + let mut statement = connection + .prepare("SELECT pack_id,length FROM pack ORDER BY pack_id") + .unwrap(); + let lengths: BTreeMap = statement + .query_map([], |row| { + Ok((row.get::<_, i64>(0)?, row.get::<_, i64>(1)? as u64)) + }) + .unwrap() + .map(Result::unwrap) + .collect(); + let retained: Vec = lengths + .keys() + .copied() + .filter(|pack| !dead_packs.contains(pack)) + .collect(); + let dead_bytes: u64 = dead_packs.iter().map(|pack| lengths[pack]).sum(); + (lengths, retained, dead_bytes) + }; + assert!(dead_bytes > GAP_BYTES, "the dead run is {dead_bytes} bytes"); + let file = std::fs::metadata(&path).unwrap().len(); + let plan = windows(&lengths, &retained, file).unwrap(); + assert_eq!(plan.len(), 2, "{plan:?}"); + assert!(plan[0].last < plan[1].first); + + let receipt = handles.compact(&request(48)).unwrap(); + assert_eq!(receipt.path, CompactPath::Rebuild); + assert_eq!(receipt.dead_bytes_bound, dead_bytes); + let reopened = reopen(&path); + assert_eq!(generation(&reopened), before + 1); + for id in [early, late] { + assert!(located(&reopened, id), "retained root {id} survives"); + } + for id in &dead { + assert!(!located(&reopened, *id), "dead object {id} is collected"); + } + let rebuilt = + rusqlite::Connection::open_with_flags(&path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY) + .unwrap(); + let packs: Vec = rebuilt + .prepare("SELECT pack_id FROM pack ORDER BY pack_id") + .unwrap() + .query_map([], |row| row.get(0)) + .unwrap() + .map(Result::unwrap) + .collect(); + assert_eq!(packs, retained, "every retained pack, and only those"); + drop(rebuilt); + clean(&reopened); +} + #[test] fn retained_history_and_its_objects_survive_a_rebuild() { let (_temp, handles, path) = create_store(); diff --git a/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs b/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs index 1389ac5fb2..7548a3ffa5 100644 --- a/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs +++ b/engine/crates/layerfs-persistence/tests/sqlite_compact_units.rs @@ -261,3 +261,101 @@ fn a_sized_fresh_file_loads_and_is_cut_back_to_its_page_count() { .unwrap(); assert_eq!(u64::try_from(reread).unwrap(), pages); } + +/// The pack copy runs one window of retained packs at a time. A window ends +/// once it holds [`WINDOW_BYTES`] of retained payload or when a run of dead +/// packs longer than [`GAP_BYTES`] follows it, and its file range is the +/// payload offsets of its packs plus every byte of the file that is not pack +/// payload. +#[test] +fn the_pack_copy_is_cut_into_windows_in_pack_order() { + use layerfs_persistence::internals::{windows, Window, GAP_BYTES, WINDOW_BYTES}; + use std::collections::BTreeMap; + + const MIB: u64 = 1024 * 1024; + let window = std::hint::black_box(WINDOW_BYTES); + let gap = std::hint::black_box(GAP_BYTES); + assert_eq!((window, gap), (64 * MIB, 8 * MIB)); + + // Ten packs of 16 MiB, all retained, in a file with 1 MiB that is not pack + // payload: four packs fill a window. + let lengths: BTreeMap = (1..=10).map(|pack| (pack, 16 * MIB)).collect(); + let retained: Vec = (1..=10).collect(); + let file = 160 * MIB + MIB; + assert_eq!( + windows(&lengths, &retained, file).unwrap(), + vec![ + Window { + first: 1, + last: 4, + offset: 0, + bytes: 65 * MIB + }, + Window { + first: 5, + last: 8, + offset: 64 * MIB, + bytes: 65 * MIB + }, + Window { + first: 9, + last: 10, + offset: 128 * MIB, + bytes: 33 * MIB + }, + ] + ); + + // A short dead run stays inside its window (the filter skips its rows); a + // long one ends the window, so the dead bytes are not read ahead. + let lengths: BTreeMap = [ + (1, MIB), + (2, MIB), // dead, short + (3, MIB), + (4, 9 * MIB), // dead, longer than the gap + (5, MIB), + (7, 2 * MIB), // dead, last + ] + .into_iter() + .collect(); + assert_eq!( + windows(&lengths, &[1, 3, 5], 15 * MIB).unwrap(), + vec![ + Window { + first: 1, + last: 3, + offset: 0, + bytes: 3 * MIB + }, + Window { + first: 5, + last: 5, + offset: 12 * MIB, + bytes: MIB + }, + ] + ); + + // No retained pack is no window; a file shorter than its packs has no + // slack to add. + assert_eq!(windows(&lengths, &[], 15 * MIB).unwrap(), Vec::new()); + assert_eq!( + windows(&lengths, &[7], MIB).unwrap(), + vec![Window { + first: 7, + last: 7, + offset: 13 * MIB, + bytes: 2 * MIB + }] + ); + + // A retained pack the Store does not hold, or a mark out of order, is an + // integrity failure, never a silently shorter copy. + for broken in [&[6][..], &[3, 1][..], &[1, 1][..], &[5, 8][..]] { + let error = windows(&lengths, broken, 15 * MIB).unwrap_err(); + assert!( + matches!(error, HistoryError::Integrity("mark pack")), + "{broken:?}: {error:?}" + ); + } +} From 7f5bb75b34ed310724631d1da6aaa738a7286f68 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:34:37 +0800 Subject: [PATCH 06/31] perf(gc): drop the post-mark record rewrite and the post-build directory sync (layerfs-project#41) Two steps of a SQLite rebuild changed no outcome: - After the mark the run rewrote `.compact-op` (write, fsync, rename, directory fsync) only to add the mark's tuple and digest. No reader uses them: a later call judges a leftover by the run, generation and owner the admission record already names, and `store.status` reads the admission phase as `mark`. The values go out with the next write (`build`). A stop point still writes its own record with them. - After the build the run synchronized the Store's directory "before the verification reads them back". Verification reads through the page cache, the fresh file is synchronized before the rename and the directory after it, and the record written next is synchronized itself. Round trips per rebuild: 6 op record writes -> 5, 16 fsync calls -> 13 (each is F_FULLFSYNC on macOS). The `none` path is unchanged. docs/architecture/engine/store/sqlite.md gains a "Compaction rebuild" section. It also covers the two previous commits (the fresh file's size, the pack copy's windows and the read-ahead), which changed the build's steps without a page to update. Measured, release, macOS APFS, gc_bench --runs 1, one sample each (1-minute load 3.0 to 3.5): - small (28 MB): rebuild 106.3 -> 95.1 ms against the last sample taken at a comparable load (the first commit of this series); mark phase 17.0 -> 8.2 ms, which is the removed record rewrite. - 1.1 GB: rebuild 1039.2 -> 1138.3 ms; build 833.2 -> 800.7 ms, verify 22.4 -> 155.5 ms. The device flush that follows the 1.1 GB load is not removed by this commit, only moved: it was paid by the directory sync inside the build phase and is now paid by the first record write of the verify phase. This shape gains nothing here. Tests: no new test (an fsync is not observable through the public API). Pass: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). No test deleted. Production LOC: 188,160 -> 188,152 (delta -8) core 40,589 -> 40,589 (0) engine 89,971 -> 89,963 (-8) layerfs-persistence -8 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent d463879336 and the staged tree Co-Authored-By: Claude Fable 5.2 --- docs/architecture/engine/store/sqlite.md | 16 ++++++++++++++++ .../src/backend/sqlite/compact/run.rs | 15 ++++----------- 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/architecture/engine/store/sqlite.md b/docs/architecture/engine/store/sqlite.md index b689a298a7..fd68037526 100644 --- a/docs/architecture/engine/store/sqlite.md +++ b/docs/architecture/engine/store/sqlite.md @@ -160,6 +160,22 @@ quarantines the session and keeps the lock | Free-space guard | A pack batch must leave 64 MiB after counting its bytes twice (WAL, then file); metadata-only batches are not guarded | [`headroom.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/lifecycle/headroom.rs) | | `reclaim_space(pages)` | `1..=RECLAMATION_PAGE_LIMIT` (512) pages by `incremental_vacuum`, stepped in one short write until the free list is empty or the page budget is spent (SQLite 3.40 removes one page per statement, not `N`); only Stores created with `auto_vacuum=2` | [`reclamation.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/lifecycle/reclamation.rs) | +## Compaction rebuild + +`SqliteHandles::compact` marks under the writer lane, seals the file, builds +`.compact` beside it from the retained rows, checks it through an +`immutable=1` reader and renames it into place. The build's cost is the copy +of the retained packs, so its steps are shaped around that copy. + +| Step | Rule | Source | +| --- | --- | --- | +| Op record | `.compact-op` is rewritten by rename at admission and at the build, verify, install and cleanup phases. The admission record stands through the mark: a later call judges a leftover by the run, generation and owner it names, and `store.status` reads its phase as `mark` | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`run.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs) | +| Fresh file | The sealed Store's tables, then the rows, then its indexes and triggers; `journal_mode=OFF` and `synchronous=OFF` while loading, then WAL. The sealed file is attached `immutable=1` with a 256 MiB `mmap_size` | [`build.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs) | +| File size | macOS sizes the fresh file for the retained pack bytes before the load (one advisory `F_PREALLOCATE`, one `set_len`), because on APFS a page write that also moves the end of the file costs about four times one inside a sized file. The closed file is cut to the pages its header counts. Linux leaves the file to grow | [`space.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs) | +| Pack copy | The table that holds the pack payload (`pack`, or `pack_unit` in the group-row layouts) is copied one window per statement: `WINDOW_BYTES` (64 MiB) of retained payload, ended early by a dead run longer than `GAP_BYTES` (8 MiB). Every other table is one statement | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs), [`build.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs) | +| Read-ahead | Before each window the sealed file's matching range is asked for once, through a second read-only descriptor: `F_RDADVISE` on macOS, `POSIX_FADV_WILLNEED` in 128 KiB steps on Linux. Without it a source that is not in memory is read one pack per device round trip. The range assumes packs lie in the file in id order; a hint only, so another layout loses the benefit and nothing else | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs) | +| Synchronization | The op record is synchronized at each write, the fresh file before the rename and the directory after it | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`install.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs) | + ## Heads | Function | Returns | Source | diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs index 2482d4a991..3aaea6a7ba 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs @@ -278,14 +278,13 @@ impl SqliteHandles { // §7 step 2: a stop here leaves the open session, the writer lane, // the committed intent, the visited scratch and the op record; the - // live Store is at `G`. + // live Store is at `G`. The admission record already names this + // run, its generation and its owner, which is all a later call + // judges a leftover by, and `store.status` reads its phase as + // `mark`; the mark's tuple and digest go out with the next write. op.tuple = tuple.clone(); op.live_heads_digest = mark.live_heads_digest(); op.phase = "mark".to_owned(); - if let Err(error) = op.write(&store) { - abort_run(&history, &store, &op_id); - return Err(error); - } // §7 step 1: with the lane held, no writer can append while the // intent, the op record and the WAL are settled together. if let Err(error) = session.checkpoint_holding() { @@ -405,12 +404,6 @@ impl SqliteHandles { peak_temp: built.peak_temp_bytes(), total_stored: total, }); - // §7 step 7: every artifact and the op record are durable before - // the verification reads them back. - if let Err(error) = sync_directory(&store) { - abort_before_install(&store, &op_id, true); - return Err(error); - } stop_at(request, CompactFailpoint::AfterBuild, &op, &store)?; stop_at(request, CompactFailpoint::AfterWalSwitch, &op, &store)?; metrics.end_phase(); From 8e7acb30bb4c02c0b15d693e23d65f19de588973 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:38:34 +0800 Subject: [PATCH 07/31] perf(gc): a Disposable Store's compaction asks for no fsync (layerfs-project#41) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NEEDS THE OWNER'S DECISION: spec §7 step 7 reads "chmod 0600; sync_all; rename; fsync the directory" without naming a profile. This commit keeps that sequence for a Store whose profile synchronizes (Normal, Durable) and skips the synchronization for a Disposable one. It is one flag; reverting this commit restores the unconditional calls. A SQLite rebuild made 13 fsync calls of its own (16 before the previous commit): two per op record write, the fresh file before the rename, the directory after it and after the settle. Each is F_FULLFSYNC on macOS. On the 28 MB shape they were about two thirds of the whole rebuild, and three of the `none` path's work was the same calls. A Disposable Store (WAL, synchronous=OFF) survives a process crash and claims nothing after a power loss: its own commits and checkpoints are never synchronized, so a compaction that synchronizes its record and its install protects a Store that the same power loss may already have torn. The run now takes `durable = profile.synchronous != 0` once: - op.rs `RunOp.durable`: a record write is still temp file plus rename (what a process crash needs); the file and directory fsync run only when durable. A record read back from disk is written synchronized, and `mark_returned` takes the run's flag. - install.rs `prepare_target(target, durable)`: chmod always, sync_all only when durable. `settle` synchronizes the directory only when durable. - run.rs: the directory fsync after the rename and at the end of the `none` path run only when durable. The abort paths are unchanged. A Normal Store's compaction is unchanged by this commit: the same 13 calls. The product opens Stores Normal since 2026-11-17 while GC is measured under Disposable (#42 item 42), so from here the Disposable bench numbers exclude synchronization the product profile still pays. Measured on the same fixtures opened Normal (rebuild_probe, a local diagnostic; one sample, load 3.9): 28 MB rebuild 142 ms against 46 ms Disposable; 1.1 GB rebuild 1100 ms against 963 ms. Measured, release, macOS APFS, gc_bench --runs 1 (Disposable), one sample each, 1-minute load 4.9: - small (28 MB): rebuild 95.1 -> 32.7 ms (build 31.8 -> 20.8, verify 10.5 -> 1.0, install 34.2 -> 2.1 ms); `none` 24 -> 9 ms. - 1.1 GB: rebuild 1138.3 -> 982.5 ms (build 800.7 -> 782.2, verify 155.5 -> 11.3, install 78.7 -> 94.1 ms); `none` 115 -> 93 ms. What is left of install is the rename, which frees the old file's 1.26 GB. Tests: no new test (an fsync is not observable through the public API). Pass: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). No test deleted. Production LOC: 188,152 -> 188,168 (delta +16) core 40,589 -> 40,589 (0) engine 89,963 -> 89,979 (+16) layerfs-persistence +16 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent 7f5bb75b34 and the staged tree Co-Authored-By: Claude Fable 5.2 --- docs/architecture/engine/store/sqlite.md | 2 +- .../src/backend/sqlite/compact/install.rs | 25 ++++++++++++------- .../src/backend/sqlite/compact/op.rs | 22 ++++++++++++---- .../src/backend/sqlite/compact/run.rs | 21 +++++++++++----- 4 files changed, 49 insertions(+), 21 deletions(-) diff --git a/docs/architecture/engine/store/sqlite.md b/docs/architecture/engine/store/sqlite.md index fd68037526..25929436f9 100644 --- a/docs/architecture/engine/store/sqlite.md +++ b/docs/architecture/engine/store/sqlite.md @@ -174,7 +174,7 @@ of the retained packs, so its steps are shaped around that copy. | File size | macOS sizes the fresh file for the retained pack bytes before the load (one advisory `F_PREALLOCATE`, one `set_len`), because on APFS a page write that also moves the end of the file costs about four times one inside a sized file. The closed file is cut to the pages its header counts. Linux leaves the file to grow | [`space.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs) | | Pack copy | The table that holds the pack payload (`pack`, or `pack_unit` in the group-row layouts) is copied one window per statement: `WINDOW_BYTES` (64 MiB) of retained payload, ended early by a dead run longer than `GAP_BYTES` (8 MiB). Every other table is one statement | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs), [`build.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs) | | Read-ahead | Before each window the sealed file's matching range is asked for once, through a second read-only descriptor: `F_RDADVISE` on macOS, `POSIX_FADV_WILLNEED` in 128 KiB steps on Linux. Without it a source that is not in memory is read one pack per device round trip. The range assumes packs lie in the file in id order; a hint only, so another layout loses the benefit and nothing else | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs) | -| Synchronization | The op record is synchronized at each write, the fresh file before the rename and the directory after it | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`install.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs) | +| Synchronization | Follows the Store's profile. Normal: the op record (file and directory) at each write, the fresh file before the rename and the directory after it. Disposable (`synchronous=OFF`) claims nothing after a power loss, so its compaction asks for no fsync: the record is still written by rename and the install is still one rename, which is what survives a process crash | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`install.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs), [`run.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs) | ## Heads diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs index c7d0c22d24..036f664f0c 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs @@ -116,20 +116,25 @@ pub(crate) fn install_fence(facts: &SealedFacts, tuple: &MarkTuple) -> HistoryRe Ok(()) } -/// §7 step 7: `chmod 0600` and `sync_all` before the rename. -pub(crate) fn prepare_target(target: &Path) -> HistoryResult<()> { +/// §7 step 7: `chmod 0600` and `sync_all` before the rename. A Store whose +/// profile synchronizes nothing (`durable` false) skips the `sync_all`: the +/// rename then installs a file that is complete in the page cache, which is +/// all that profile promises of the Store itself. +pub(crate) fn prepare_target(target: &Path, durable: bool) -> HistoryResult<()> { #[cfg(unix)] { use std::os::unix::fs::PermissionsExt; std::fs::set_permissions(target, std::fs::Permissions::from_mode(0o600)) .map_err(io_error)?; } - let file = std::fs::OpenOptions::new() - .read(true) - .write(true) - .open(target) - .map_err(io_error)?; - file.sync_all().map_err(io_error)?; + if durable { + let file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(target) + .map_err(io_error)?; + file.sync_all().map_err(io_error)?; + } Ok(()) } @@ -212,7 +217,9 @@ pub(crate) fn settle( |name| name.to_string_lossy().into_owned(), )); } - let _ = sync_directory(store); + if op.durable { + let _ = sync_directory(store); + } if !residue.artifacts.is_empty() && residue.reason.is_none() { residue.reason = Some("the artifact set could not be removed".to_owned()); } diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs index 79a65b204a..57e46660f5 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs @@ -121,6 +121,12 @@ pub(crate) struct RunOp { /// The §11 totals once the rebuild's target is built; `None` before any /// report value exists and in a record written before they were kept. pub(crate) installed: Option, + /// Whether this run synchronizes what it writes beside the Store. It is + /// the Store's own profile, not part of the record: a Disposable Store + /// (`synchronous=OFF`) survives a process crash and claims nothing after + /// a power loss, so its compaction asks for no fsync either. A record + /// read back from disk is written synchronized. + pub(crate) durable: bool, } impl RunOp { @@ -165,12 +171,16 @@ live_heads_digest={}\nreturned={}\n", let path = op_path(store); let temporary = op_temporary_path(store); std::fs::write(&temporary, self.encode()).map_err(io_error)?; - if let Ok(file) = std::fs::File::open(&temporary) { - let _ = file.sync_all(); + if self.durable { + if let Ok(file) = std::fs::File::open(&temporary) { + let _ = file.sync_all(); + } } std::fs::rename(&temporary, &path).map_err(io_error)?; - if let Some(parent) = path.parent() { - let _ = std::fs::File::open(parent).and_then(|directory| directory.sync_all()); + if self.durable { + if let Some(parent) = path.parent() { + let _ = std::fs::File::open(parent).and_then(|directory| directory.sync_all()); + } } Ok(()) } @@ -238,6 +248,7 @@ live_heads_digest={}\nreturned={}\n", live_heads_digest, returned, installed: InstalledReport::read(&values), + durable: true, }) } @@ -283,7 +294,7 @@ pub(crate) fn remove_op_record(store: &Path, op_id: &[u8; 16]) -> HistoryResult< /// so the next `store.compact` judges a leftover even though the owner pid is /// still alive (§9). Only the run's own record is marked: a record another run /// wrote in the settle window is never touched. -pub(crate) fn mark_returned(store: &Path, op_id: &[u8; 16]) -> HistoryResult<()> { +pub(crate) fn mark_returned(store: &Path, op_id: &[u8; 16], durable: bool) -> HistoryResult<()> { let Some(mut op) = RunOp::read(store) else { return Ok(()); }; @@ -291,6 +302,7 @@ pub(crate) fn mark_returned(store: &Path, op_id: &[u8; 16]) -> HistoryResult<()> return Ok(()); } op.returned = true; + op.durable = durable; op.write(store) } diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs index 3aaea6a7ba..b604913573 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs @@ -61,6 +61,7 @@ impl SqliteHandles { /// op record are kept for the next `store.compact` to judge (§9). pub fn compact(self, request: &CompactRequest) -> HistoryResult { let store = self.storage.session.path.clone(); + let durable = self.storage.session.profile.synchronous != 0; let mut metrics = RunMetrics::start(); let result = self.compact_measured(request, &mut metrics); // §9: the run is replying, so its own record (when it leaves one) is @@ -68,7 +69,7 @@ impl SqliteHandles { // `store.compact` judges the leftover even while this process is still // alive; the window before this point stays protected by the owner // liveness guard. - let _ = mark_returned(&store, &request.op_id); + let _ = mark_returned(&store, &request.op_id, durable); metrics.finish(&result); result } @@ -98,6 +99,11 @@ impl SqliteHandles { let layout = session.layout; let acquisition_schema = session.acquisition; let synchronous = profile.synchronous; + // The run synchronizes what the Store's profile does: a + // Disposable Store (`synchronous=OFF`) claims nothing after a + // power loss, so its compaction asks for no fsync of the op + // record, the fresh file or the directory. + let durable = synchronous != 0; // §5.1/§7 admission: a client that holds the Store keeps its // sidecars, and the seal cannot remove one its owner keeps: it @@ -158,8 +164,8 @@ impl SqliteHandles { // The op record is written before the mark so a crash inside the // mark transaction (which creates the §6.3 visited scratch before // it decides) leaves a *recorded* run for the next call to adopt, - // never an unrecorded staged name (§9). The mark phase overwrites - // it once the mark's tuple and digest are known. + // never an unrecorded staged name (§9). It stands through the + // mark; the build's write carries the mark's tuple and digest. let mut op = RunOp { op_id, generation: live.generation, @@ -174,6 +180,7 @@ impl SqliteHandles { live_heads_digest: [0u8; 32], returned: false, installed: None, + durable, }; op.write(&store)?; @@ -258,7 +265,9 @@ impl SqliteHandles { abort_before_install(&store, &op_id, false); return Err(error); } - let _ = sync_directory(&store); + if durable { + let _ = sync_directory(&store); + } drop(lane); return Ok(receipt( CompactPath::None, @@ -443,7 +452,7 @@ impl SqliteHandles { abort_before_install(&store, &op_id, true); return Err(error); } - if let Err(error) = prepare_target(&built.target) { + if let Err(error) = prepare_target(&built.target, durable) { abort_before_install(&store, &op_id, true); return Err(error); } @@ -459,7 +468,7 @@ impl SqliteHandles { // §7 step 7: the rename happened; a crash before the directory // fsync leaves an installed-but-uncertain `G + 1`. stop_at(request, CompactFailpoint::DuringRename, &op, &store)?; - if let Err(_error) = sync_directory(&facts.path) { + if durable && sync_directory(&facts.path).is_err() { op.phase = "install".to_owned(); let _ = op.write(&store); return Err(HistoryError::UnknownOutcome); From c1c2f98198798eeeb6d0f8e61982234599c592c7 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:44:11 +0800 Subject: [PATCH 08/31] perf(gc): mark reads ranges for a whole wave in a few calls (layerfs-project#41) After the batched head read, a pack that needed more than its head still took scoped acquisitions of its own: a statement and a BLOB handle per pack and per round on SQLite, and on PostgreSQL, where mark's time follows round trips, one or more round trips per pack. - backend/seam.rs: Mechanics::pack_ranges returns selected byte ranges of several packs in one call. Its default is one scoped acquisition per pack, which is what PostgreSQL uses until it overrides it. - backend/sqlite: one BLOB handle moved from row to row; a pack's ranges ascend, so its overflow chain is followed once. The GroupRows layouts keep the scoped read. - history/objects/edges.rs: rounds are per wave batch, not per pack. Each round resolves every pack of the batch against the bytes held and reads what all of them still need in bounded calls (1,024 ranges, 512 KiB). The per-pack range plan and acquisition are gone; the kept-waste directory read uses the same call. Measured (gc_profile, release, warm cache, macOS, load about 3, one run): - 28 MB small Store (737 objects in 158 packs of up to six groups): mark 6.8 -> 5.4 ms; statements 172 -> 58; BLOB handles 150 -> 37; reads 996 -> 768. - 1.13 GB Store (8,913 objects in 8,816 single-group packs): unchanged, 37.0 ms, 298 statements, 229 handles. Tests: kept_objects, kept_node, compact_metrics, sqlite_compact, sqlite_compact_crash, space_reclamation, sqlite_group_rows, sqlite_group_rows_indexed, postgresql_gc_mark, postgresql_compact, postgresql_compact_fixtures pass. sqlite_compact_matrix failed once and passed on the next run with no source change: its AC-49 arm races a Commit thread against the rebuild and unwraps the Save inside that thread (tests/sqlite_compact_matrix.rs:149). Reported, not changed here. No test added or deleted. Production LOC: 188037 -> 188192 (delta +155) Scope and method: tools/production_loc.py at this commit, first parent against the committed tree; core 40589 -> 40589, engine 89848 -> 90003, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd1917812). Co-Authored-By: Claude Fable 5.2 --- .../layerfs-persistence/src/backend/seam.rs | 56 ++- .../src/backend/sqlite/mechanics.rs | 12 + .../src/backend/sqlite/objects/read.rs | 57 ++++ .../src/history/objects/edges.rs | 321 ++++++++++-------- 4 files changed, 308 insertions(+), 138 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/seam.rs b/engine/crates/layerfs-persistence/src/backend/seam.rs index 272fdc567e..9696163fb2 100644 --- a/engine/crates/layerfs-persistence/src/backend/seam.rs +++ b/engine/crates/layerfs-persistence/src/backend/seam.rs @@ -3,8 +3,11 @@ use super::records::{BackendError, Param, Record}; use layerfs_content::ObjectId; use layerfs_storage::{ + location::PackInfo, location::{InodeValueRow, LocatedObject, SignatureRow}, - port::{PersistedPack, PersistenceError}, + port::{ + AcquiredPackRead, PackRange, PackReadChoice, PackReadPlan, PersistedPack, PersistenceError, + }, StoragePolicy, }; use std::sync::Arc; @@ -105,6 +108,18 @@ pub(crate) trait Mechanics { } Ok(()) } + /// The bytes of selected ranges of several packs in one call: for each + /// `(pack, ranges)` of `requests`, in order, the bytes of each range, in + /// order. A pack's ranges are sorted and disjoint. The bytes are not + /// authenticated; the consumer does that. An engine answers from one + /// cursor or one statement; this default acquires each pack on its own. + fn pack_ranges( + &self, + requests: &[(i64, Vec)], + out: &mut Vec>>, + ) -> Result<(), PersistenceError> { + ranges_by_acquisition(self, requests, out) + } /// One pack through the caller's bounded range plan: only the control /// prefix and the selected complete groups are read, never the body behind /// them. The storage port owns the plan and the returned units. @@ -118,3 +133,42 @@ pub(crate) trait Mechanics { }) } } + +/// A plan that selects ranges chosen before the acquisition starts. +struct FixedRanges(Option>); + +impl PackReadPlan for FixedRanges { + fn select( + &mut self, + _info: PackInfo, + _prefix: &[u8], + ) -> Result { + self.0 + .take() + .map(PackReadChoice::Ranges) + .ok_or(PersistenceError::Malformed) + } +} + +/// [`Mechanics::pack_ranges`] as one scoped acquisition per pack. +pub(crate) fn ranges_by_acquisition( + mechanics: &M, + requests: &[(i64, Vec)], + out: &mut Vec>>, +) -> Result<(), PersistenceError> { + for (id, ranges) in requests { + let mut plan = FixedRanges(Some(ranges.clone())); + let AcquiredPackRead::Units(units) = mechanics.read_scoped_pack(*id, &mut plan)? else { + return Err(PersistenceError::Malformed); + }; + out.push( + units + .into_parts() + .2 + .into_iter() + .map(|(_, bytes)| bytes) + .collect(), + ); + } + Ok(()) +} diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs index 64ddc3d5c6..97b20f7d5b 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/mechanics.rs @@ -69,6 +69,18 @@ impl Mechanics for Transaction<'_> { ) -> Result<(), PersistenceError> { super::objects::read::prefixes(self, ids, limit, out) } + fn pack_ranges( + &self, + requests: &[(i64, Vec)], + out: &mut Vec>>, + ) -> Result<(), PersistenceError> { + if self.layout().uses_units() { + // Group rows are separate BLOBs: the scoped read maps offsets to + // them, one pack at a time. + return crate::backend::seam::ranges_by_acquisition(self, requests, out); + } + super::objects::read::ranges(self, requests, out) + } fn read_scoped_pack( &self, id: i64, diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs index 053fb6937b..d7a433a041 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/objects/read.rs @@ -201,3 +201,60 @@ pub(crate) fn prefixes( })(); closed(tx, blob, read) } + +/// Selected ranges of several packs through one BLOB handle moved from row +/// to row. A pack's ranges ascend, so its overflow chain is followed once. +pub(crate) fn ranges( + tx: &Transaction<'_>, + requests: &[(i64, Vec)], + out: &mut Vec>>, +) -> Result<(), PersistenceError> { + let Some((first, _)) = requests.first() else { + return Ok(()); + }; + if requests.iter().any(|(id, _)| *id <= 0) { + return Err(PersistenceError::Malformed); + } + tx.work.borrow_mut().blob_open_calls += 1; + let mut blob = tx + .connection + .blob_open("main", "pack", "body", *first, true) + .map_err(rows::error)?; + let read = (|| { + for (index, (id, ranges)) in requests.iter().enumerate() { + if index > 0 { + tx.work.borrow_mut().blob_reopen_calls += 1; + blob.reopen(*id).map_err(rows::error)?; + } + let mut bodies = Vec::with_capacity(ranges.len()); + let mut end = 0; + for range in ranges { + let stop = range + .offset + .checked_add(range.length) + .filter(|stop| range.length > 0 && range.offset >= end && *stop <= blob.len()) + .ok_or(PersistenceError::Malformed)?; + end = stop; + let mut body = vec![0; range.length]; + { + let mut work = tx.work.borrow_mut(); + work.blob_read_calls += 1; + work.blob_requested_bytes += body.len() as u64; + } + let start = Instant::now(); + let read = blob + .read_at_exact(&mut body, range.offset) + .map_err(rows::error); + let mut work = tx.work.borrow_mut(); + work.blob_read_ns += start.elapsed().as_nanos() as u64; + read?; + work.blob_read_bytes += body.len() as u64; + drop(work); + bodies.push(body); + } + out.push(bodies); + } + Ok(()) + })(); + closed(tx, blob, read) +} diff --git a/engine/crates/layerfs-persistence/src/history/objects/edges.rs b/engine/crates/layerfs-persistence/src/history/objects/edges.rs index b3fd1ea5fa..bc3cee809d 100644 --- a/engine/crates/layerfs-persistence/src/history/objects/edges.rs +++ b/engine/crates/layerfs-persistence/src/history/objects/edges.rs @@ -7,10 +7,9 @@ //! mark costs a few hundred bytes per pack instead of the pack. Compressed //! groups and the `Ordinary` lane answer `None` and keep the decoding reader. //! -//! The first read of a wave is one call for the heads of many packs. A pack -//! whose directory and record headers sit in its head, the common shape of a -//! pack of large records, costs no statement of its own; only a pack that -//! needs more is acquired on its own. +//! Reads are per wave, not per pack: one call for the heads of many packs, +//! then at most a few calls for whatever ranges those packs still need. A +//! pack costs no statement and no round trip of its own. use super::{storage, ClosureSource}; use layerfs_content::ObjectId; use layerfs_history::error::{HistoryError, HistoryResult}; @@ -19,12 +18,9 @@ use layerfs_storage::{ has_sparse_edges, sparse_directory, sparse_edge, RecordDirectory, Sparse, }, encoding::delta::read::ChainEdge, - location::{ObjectLocation, PackInfo}, + location::ObjectLocation, pack::layout::{self, GroupView, PackHeader}, - port::{ - AcquiredPackRead, PackRange, PackReadChoice, PackReadPlan, PersistenceError, - PACK_READ_PREFIX_BYTES, - }, + port::{PackRange, PACK_READ_PREFIX_BYTES}, source::Source, StorageError, }; @@ -39,9 +35,9 @@ const ROUNDS: usize = 4; const HEAD_BYTES: usize = 1024; /// Packs whose heads one call reads. const HEAD_PACKS: usize = 256; -/// Ranges one acquisition selects; far below the port's own limits. +/// Ranges one call asks for, over all its packs. const BATCH_RANGES: usize = 1024; -/// Bytes one acquisition selects. +/// Bytes one call asks for, over all its packs. const BATCH_BYTES: usize = 512 * 1024; /// The pack bytes held so far, by pack offset. @@ -159,85 +155,65 @@ fn resolve( Ok(merged(needs)) } -/// One acquisition's ranges, chosen before it starts. -struct Ranges { - pack_id: i64, - ranges: Option>, -} - -impl PackReadPlan for Ranges { - fn select( - &mut self, - info: PackInfo, - _prefix: &[u8], - ) -> Result { - if info.pack_id != self.pack_id { - return Err(PersistenceError::Malformed); - } - self.ranges - .take() - .map(PackReadChoice::Ranges) - .ok_or(PersistenceError::Malformed) - } -} +/// One request of a round: the pack, its declared length, and the ranges. +type Request = (i64, usize, Vec); -/// One acquisition of `ranges`, counted in the mark's bytes. -fn acquire( +/// Reads every request's ranges in bounded calls and returns each request's +/// bytes, range by range. +fn read_ranges( source: &ClosureSource<'_>, - pack_id: i64, - ranges: Vec, -) -> HistoryResult)>> { - let mut plan = Ranges { - pack_id, - ranges: Some(ranges), - }; - let read = source - .mechanics - .read_scoped_pack(pack_id, &mut plan) - .map_err(StorageError::from) - .map_err(storage)?; - let AcquiredPackRead::Units(units) = read else { - return Err(HistoryError::Integrity("mark edge acquisition")); - }; - source.note_scoped_body( - pack_id, - u64::try_from(units.acquired_bytes()).unwrap_or(u64::MAX), - u64::try_from(units.info().length).unwrap_or(u64::MAX), - ); - Ok(units.into_parts().2) -} - -/// Acquires `needs`, sorted and disjoint ranges of pack `pack_id`, in bounded -/// batches, and adds them to `held`. -fn acquire_all( - source: &ClosureSource<'_>, - pack_id: i64, - needs: Vec, - held: &mut Pieces, -) -> HistoryResult<()> { - let mut acquired = Vec::new(); - let mut flush = |batch: &mut Vec| -> HistoryResult<()> { - if batch.is_empty() { - return Ok(()); - } - acquired.extend(acquire(source, pack_id, std::mem::take(batch))?); - Ok(()) - }; - let mut batch = Vec::new(); - let mut bytes = 0; - for range in needs { - if batch.len() == BATCH_RANGES || bytes + range.length > BATCH_BYTES { - flush(&mut batch)?; - bytes = 0; + requests: &[Request], +) -> HistoryResult>>> { + let mut bodies: Vec>> = requests.iter().map(|_| Vec::new()).collect(); + let mut call: Vec<(i64, Vec)> = Vec::new(); + let mut owners: Vec = Vec::new(); + let mut flush = + |call: &mut Vec<(i64, Vec)>, owners: &mut Vec| -> HistoryResult<()> { + if call.is_empty() { + return Ok(()); + } + let mut read = Vec::with_capacity(call.len()); + source + .mechanics + .pack_ranges(call, &mut read) + .map_err(StorageError::from) + .map_err(storage)?; + if read.len() != call.len() { + return Err(HistoryError::Integrity("mark range read")); + } + for ((owner, (_, asked)), got) in owners.drain(..).zip(call.drain(..)).zip(read) { + let exact = got.len() == asked.len() + && asked + .iter() + .zip(&got) + .all(|(range, bytes)| bytes.len() == range.length); + if !exact { + return Err(HistoryError::Integrity("mark range read")); + } + bodies[owner].extend(got); + } + Ok(()) + }; + let (mut count, mut bytes) = (0, 0); + for (index, (pack_id, _, ranges)) in requests.iter().enumerate() { + for range in ranges { + if count > 0 && (count == BATCH_RANGES || bytes + range.length > BATCH_BYTES) { + flush(&mut call, &mut owners)?; + (count, bytes) = (0, 0); + } + match call.last_mut() { + Some((_, list)) if owners.last() == Some(&index) => list.push(*range), + _ => { + call.push((*pack_id, vec![*range])); + owners.push(index); + } + } + count += 1; + bytes += range.length; } - bytes += range.length; - batch.push(range); } - flush(&mut batch)?; - for (range, bytes) in acquired { - held.add(range.offset, bytes); - } - Ok(()) + flush(&mut call, &mut owners)?; + Ok(bodies) } /// The record directories of `groups`, uncompressed groups of pack `pack_id` @@ -264,60 +240,122 @@ pub(super) fn group_directories( if needs.is_empty() { return Ok(directories); } - acquire_all(source, pack_id, merged(needs), &mut held)?; + let request = [(pack_id, usize::MAX, merged(needs))]; + let mut read = 0u64; + for (range, bytes) in request[0] + .2 + .iter() + .zip(read_ranges(source, &request)?.remove(0)) + { + read += bytes.len() as u64; + held.add(range.offset, bytes); + } + source.note_scoped_body(pack_id, read, u64::MAX); } Err(HistoryError::Integrity("mark group directory")) } -/// The edges of `items`, all records of pack `pack_id`, given the pack's -/// declared length and head: `None` where the record's group is encoded and -/// must take the decoding reader. -fn pack_edges( - source: &ClosureSource<'_>, +/// One pack's records in a wave, and what the wave has read of the pack. +struct PackWork<'a> { pack_id: i64, - items: &[ObjectLocation], length: usize, - head: Vec, -) -> HistoryResult>> { - let control = layout::parse_control_header(&head, length).map_err(storage)?; - let head = if head.len() < control.body_offset { - source - .mechanics - .pack_prefix(pack_id, PACK_READ_PREFIX_BYTES) - .map_err(StorageError::from) - .map_err(storage)? - .filter(|(wide, _)| *wide == length) - .ok_or(HistoryError::Integrity("mark pack directory"))? - .1 - } else { - head - }; - let header = layout::parse_directory_header(&head, length).map_err(storage)?; - let views = - layout::directory_group_views(&head[..header.body_offset], header).map_err(storage)?; - // The head's bytes past the directory are body bytes this mark read. - source.note_scoped_body( - pack_id, - u64::try_from(head.len() - header.body_offset).unwrap_or(u64::MAX), - u64::try_from(length).unwrap_or(u64::MAX), - ); - let mut held = Pieces::default(); - held.add(0, head); - let mut answers = vec![Answer::Pending; items.len()]; - for _ in 0..=ROUNDS { - let needs = resolve(header, &views, items, &held, &mut answers).map_err(storage)?; - if needs.is_empty() { - source.keep_layout(pack_id, views); - return answers - .into_iter() - .map(|answer| match answer { - Answer::Edge(edge) => Ok(Some(edge)), - Answer::Encoded => Ok(None), - Answer::Pending => Err(HistoryError::Integrity("mark edge read")), - }) - .collect(); + items: &'a [ObjectLocation], + header: PackHeader, + views: Vec, + held: Pieces, + answers: Vec, + settled: bool, +} + +impl<'a> PackWork<'a> { + /// The pack's layout from its declared length and head. A directory wider + /// than the head is read again, whole. + fn open( + source: &ClosureSource<'_>, + pack_id: i64, + items: &'a [ObjectLocation], + length: usize, + head: Vec, + ) -> HistoryResult { + let control = layout::parse_control_header(&head, length).map_err(storage)?; + let head = if head.len() < control.body_offset { + source + .mechanics + .pack_prefix(pack_id, PACK_READ_PREFIX_BYTES) + .map_err(StorageError::from) + .map_err(storage)? + .filter(|(wide, _)| *wide == length) + .ok_or(HistoryError::Integrity("mark pack directory"))? + .1 + } else { + head + }; + let header = layout::parse_directory_header(&head, length).map_err(storage)?; + let views = + layout::directory_group_views(&head[..header.body_offset], header).map_err(storage)?; + // The head's bytes past the directory are body bytes this mark read. + source.note_scoped_body( + pack_id, + u64::try_from(head.len() - header.body_offset).unwrap_or(u64::MAX), + u64::try_from(length).unwrap_or(u64::MAX), + ); + let mut held = Pieces::default(); + held.add(0, head); + Ok(Self { + pack_id, + length, + items, + header, + views, + held, + answers: vec![Answer::Pending; items.len()], + settled: false, + }) + } +} + +/// Answers every record of `work`: each round resolves what the held bytes +/// can and reads, in one bounded set of calls, what all the packs still need. +fn settle(source: &ClosureSource<'_>, work: &mut [PackWork<'_>]) -> HistoryResult<()> { + for round in 0..=ROUNDS { + let mut requests: Vec = Vec::new(); + let mut owners = Vec::new(); + for (index, pack) in work.iter_mut().enumerate() { + if pack.settled { + continue; + } + let needs = resolve( + pack.header, + &pack.views, + pack.items, + &pack.held, + &mut pack.answers, + ) + .map_err(storage)?; + if needs.is_empty() { + pack.settled = true; + } else { + requests.push((pack.pack_id, pack.length, needs)); + owners.push(index); + } + } + if requests.is_empty() { + return Ok(()); + } + if round == ROUNDS { + break; + } + let bodies = read_ranges(source, &requests)?; + for ((owner, (pack_id, length, ranges)), read) in + owners.into_iter().zip(requests).zip(bodies) + { + let mut total = 0u64; + for (range, bytes) in ranges.iter().zip(read) { + total += bytes.len() as u64; + work[owner].held.add(range.offset, bytes); + } + source.note_scoped_body(pack_id, total, u64::try_from(length).unwrap_or(u64::MAX)); } - acquire_all(source, pack_id, needs, &mut held)?; } Err(HistoryError::Integrity("mark edge read")) } @@ -356,13 +394,22 @@ pub(super) fn wave_edges( if heads.len() != batch.len() { return Err(HistoryError::Integrity("mark pack heads")); } + let mut work = Vec::with_capacity(batch.len()); for ((pack_id, items), (length, head)) in batch.iter().zip(heads) { - let answers = pack_edges(source, *pack_id, items, length, head)?; - for (item, edge) in items.iter().zip(answers) { - if let Some(edge) = edge { - edges.insert(item.object_id, edge); + work.push(PackWork::open(source, *pack_id, items, length, head)?); + } + settle(source, &mut work)?; + for pack in work { + for (item, answer) in pack.items.iter().zip(pack.answers) { + match answer { + Answer::Edge(edge) => { + edges.insert(item.object_id, edge); + } + Answer::Encoded => {} + Answer::Pending => return Err(HistoryError::Integrity("mark edge read")), } } + source.keep_layout(pack.pack_id, pack.views); } } Ok(edges) From 4753f7ae6d80b92238598cd70f9eb2de550f385a Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:44:50 +0800 Subject: [PATCH 09/31] perf(gc): one statement where compaction repeated a scan or an open (layerfs-project#41) Counts, not time: none of these is visible in a one-sample wall time on the bench shapes. They remove statements, table passes and connection opens that produced nothing the run did not already have. - compact_visit.rs: the mark ran two inserts per reached object, one into `visited` and one `INSERT OR IGNORE` into `live_pack`. The retained packs are the visited rows' packs, so `finish` now derives them with one ordered `INSERT ... SELECT DISTINCT`. Statements per mark: 2N -> N + 1 (N = reached objects; about 9,000 on the 1.1 GB shape, so about 18,000 -> 9,000). - verify.rs `verify`: four count statements and one totals statement became one statement. Passes over the fresh file: `pack` 3 -> 1, `object_location` 2 -> 1, `store_totals` 5 reads -> 1 (the tuple already reads the row). Own statements 8 -> 4. Every check, its error and the order of the checks around the `DuringVerify` stop are unchanged. - verify.rs `sealed_state`: the sealed file's tuple and live-heads digest are read through one `immutable=1` connection instead of two (`immutable_tuple` and `immutable_digest` are removed). Connections opened per rebuild: 7 -> 6 besides the session. - build.rs: the totals rewrite reads `pack` once for `stored` and `metadata` instead of twice. Measured, release, macOS APFS, gc_bench --runs 1 (Disposable), one sample each, 1-minute load 3.7 and 3.4: - small (28 MB): rebuild 32.7 -> 34.6 ms (mark 7.3 -> 7.6, build 20.8 -> 22.2, verify 0.96 -> 0.94, install 2.06 -> 2.03 ms); `none` 9 -> 9 ms. - 1.1 GB: rebuild 982.5 -> 927.5 ms (mark 92.3 -> 88.5, build 782.2 -> 730.4, verify 11.3 -> 10.2, install 94.1 -> 96.2 ms); `none` 93 -> 92 ms. The build difference is sample noise in the pack copy, not this change. Tests: pass, unchanged: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). The rebuild tests cover `live_pack`: verification refuses a fresh file whose pack count is not the mark's. No test deleted. Production LOC: 188,168 -> 188,162 (delta -6) core 40,589 -> 40,589 (0) engine 89,979 -> 89,973 (-6) layerfs-persistence -6 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent 8e7acb30bb and the staged tree Co-Authored-By: Claude Fable 5.2 --- docs/architecture/engine/store/sqlite.md | 2 + .../src/backend/sqlite/compact/build.rs | 4 +- .../src/backend/sqlite/compact/run.rs | 24 ++--- .../src/backend/sqlite/compact/verify.rs | 98 ++++++++++--------- .../src/backend/sqlite/compact_visit.rs | 17 ++-- 5 files changed, 76 insertions(+), 69 deletions(-) diff --git a/docs/architecture/engine/store/sqlite.md b/docs/architecture/engine/store/sqlite.md index 25929436f9..58b52d4214 100644 --- a/docs/architecture/engine/store/sqlite.md +++ b/docs/architecture/engine/store/sqlite.md @@ -174,6 +174,8 @@ of the retained packs, so its steps are shaped around that copy. | File size | macOS sizes the fresh file for the retained pack bytes before the load (one advisory `F_PREALLOCATE`, one `set_len`), because on APFS a page write that also moves the end of the file costs about four times one inside a sized file. The closed file is cut to the pages its header counts. Linux leaves the file to grow | [`space.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/space.rs) | | Pack copy | The table that holds the pack payload (`pack`, or `pack_unit` in the group-row layouts) is copied one window per statement: `WINDOW_BYTES` (64 MiB) of retained payload, ended early by a dead run longer than `GAP_BYTES` (8 MiB). Every other table is one statement | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs), [`build.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs) | | Read-ahead | Before each window the sealed file's matching range is asked for once, through a second read-only descriptor: `F_RDADVISE` on macOS, `POSIX_FADV_WILLNEED` in 128 KiB steps on Linux. Without it a source that is not in memory is read one pack per device round trip. The range assumes packs lie in the file in id order; a hint only, so another layout loses the benefit and nothing else | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs) | +| Visited scratch | The mark writes one `visited` row per reached object into `.compact-visited`; the retained packs (`live_pack`) are derived from those rows in one statement when the mark finishes | [`compact_visit.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact_visit.rs) | +| Verification | One `immutable=1` reader of the fresh file: the schema check, one statement that reads the row counts and recomputed sums with one pass per table, the live-heads digest and the tuple. The sealed file's tuple and digest are read back through one reader as well | [`verify.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/verify.rs) | | Synchronization | Follows the Store's profile. Normal: the op record (file and directory) at each write, the fresh file before the rename and the directory after it. Disposable (`synchronous=OFF`) claims nothing after a power loss, so its compaction asks for no fsync: the record is still written by rename and the install is still one rename, which is what survives a process crash | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`install.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs), [`run.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs) | ## Heads diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs index c7afc41a30..e70e385cb6 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs @@ -415,8 +415,8 @@ SELECT name,seq FROM src.sqlite_sequence", logical=(SELECT COALESCE(SUM(logical),0) FROM main.\"commit\")+\ (SELECT COALESCE(SUM(logical),0) FROM main.layer),\ canonical=(SELECT COALESCE(SUM(canonical_length),0) FROM main.object_location),\ -stored=(SELECT COALESCE(SUM(length),0) FROM main.pack),\ -metadata=(SELECT COALESCE(SUM(length),0) FROM main.pack WHERE domain=0) WHERE id=1;\ +(stored,metadata)=(SELECT COALESCE(SUM(length),0),\ +COALESCE(SUM(CASE WHEN domain=0 THEN length ELSE 0 END),0) FROM main.pack) WHERE id=1;\ UPDATE main.history_meta SET catalog_incarnation={generation} WHERE id=1;" )) .map_err(db_error)?; diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs index b604913573..09f1461ae6 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs @@ -30,9 +30,8 @@ use super::types::CompactPath; use super::types::CompactReceipt; use super::types::CompactRequest; use super::types::CompactResidue; -use super::verify::immutable_digest; -use super::verify::immutable_tuple; use super::verify::live_tuple; +use super::verify::sealed_state; use super::verify::verify; use super::verify::VerifyInput; use crate::compact_metrics::CompactPhase; @@ -342,8 +341,8 @@ impl SqliteHandles { // §7 step 1.5's fence: the seal must not have moved the generation // or the heads the mark recorded. - let sealed_tuple = match immutable_tuple(&facts.path) { - Ok(tuple) => tuple, + let (sealed_tuple, sealed_digest) = match sealed_state(&facts.path) { + Ok(state) => state, Err(error) => { abort_before_install(&store, &op_id, true); return Err(error); @@ -353,18 +352,11 @@ impl SqliteHandles { abort_before_install(&store, &op_id, true); return Err(HistoryError::Conflict("the Store changed after the mark")); } - match immutable_digest(&facts.path) { - Ok(digest) if digest == mark.live_heads_digest() => {} - Ok(_) => { - abort_before_install(&store, &op_id, true); - return Err(HistoryError::Conflict( - "the Store heads changed after the mark", - )); - } - Err(error) => { - abort_before_install(&store, &op_id, true); - return Err(error); - } + if sealed_digest != mark.live_heads_digest() { + abort_before_install(&store, &op_id, true); + return Err(HistoryError::Conflict( + "the Store heads changed after the mark", + )); } stop_at(request, CompactFailpoint::AfterSeal, &op, &store)?; diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/verify.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/verify.rs index 7b242aef75..1a88691711 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/verify.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/verify.rs @@ -164,6 +164,30 @@ pub(crate) fn file_heads_digest(connection: &Connection) -> HistoryResult<[u8; 3 Ok(*hasher.finalize().as_bytes()) } +/// The staged file's row counts and recomputed sums, one pass per table. +const SCAN_SQL: &str = "SELECT \ +(SELECT count(*) FROM pragma_foreign_key_check),\ +o.n,o.canonical,p.n,p.stored,p.metadata,\ +(SELECT count(*) FROM kept_node),\ +(SELECT COALESCE(SUM(logical),0) FROM \"commit\")+\ +(SELECT COALESCE(SUM(logical),0) FROM layer) \ +FROM (SELECT count(*) AS n,COALESCE(SUM(canonical_length),0) AS canonical \ +FROM object_location) o,\ +(SELECT count(*) AS n,COALESCE(SUM(length),0) AS stored,\ +COALESCE(SUM(CASE WHEN domain=0 THEN length ELSE 0 END),0) AS metadata FROM pack) p"; + +/// What [`SCAN_SQL`] returns, in column order. +struct Scan { + foreign_keys: i64, + objects: i64, + canonical: i64, + packs: i64, + stored: i64, + metadata: i64, + history: i64, + logical: i64, +} + /// The staged Store and the mark a verification reads it against. pub(crate) struct VerifyInput<'a> { pub(crate) store: &'a Path, @@ -198,22 +222,35 @@ pub(crate) fn verify( (layout, acquisition), ) .map_err(HistoryError::from)?; - let count = |sql: &str| -> HistoryResult { - let value: i64 = connection - .query_row(sql, params![], |row| row.get(0)) - .map_err(db_error)?; + // One statement and one pass over each table: the row counts the mark + // fixes and the sums the totals row must equal come from the same scan. + let scanned = connection + .query_row(SCAN_SQL, params![], |row| { + Ok(Scan { + foreign_keys: row.get(0)?, + objects: row.get(1)?, + canonical: row.get(2)?, + packs: row.get(3)?, + stored: row.get(4)?, + metadata: row.get(5)?, + history: row.get(6)?, + logical: row.get(7)?, + }) + }) + .map_err(db_error)?; + let counted = |value: i64| -> HistoryResult { u64::try_from(value).map_err(|_| HistoryError::Integrity("verification count")) }; - if count("SELECT count(*) FROM pragma_foreign_key_check")? != 0 { + if counted(scanned.foreign_keys)? != 0 { return Err(HistoryError::Integrity("staged Store foreign key")); } - if count("SELECT count(*) FROM object_location")? != mark.objects().len() as u64 { + if counted(scanned.objects)? != mark.objects().len() as u64 { return Err(HistoryError::Integrity("staged Store object count")); } - if count("SELECT count(*) FROM pack")? != mark.objects().packs().len() as u64 { + if counted(scanned.packs)? != mark.objects().packs().len() as u64 { return Err(HistoryError::Integrity("staged Store pack count")); } - if count("SELECT count(*) FROM kept_node")? != mark.history().len() as u64 { + if counted(scanned.history)? != mark.history().len() as u64 { return Err(HistoryError::Integrity("staged Store history count")); } stop_at(request, CompactFailpoint::DuringVerify, op, store)?; @@ -232,34 +269,11 @@ pub(crate) fn verify( { return Err(HistoryError::Integrity("staged Store allocator tuple")); } - let totals = connection - .query_row( - "SELECT \ -(SELECT logical FROM store_totals WHERE id=1),\ -(SELECT COALESCE(SUM(logical),0) FROM \"commit\")+\ -(SELECT COALESCE(SUM(logical),0) FROM layer),\ -(SELECT canonical FROM store_totals WHERE id=1),\ -(SELECT COALESCE(SUM(canonical_length),0) FROM object_location),\ -(SELECT stored FROM store_totals WHERE id=1),\ -(SELECT COALESCE(SUM(length),0) FROM pack),\ -(SELECT metadata FROM store_totals WHERE id=1),\ -(SELECT COALESCE(SUM(length),0) FROM pack WHERE domain=0)", - params![], - |row| { - Ok(( - row.get::<_, i64>(0)?, - row.get::<_, i64>(1)?, - row.get::<_, i64>(2)?, - row.get::<_, i64>(3)?, - row.get::<_, i64>(4)?, - row.get::<_, i64>(5)?, - row.get::<_, i64>(6)?, - row.get::<_, i64>(7)?, - )) - }, - ) - .map_err(db_error)?; - if totals.0 != totals.1 || totals.2 != totals.3 || totals.4 != totals.5 || totals.6 != totals.7 + // The totals row the tuple read, against the sums the scan recomputed. + if checked.logical != scanned.logical + || checked.canonical != scanned.canonical + || checked.stored != scanned.stored + || checked.metadata != scanned.metadata { return Err(HistoryError::Integrity("staged Store totals")); } @@ -278,16 +292,12 @@ pub(crate) fn verify( Ok(()) } -pub(crate) fn immutable_tuple(path: &Path) -> HistoryResult { +/// The sealed file's tuple and live-heads digest through one `immutable=1` +/// reader (§7 step 1.5). +pub(crate) fn sealed_state(path: &Path) -> HistoryResult<(MarkTuple, [u8; 32])> { let connection = open_immutable(path)?; let tuple = file_tuple(&connection)?; - connection.close().map_err(|(_, error)| db_error(error))?; - Ok(tuple) -} - -pub(crate) fn immutable_digest(path: &Path) -> HistoryResult<[u8; 32]> { - let connection = open_immutable(path)?; let digest = file_heads_digest(&connection)?; connection.close().map_err(|(_, error)| db_error(error))?; - Ok(digest) + Ok((tuple, digest)) } diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact_visit.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact_visit.rs index a966e83040..d3d85f72c7 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact_visit.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact_visit.rs @@ -87,21 +87,24 @@ BEGIN IMMEDIATE;", .map_err(db_error)? .execute(params![id.as_bytes().as_slice(), pack_id]) .map_err(db_error)?; - connection - .prepare_cached("INSERT OR IGNORE INTO live_pack(pack_id) VALUES(?1)") - .map_err(db_error)? - .execute(params![pack_id]) - .map_err(db_error)?; Ok(()) } - /// Commits the rows, closes the connection and refuses a sidecar. + /// Derives the retained packs, commits, closes the connection and refuses + /// a sidecar. pub(crate) fn finish(mut self) -> HistoryResult { let connection = self .connection .take() .ok_or(HistoryError::Integrity("visited scratch"))?; - connection.execute_batch("COMMIT").map_err(db_error)?; + // The retained packs are the visited rows' packs: one ordered + // statement at the end instead of one insert per reached object. + connection + .execute_batch( + "INSERT INTO live_pack(pack_id) \ +SELECT DISTINCT pack_id FROM visited ORDER BY pack_id; COMMIT", + ) + .map_err(db_error)?; connection.close().map_err(|(_, error)| db_error(error))?; for sidecar in sidecars(&self.path) { if std::fs::symlink_metadata(&sidecar).is_ok() { From f24eadbb6e7d2a6e4587f56bbd4b214261d41e37 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:12:55 +0800 Subject: [PATCH 10/31] Revert "perf(gc): a Disposable Store's compaction issues no fsync" This reverts commit 8e7acb30bb. A review of the merged rebuild commits found that skipping every fsync on a Disposable Store changes what a power loss can do during a compaction: - after the rename, the fresh file's pages may not be on disk, so the whole Store can be lost, not only the compaction; - an operation record that was never synchronized can come back unreadable, and the leftover files then make `store.compact` refuse until someone deletes them by hand; - the synchronization before the rename was also the last point where a deferred write error of the fresh file could stop the install. The spec's behaviour (section 8: synchronize the fresh file, the record and the directory around the rename) stays until the owner decides otherwise. Cost on the 1.1 GB bench shape, release, one sample, load 4.4: install 71.6 ms of a 1.045 s rebuild. Tests: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), sqlite_group_rows (7), kept_objects (7) pass. sqlite_compact_matrix failed once and passed once with no source change between: its AC-49 arm unwraps a Save inside the racing Commit thread (tests/sqlite_compact_matrix.rs:149), which panics when the rename lands during that Save. The test is unchanged on this branch; not fixed here. Production LOC: 188445 -> 188429 (delta -16) Scope: core, engine, control and api product source; counted with tools/production_loc.py on the first parent and the committed tree. Co-Authored-By: Claude Fable 5.2 --- docs/architecture/engine/store/sqlite.md | 2 +- .../src/backend/sqlite/compact/install.rs | 25 +++++++------------ .../src/backend/sqlite/compact/op.rs | 22 ++++------------ .../src/backend/sqlite/compact/run.rs | 21 +++++----------- 4 files changed, 21 insertions(+), 49 deletions(-) diff --git a/docs/architecture/engine/store/sqlite.md b/docs/architecture/engine/store/sqlite.md index 58b52d4214..99f7d9ba86 100644 --- a/docs/architecture/engine/store/sqlite.md +++ b/docs/architecture/engine/store/sqlite.md @@ -176,7 +176,7 @@ of the retained packs, so its steps are shaped around that copy. | Read-ahead | Before each window the sealed file's matching range is asked for once, through a second read-only descriptor: `F_RDADVISE` on macOS, `POSIX_FADV_WILLNEED` in 128 KiB steps on Linux. Without it a source that is not in memory is read one pack per device round trip. The range assumes packs lie in the file in id order; a hint only, so another layout loses the benefit and nothing else | [`window.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/window.rs) | | Visited scratch | The mark writes one `visited` row per reached object into `.compact-visited`; the retained packs (`live_pack`) are derived from those rows in one statement when the mark finishes | [`compact_visit.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact_visit.rs) | | Verification | One `immutable=1` reader of the fresh file: the schema check, one statement that reads the row counts and recomputed sums with one pass per table, the live-heads digest and the tuple. The sealed file's tuple and digest are read back through one reader as well | [`verify.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/verify.rs) | -| Synchronization | Follows the Store's profile. Normal: the op record (file and directory) at each write, the fresh file before the rename and the directory after it. Disposable (`synchronous=OFF`) claims nothing after a power loss, so its compaction asks for no fsync: the record is still written by rename and the install is still one rename, which is what survives a process crash | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`install.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs), [`run.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs) | +| Synchronization | The op record is synchronized at each write, the fresh file before the rename and the directory after it | [`op.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs), [`install.rs`](../../../../engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs) | ## Heads diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs index 036f664f0c..c7d0c22d24 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/install.rs @@ -116,25 +116,20 @@ pub(crate) fn install_fence(facts: &SealedFacts, tuple: &MarkTuple) -> HistoryRe Ok(()) } -/// §7 step 7: `chmod 0600` and `sync_all` before the rename. A Store whose -/// profile synchronizes nothing (`durable` false) skips the `sync_all`: the -/// rename then installs a file that is complete in the page cache, which is -/// all that profile promises of the Store itself. -pub(crate) fn prepare_target(target: &Path, durable: bool) -> HistoryResult<()> { +/// §7 step 7: `chmod 0600` and `sync_all` before the rename. +pub(crate) fn prepare_target(target: &Path) -> HistoryResult<()> { #[cfg(unix)] { use std::os::unix::fs::PermissionsExt; std::fs::set_permissions(target, std::fs::Permissions::from_mode(0o600)) .map_err(io_error)?; } - if durable { - let file = std::fs::OpenOptions::new() - .read(true) - .write(true) - .open(target) - .map_err(io_error)?; - file.sync_all().map_err(io_error)?; - } + let file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(target) + .map_err(io_error)?; + file.sync_all().map_err(io_error)?; Ok(()) } @@ -217,9 +212,7 @@ pub(crate) fn settle( |name| name.to_string_lossy().into_owned(), )); } - if op.durable { - let _ = sync_directory(store); - } + let _ = sync_directory(store); if !residue.artifacts.is_empty() && residue.reason.is_none() { residue.reason = Some("the artifact set could not be removed".to_owned()); } diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs index 57e46660f5..79a65b204a 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/op.rs @@ -121,12 +121,6 @@ pub(crate) struct RunOp { /// The §11 totals once the rebuild's target is built; `None` before any /// report value exists and in a record written before they were kept. pub(crate) installed: Option, - /// Whether this run synchronizes what it writes beside the Store. It is - /// the Store's own profile, not part of the record: a Disposable Store - /// (`synchronous=OFF`) survives a process crash and claims nothing after - /// a power loss, so its compaction asks for no fsync either. A record - /// read back from disk is written synchronized. - pub(crate) durable: bool, } impl RunOp { @@ -171,16 +165,12 @@ live_heads_digest={}\nreturned={}\n", let path = op_path(store); let temporary = op_temporary_path(store); std::fs::write(&temporary, self.encode()).map_err(io_error)?; - if self.durable { - if let Ok(file) = std::fs::File::open(&temporary) { - let _ = file.sync_all(); - } + if let Ok(file) = std::fs::File::open(&temporary) { + let _ = file.sync_all(); } std::fs::rename(&temporary, &path).map_err(io_error)?; - if self.durable { - if let Some(parent) = path.parent() { - let _ = std::fs::File::open(parent).and_then(|directory| directory.sync_all()); - } + if let Some(parent) = path.parent() { + let _ = std::fs::File::open(parent).and_then(|directory| directory.sync_all()); } Ok(()) } @@ -248,7 +238,6 @@ live_heads_digest={}\nreturned={}\n", live_heads_digest, returned, installed: InstalledReport::read(&values), - durable: true, }) } @@ -294,7 +283,7 @@ pub(crate) fn remove_op_record(store: &Path, op_id: &[u8; 16]) -> HistoryResult< /// so the next `store.compact` judges a leftover even though the owner pid is /// still alive (§9). Only the run's own record is marked: a record another run /// wrote in the settle window is never touched. -pub(crate) fn mark_returned(store: &Path, op_id: &[u8; 16], durable: bool) -> HistoryResult<()> { +pub(crate) fn mark_returned(store: &Path, op_id: &[u8; 16]) -> HistoryResult<()> { let Some(mut op) = RunOp::read(store) else { return Ok(()); }; @@ -302,7 +291,6 @@ pub(crate) fn mark_returned(store: &Path, op_id: &[u8; 16], durable: bool) -> Hi return Ok(()); } op.returned = true; - op.durable = durable; op.write(store) } diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs index 09f1461ae6..2508cd45a2 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/run.rs @@ -60,7 +60,6 @@ impl SqliteHandles { /// op record are kept for the next `store.compact` to judge (§9). pub fn compact(self, request: &CompactRequest) -> HistoryResult { let store = self.storage.session.path.clone(); - let durable = self.storage.session.profile.synchronous != 0; let mut metrics = RunMetrics::start(); let result = self.compact_measured(request, &mut metrics); // §9: the run is replying, so its own record (when it leaves one) is @@ -68,7 +67,7 @@ impl SqliteHandles { // `store.compact` judges the leftover even while this process is still // alive; the window before this point stays protected by the owner // liveness guard. - let _ = mark_returned(&store, &request.op_id, durable); + let _ = mark_returned(&store, &request.op_id); metrics.finish(&result); result } @@ -98,11 +97,6 @@ impl SqliteHandles { let layout = session.layout; let acquisition_schema = session.acquisition; let synchronous = profile.synchronous; - // The run synchronizes what the Store's profile does: a - // Disposable Store (`synchronous=OFF`) claims nothing after a - // power loss, so its compaction asks for no fsync of the op - // record, the fresh file or the directory. - let durable = synchronous != 0; // §5.1/§7 admission: a client that holds the Store keeps its // sidecars, and the seal cannot remove one its owner keeps: it @@ -163,8 +157,8 @@ impl SqliteHandles { // The op record is written before the mark so a crash inside the // mark transaction (which creates the §6.3 visited scratch before // it decides) leaves a *recorded* run for the next call to adopt, - // never an unrecorded staged name (§9). It stands through the - // mark; the build's write carries the mark's tuple and digest. + // never an unrecorded staged name (§9). The mark phase overwrites + // it once the mark's tuple and digest are known. let mut op = RunOp { op_id, generation: live.generation, @@ -179,7 +173,6 @@ impl SqliteHandles { live_heads_digest: [0u8; 32], returned: false, installed: None, - durable, }; op.write(&store)?; @@ -264,9 +257,7 @@ impl SqliteHandles { abort_before_install(&store, &op_id, false); return Err(error); } - if durable { - let _ = sync_directory(&store); - } + let _ = sync_directory(&store); drop(lane); return Ok(receipt( CompactPath::None, @@ -444,7 +435,7 @@ impl SqliteHandles { abort_before_install(&store, &op_id, true); return Err(error); } - if let Err(error) = prepare_target(&built.target, durable) { + if let Err(error) = prepare_target(&built.target) { abort_before_install(&store, &op_id, true); return Err(error); } @@ -460,7 +451,7 @@ impl SqliteHandles { // §7 step 7: the rename happened; a crash before the directory // fsync leaves an installed-but-uncertain `G + 1`. stop_at(request, CompactFailpoint::DuringRename, &op, &store)?; - if durable && sync_directory(&facts.path).is_err() { + if let Err(_error) = sync_directory(&facts.path) { op.phase = "install".to_owned(); let _ = op.write(&store); return Err(HistoryError::UnknownOutcome); From cb0a97bbc54e655f2e76f989f293fac221e8f6e1 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:13:34 +0800 Subject: [PATCH 11/31] perf(gc): the windowed pack copy seeks only its own window's packs `copy_packs` copies retained packs one 64 MiB window of pack ids at a time. The range sat on the source table only, and SQLite drives that statement from `scratch.live_pack`, so every window walked every retained pack of the Store and seeked the source for each: windows times retained packs seeks. A review measured 200 to 250 ms per window at 800,000 retained packs against 0.1 ms with the range inside the subquery. The range now bounds the `live_pack` scan itself, so a window touches only its own packs and the copy is linear in retained packs. 1.1 GB bench shape, release, one sample, load 4.4: rebuild 1.045 s (mark 71.7 ms, build 791.6 ms, verify 97.1 ms, install 71.6 ms). That Store has too few windows to show the difference; the count argument above is the evidence. Tests: as the parent commit (same build), including the sqlite_compact_matrix note there. Production LOC: 188429 -> 188429 (delta 0) Scope: core, engine, control and api product source; counted with tools/production_loc.py on the first parent and the committed tree. Co-Authored-By: Claude Fable 5.2 --- .../src/backend/sqlite/compact/build.rs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs index e70e385cb6..470b3a7115 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/compact/build.rs @@ -230,8 +230,11 @@ fn copy_packs( let name = identifier(table); let mut statement = connection .prepare(&format!( - "INSERT INTO main.{name} SELECT * FROM src.{name} \ -WHERE pack_id BETWEEN ?1 AND ?2 AND {RETAINED_PACK}" + // The range goes inside the subquery: SQLite drives this loop + // from `live_pack`, so a range on the outer table alone would + // seek every retained pack of the Store once per window. + "INSERT INTO main.{name} SELECT * FROM src.{name} WHERE pack_id IN \ +(SELECT pack_id FROM scratch.live_pack WHERE pack_id BETWEEN ?1 AND ?2)" )) .map_err(db_error)?; for window in plan { From c15826e2918510a905852857de5adce9ef049681 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:19:12 +0800 Subject: [PATCH 12/31] perf(gc): mark reads object locations without sorting them The mark loads every `object_location` row once. The query asked for pack order, which no index supplies, so each mark made the engine sort the whole table (`USE TEMP B-TREE FOR ORDER BY` on SQLite, a sort node on PostgreSQL). The order bought nothing: `object_id` is the table's key, so an object has one row and there is no winner to pick, and both maps the rows fill are keyed. The query now reads the table in its own key order on both backends. gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs), release, warm second mark, one sample, load 6.7: engine steps of a mark 217,406 -> 128,273 sorts 6 -> 5 the statement 3.87 ms -> 1.88 ms Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30), postgresql_statement_schema (2) pass; Clippy and fmt clean. Production LOC: 188,429 -> 188,428 (delta -1) core 40,589 -> 40,589 (0) engine 90,240 -> 90,239 (-1) layerfs-persistence -1 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent cb0a97bbc5 and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../queries/history/gc_object_locations.sql | 1 - .../backend/postgresql/history/statements.rs | 2 +- .../layerfs-persistence/src/history/objects.rs | 17 +++++++++++------ 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/gc_object_locations.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/gc_object_locations.sql index e771d5bf4c..be156c6979 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/gc_object_locations.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/gc_object_locations.sql @@ -1,3 +1,2 @@ SELECT object_id,pack_id,group_number,record_number,role,canonical_length FROM object_location - ORDER BY pack_id,group_number,record_number diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs index c81ef56fb4..951d9d8044 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs @@ -13,7 +13,7 @@ pub(crate) fn statement(key: &str) -> Result<&'static str, BackendError> { "../../../../sql/postgresql/queries/history/statistics_totals.sql" )), "gc_object_locations" => Ok( - "SELECT object_id,pack_id,group_number::bigint,record_number::bigint,role::bigint,canonical_length::bigint FROM object_location ORDER BY pack_id,group_number,record_number", + "SELECT object_id,pack_id,group_number::bigint,record_number::bigint,role::bigint,canonical_length::bigint FROM object_location", ), "gc_epoch_read" => Ok(include_str!( "../../../../sql/postgresql/queries/history/gc_epoch_read.sql" diff --git a/engine/crates/layerfs-persistence/src/history/objects.rs b/engine/crates/layerfs-persistence/src/history/objects.rs index 97fde644e1..48049ebe5c 100644 --- a/engine/crates/layerfs-persistence/src/history/objects.rs +++ b/engine/crates/layerfs-persistence/src/history/objects.rs @@ -2,7 +2,7 @@ //! digest. //! //! A mark starts at the roots of every Commit and Layer the §6.1 history -//! closure retained. Every locator is loaded once, in pack order, and the walk +//! closure retained. Every locator is loaded once, unsorted, and the walk //! follows each object's direct logical references and its record-header delta //! base to the fixpoint. The walk is metadata-only: a role that names no other //! object costs one header read, not a payload decode, and reference-bearing @@ -252,9 +252,9 @@ fn mark_recorded( let history = kept::mark(tx)?; let policy = tx.mechanics().policy().map_err(storage)?; let capacities = StorageCapacities::from_policy(policy).map_err(storage)?; - // Every locator is loaded once, in pack order. The walk's queue and the - // kept-waste attribution share that one read, so a mark never asks the - // engine for a single object location again. + // Every locator is loaded once. The walk's queue and the kept-waste + // attribution share that one read, so a mark never asks the engine for a + // single object location again. let located = locate_all(tx)?; let source = ClosureSource { mechanics: tx.mechanics(), @@ -290,7 +290,12 @@ fn roots(tx: &Transaction<'_>) -> HistoryResult> { Ok(roots.into_iter().collect()) } -/// Every stored locator, read once in pack order. +/// Every stored locator, read once in the table's own key order. +/// +/// An object has one locator row (its identity is the table's key), so no +/// order is needed to pick a winner, and asking for pack order made the +/// engine sort every row of the table for each mark. Both maps below are +/// keyed, so they do not need sorted input either. /// /// `locations` answers the walk; `inventory` answers kept-waste attribution. /// They are two views of the same single `gc_object_locations` read, so the @@ -342,7 +347,7 @@ fn locate_all(tx: &Transaction<'_>) -> HistoryResult { Ok(located) } -/// The in-memory closure of one mark, before `keptWaste` is attributed./// The in-memory closure of one mark, before `keptWaste` is attributed. +/// The in-memory closure of one mark, before `keptWaste` is attributed. struct Closure { /// Every reached object and its winning locator. visited: BTreeMap, From c1d48b6d4fbe7c30cd4034d9d293fd20e1e332e0 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:22:57 +0800 Subject: [PATCH 13/31] perf(gc): mark's first read of a pack stays on the pack row's own page The mark asks each pack for its first bytes: the directory and, for the common one-record pack, the record header that names its delta base. It asked for 1,024 bytes. SQLite keeps only the first part of a large row on the row's own page (at least 489 bytes of a row too large for a 4,096-byte page) and the rest in a chain of further pages, so most of those reads went one page past the row: a second page load per pack, and a random read on a cold cache. The first read is now 432 bytes. The columns before the body take at most 49 bytes, so up to 440 bytes of body are always on the page the row lookup has already loaded. A pack with a wider directory is read again, as before. gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs, every pack at the 489-byte minimum), release, one run of three marks per size: first read time in pack reads warm mark wall 1-min load 1,024 bytes 8.2 to 8.6 ms 38.8 to 40.8 ms 4.9 448 bytes 8.9 to 9.4 ms 39.9 to 43.9 ms 6.8 440 bytes 0.2 ms 28.9 to 34.7 ms 6.5 432 bytes 0.2 ms 32.7 to 40.4 ms 9.2 Read calls are unchanged at 9,104; bytes read fall from 8,972,342 to 3,857,332. The wall times were taken under different load, so the pack-read time is the comparison to rely on. Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30) pass; Clippy and fmt clean. Production LOC: 188,428 -> 188,428 (delta 0) core 40,589 -> 40,589 (0) engine 90,239 -> 90,239 (0) control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent c15826e291 and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/history/objects/edges.rs | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/history/objects/edges.rs b/engine/crates/layerfs-persistence/src/history/objects/edges.rs index bc3cee809d..85bf9d083e 100644 --- a/engine/crates/layerfs-persistence/src/history/objects/edges.rs +++ b/engine/crates/layerfs-persistence/src/history/objects/edges.rs @@ -29,10 +29,17 @@ use std::collections::BTreeMap; /// After the head: a wider directory, then record headers. No pack needs more. const ROUNDS: usize = 4; /// Bytes of each pack the first read asks for: the directory of a pack of up -/// to about sixty groups, its first group's record directory and first record -/// header. Small on purpose: on a cold cache each further page of a pack is -/// one more random read. A pack with a wider directory is read again. -const HEAD_BYTES: usize = 1024; +/// to about twenty groups, its first group's record directory and first +/// record header. A pack with a wider directory is read again. +/// +/// The size is chosen so the read never leaves the pack row's own page in a +/// SQLite Store. A row too large for its 4,096-byte page keeps at least its +/// first 489 bytes there and the rest in a chain of further pages; the +/// columns before the body take at most 49 of those. Up to 440 bytes of body +/// are therefore always on the page the row lookup has already loaded, and +/// one byte more costs every such pack another page read: a random one on a +/// cold cache. A PostgreSQL Store reads its first stored piece either way. +const HEAD_BYTES: usize = 432; /// Packs whose heads one call reads. const HEAD_PACKS: usize = 256; /// Ranges one call asks for, over all its packs. From d60042965fb7ac09ea56d4f989a03cf07456e0c0 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:31:39 +0800 Subject: [PATCH 14/31] perf(gc): mark's walk looks objects up in constant time A CPU sample of warm marks (after the pack reads became cheap) put more than half of a mark in map bookkeeping: the walk and the kept-waste pass ask ordered maps keyed by 32-byte identity about every object several times, and each question pays a 32-byte comparison per tree level. Those maps are now hash maps (`history/objects/ids.rs`). The standard hasher was tried first and cost as much as the comparisons it replaced, because it runs its permutation over all 32 bytes. An identity is a BLAKE3 digest, so `IdState` takes its first eight bytes and mixes them with a mask and multiplier drawn per map from the standard random source. The key is there because identities come from stored content: a writer could otherwise choose content whose identities share table slots. Results do not depend on it: the two lists a mark returns in identity order (`KeptObjects::objects`, the delta base references) are sorted after collection, and the walk still orders each wave by locator. The per-pack maps (`read_bytes`, `layouts`) are hashed too. The pack inventory keeps its ordered maps. `MarkGroups` moves to `history/objects/plan.rs`, unchanged, so `objects.rs` stays under the 999-line ceiling (958 -> 895 lines). 100 warm marks of the kept 1.1 GB bench Store (8,913 objects, 8,816 packs), release, two runs per build, interleaved, 1-min load 7.6 to 7.8: ordered maps user 3.83 s, 3.92 s mean mark 43.1 ms, 42.0 ms standard hasher user 3.50 s, 3.18 s (load 9.0 to 9.2; not kept) this commit user 2.82 s, 2.86 s mean mark 30.7 ms, 31.3 ms Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30) pass; Clippy, fmt and the product boundary guard clean. Production LOC: 188,428 -> 188,488 (delta +60) core 40,589 -> 40,589 (0) engine 90,239 -> 90,299 (+60) layerfs-persistence +60 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent c1d48b6d4f and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/history/objects.rs | 124 +++++------------- .../src/history/objects/edges.rs | 5 +- .../src/history/objects/ids.rs | 81 ++++++++++++ .../src/history/objects/plan.rs | 71 ++++++++++ 4 files changed, 191 insertions(+), 90 deletions(-) create mode 100644 engine/crates/layerfs-persistence/src/history/objects/ids.rs create mode 100644 engine/crates/layerfs-persistence/src/history/objects/plan.rs diff --git a/engine/crates/layerfs-persistence/src/history/objects.rs b/engine/crates/layerfs-persistence/src/history/objects.rs index 48049ebe5c..b4dfadd207 100644 --- a/engine/crates/layerfs-persistence/src/history/objects.rs +++ b/engine/crates/layerfs-persistence/src/history/objects.rs @@ -32,20 +32,22 @@ use layerfs_storage::{ }, DecompressionWorkspace, GroupCache, PackAcquisition, PackCache, PackUnit, }, - location::{ObjectLocation, PackInfo, SignatureRow}, - pack::layout::{self, GroupCodec, PackLane}, - port::{ - AcquiredPackRead, PackRange, PackReadChoice, PackReadPlan, PersistenceError, - PACK_READ_PREFIX_BYTES, - }, + location::{ObjectLocation, SignatureRow}, + pack::layout::{self, GroupCodec}, + port::{AcquiredPackRead, PACK_READ_PREFIX_BYTES}, source::Source, StorageCapacities, StorageError, }; -use std::collections::{BTreeMap, BTreeSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::path::Path; mod edges; +mod ids; mod impls; +mod plan; + +use ids::{IdMap, IdSet}; +use plan::MarkGroups; const ROOTS: &str = "gc_kept_roots"; const LOCATIONS: &str = "gc_object_locations"; const LIVE_STACKS: &str = "gc_live_stacks"; @@ -259,8 +261,8 @@ fn mark_recorded( let source = ClosureSource { mechanics: tx.mechanics(), locations: &located.locations, - read_bytes: std::cell::RefCell::new(BTreeMap::new()), - layouts: std::cell::RefCell::new(BTreeMap::new()), + read_bytes: std::cell::RefCell::new(HashMap::new()), + layouts: std::cell::RefCell::new(HashMap::new()), }; let roots = roots(tx)?; let closure = walk(&source, &capacities, policy.construction(), roots, visit)?; @@ -268,7 +270,7 @@ fn mark_recorded( let mark_bytes = source.read_bytes(); let digest = live_heads_digest(tx)?; let objects = KeptObjects { - objects: closure.visited.keys().copied().collect(), + objects: closure.objects(), packs: closure.packs(), waste, }; @@ -302,8 +304,8 @@ fn roots(tx: &Transaction<'_>) -> HistoryResult> { /// mark no longer loads `object_location` twice (once per object and once for /// the waste inventory). struct Located { - /// The winning locator of every object, keyed by identity. - locations: BTreeMap, + /// The locator of every object, keyed by identity. + locations: IdMap, /// pack -> group -> (record, object): the pack/object inventory. inventory: BTreeMap>>, } @@ -311,7 +313,7 @@ struct Located { fn locate_all(tx: &Transaction<'_>) -> HistoryResult { let rows = tx.query(LOCATIONS, Vec::::new())?; let mut located = Located { - locations: BTreeMap::new(), + locations: IdMap::with_capacity_and_hasher(rows.len(), ids::IdState::default()), inventory: BTreeMap::new(), }; for row in &rows { @@ -349,17 +351,24 @@ fn locate_all(tx: &Transaction<'_>) -> HistoryResult { /// The in-memory closure of one mark, before `keptWaste` is attributed. struct Closure { - /// Every reached object and its winning locator. - visited: BTreeMap, + /// Every reached object and its locator. + visited: IdMap, /// The stored width a read charges each reached object's record. - widths: BTreeMap, + widths: IdMap, /// Objects reached as a direct delta base. - bases: BTreeSet, + bases: IdSet, /// Objects reached as a root or through a logical reference. - logical: BTreeSet, + logical: IdSet, } impl Closure { + /// Every reached object, in ascending identity order. + fn objects(&self) -> Vec { + let mut objects: Vec = self.visited.keys().copied().collect(); + objects.sort_unstable(); + objects + } + /// The retained packs: those holding at least one reached object. fn packs(&self) -> Vec { self.visited @@ -375,15 +384,15 @@ impl Closure { struct ClosureSource<'a> { mechanics: &'a dyn crate::backend::Mechanics, /// Every locator, exactly as the one `gc_object_locations` read returned it. - locations: &'a BTreeMap, + locations: &'a IdMap, /// pack -> the body bytes this mark read from it: a whole body, or the /// disjoint group extents it acquired. The sum is /// [`MarkResult::mark_bytes`]: a pack read in several groups counts each /// extent once, and a later whole-body read replaces the partial sum. - read_bytes: std::cell::RefCell>, + read_bytes: std::cell::RefCell>, /// pack -> its group extents, kept from the walk's own read of the pack's /// directory so the kept-waste pass does not read it a second time. - layouts: std::cell::RefCell>>, + layouts: std::cell::RefCell>>, } impl ClosureSource<'_> { @@ -416,68 +425,6 @@ impl ClosureSource<'_> { } } -/// The complete encoded groups one mark walk demands from one pack. -/// -/// A metadata-only walk needs one record header per object and one delta edge -/// per record, so the demand is sparse; the plan selects those exact group -/// ranges and the backend reads only them, never the body behind the -/// directory. A dense demand or a cache that already holds another group of -/// the pack still materializes the whole body once. -struct MarkGroups<'a> { - pack_id: i64, - groups: &'a [usize], - whole: bool, -} - -impl PackReadPlan for MarkGroups<'_> { - fn select( - &mut self, - info: PackInfo, - prefix: &[u8], - ) -> Result { - if info.pack_id != self.pack_id || self.groups.is_empty() { - return Err(PersistenceError::Malformed); - } - let header = layout::parse_directory_header(prefix, info.length) - .map_err(|_| PersistenceError::Malformed)?; - if self.whole || header.lane == PackLane::Singleton || info.length <= PACK_READ_PREFIX_BYTES - { - return Ok(PackReadChoice::Whole); - } - let views = layout::directory_group_views(prefix, header) - .map_err(|_| PersistenceError::Malformed)?; - let mut ranges: Vec = Vec::new(); - let mut useful = 0usize; - let mut previous = None; - for &group in self.groups { - if previous.is_some_and(|prior| prior >= group) { - return Err(PersistenceError::Malformed); - } - previous = Some(group); - let view = *views.get(group).ok_or(PersistenceError::Malformed)?; - useful = useful - .checked_add(view.end - view.start) - .ok_or(PersistenceError::Malformed)?; - if let Some(prior) = ranges - .last_mut() - .filter(|prior| prior.offset + prior.length == view.start) - { - prior.length += view.end - view.start; - } else { - ranges.push(PackRange { - offset: view.start, - length: view.end - view.start, - }); - } - } - if useful >= info.length.div_ceil(2) || ranges.len() > 4 { - Ok(PackReadChoice::Whole) - } else { - Ok(PackReadChoice::Ranges(ranges)) - } - } -} - impl Source for ClosureSource<'_> { fn ordered_dependencies(&self) -> bool { false @@ -607,14 +554,14 @@ fn walk( let mut counters = ChainCounters::default(); let mut edge_decodes = 0u64; let mut closure = Closure { - visited: BTreeMap::new(), - widths: BTreeMap::new(), - bases: BTreeSet::new(), + visited: IdMap::default(), + widths: IdMap::default(), + bases: IdSet::default(), logical: roots.iter().copied().collect(), }; // One hash for the whole walk, not one per metadata object. let unscoped = InodeScope::from_object(ObjectId::for_bytes(b"gc.mark/object-closure")); - let mut queued: BTreeSet = roots.iter().copied().collect(); + let mut queued: IdSet = roots.iter().copied().collect(); let mut frontier: Vec = roots; while !frontier.is_empty() { // Pack order: one body answers every object of its group, so a wave @@ -792,11 +739,12 @@ fn waste( .saturating_add(dead_raw_record_bytes(directory, records, closure)?); } } - let deltas: Vec = closure + let mut deltas: Vec = closure .bases .difference(&closure.logical) .copied() .collect(); + deltas.sort_unstable(); let delta_bases_bytes = deltas.iter().fold(0u64, |total, id| { total.saturating_add(closure.widths.get(id).copied().unwrap_or(0)) }); diff --git a/engine/crates/layerfs-persistence/src/history/objects/edges.rs b/engine/crates/layerfs-persistence/src/history/objects/edges.rs index 85bf9d083e..39401c203e 100644 --- a/engine/crates/layerfs-persistence/src/history/objects/edges.rs +++ b/engine/crates/layerfs-persistence/src/history/objects/edges.rs @@ -10,6 +10,7 @@ //! Reads are per wave, not per pack: one call for the heads of many packs, //! then at most a few calls for whatever ranges those packs still need. A //! pack costs no statement and no round trip of its own. +use super::ids::IdMap; use super::{storage, ClosureSource}; use layerfs_content::ObjectId; use layerfs_history::error::{HistoryError, HistoryResult}; @@ -375,7 +376,7 @@ fn settle(source: &ClosureSource<'_>, work: &mut [PackWork<'_>]) -> HistoryResul pub(super) fn wave_edges( source: &ClosureSource<'_>, wave: &[ObjectId], -) -> HistoryResult> { +) -> HistoryResult> { let mut packs: Vec<(i64, Vec)> = Vec::new(); for id in wave { let Some(location) = source.location(*id, i64::MAX).map_err(storage)? else { @@ -389,7 +390,7 @@ pub(super) fn wave_edges( _ => packs.push((location.pack_id, vec![location])), } } - let mut edges = BTreeMap::new(); + let mut edges = IdMap::default(); for batch in packs.chunks(HEAD_PACKS) { let ids: Vec = batch.iter().map(|(pack_id, _)| *pack_id).collect(); let mut heads = Vec::with_capacity(ids.len()); diff --git a/engine/crates/layerfs-persistence/src/history/objects/ids.rs b/engine/crates/layerfs-persistence/src/history/objects/ids.rs new file mode 100644 index 0000000000..cd0135aace --- /dev/null +++ b/engine/crates/layerfs-persistence/src/history/objects/ids.rs @@ -0,0 +1,81 @@ +//! Hash maps keyed by object identity, for the lookups of one mark. +//! +//! A walk asks several maps about every object it reaches. An ordered map +//! pays a 32-byte comparison at each level for every question, and the +//! standard hasher runs its keyed permutation over all 32 bytes; either costs +//! more than the rest of the walk's bookkeeping. An identity is a BLAKE3 +//! digest, so its first eight bytes are already uniform: [`IdState`] takes +//! those and mixes them with a key drawn per map. +//! +//! The key matters. Identities come from stored content, so a writer can +//! choose content whose identities share bits. Without a secret key such a +//! set would fall into one chain of the table and turn every lookup into a +//! scan. Results never depend on the key: nothing here is iterated in map +//! order without sorting first. +use layerfs_content::ObjectId; +use std::collections::{HashMap, HashSet}; +use std::hash::{BuildHasher, Hasher, RandomState}; + +/// A map from object identity. +pub(super) type IdMap = HashMap; +/// A set of object identities. +pub(super) type IdSet = HashSet; + +/// One map's secret: a mask and an odd multiplier. +#[derive(Clone, Copy)] +pub(super) struct IdState { + mask: u64, + multiplier: u64, +} + +impl Default for IdState { + fn default() -> Self { + // The standard source of per-process random hashing keys. + let random = RandomState::new(); + Self { + mask: random.hash_one(0u8), + multiplier: random.hash_one(1u8) | 1, + } + } +} + +impl BuildHasher for IdState { + type Hasher = IdHasher; + + fn build_hasher(&self) -> IdHasher { + IdHasher { + state: *self, + word: 0, + } + } +} + +/// Folds an identity's leading bytes; see the module comment. +pub(super) struct IdHasher { + state: IdState, + word: u64, +} + +impl Hasher for IdHasher { + fn write(&mut self, bytes: &[u8]) { + // An identity arrives as one 32-byte write, and its first eight bytes + // are enough. A shorter write is padded with zeros, so the hasher + // stays correct, if weaker, for any other key. + let mut word = [0u8; 8]; + for (index, byte) in bytes.iter().take(8).enumerate() { + word[index] = *byte; + } + self.word = self.word.rotate_left(29) ^ u64::from_le_bytes(word); + } + + fn write_usize(&mut self, _length: usize) { + // The length prefix of a fixed-width identity carries nothing. + } + + fn finish(&self) -> u64 { + // The multiplication spreads every input bit upward; the fold brings + // the high half back down, where the table takes its slot index. + let mixed = (self.word ^ self.state.mask).wrapping_mul(self.state.multiplier); + mixed ^ (mixed >> 32) + } +} diff --git a/engine/crates/layerfs-persistence/src/history/objects/plan.rs b/engine/crates/layerfs-persistence/src/history/objects/plan.rs new file mode 100644 index 0000000000..2993013c7f --- /dev/null +++ b/engine/crates/layerfs-persistence/src/history/objects/plan.rs @@ -0,0 +1,71 @@ +//! Which bytes of a pack one acquisition of the mark's walk reads. +//! +//! Kept in a submodule so `objects.rs` stays within the product source-line +//! ceiling. +use layerfs_storage::{ + location::PackInfo, + pack::layout::{self, PackLane}, + port::{PackRange, PackReadChoice, PackReadPlan, PersistenceError, PACK_READ_PREFIX_BYTES}, +}; + +/// The complete encoded groups one mark walk demands from one pack. +/// +/// A metadata-only walk needs one record header per object and one delta edge +/// per record, so the demand is sparse; the plan selects those exact group +/// ranges and the backend reads only them, never the body behind the +/// directory. A dense demand or a cache that already holds another group of +/// the pack still materializes the whole body once. +pub(super) struct MarkGroups<'a> { + pub(super) pack_id: i64, + pub(super) groups: &'a [usize], + pub(super) whole: bool, +} + +impl PackReadPlan for MarkGroups<'_> { + fn select( + &mut self, + info: PackInfo, + prefix: &[u8], + ) -> Result { + if info.pack_id != self.pack_id || self.groups.is_empty() { + return Err(PersistenceError::Malformed); + } + let header = layout::parse_directory_header(prefix, info.length) + .map_err(|_| PersistenceError::Malformed)?; + if self.whole || header.lane == PackLane::Singleton || info.length <= PACK_READ_PREFIX_BYTES + { + return Ok(PackReadChoice::Whole); + } + let views = layout::directory_group_views(prefix, header) + .map_err(|_| PersistenceError::Malformed)?; + let mut ranges: Vec = Vec::new(); + let mut useful = 0usize; + let mut previous = None; + for &group in self.groups { + if previous.is_some_and(|prior| prior >= group) { + return Err(PersistenceError::Malformed); + } + previous = Some(group); + let view = *views.get(group).ok_or(PersistenceError::Malformed)?; + useful = useful + .checked_add(view.end - view.start) + .ok_or(PersistenceError::Malformed)?; + if let Some(prior) = ranges + .last_mut() + .filter(|prior| prior.offset + prior.length == view.start) + { + prior.length += view.end - view.start; + } else { + ranges.push(PackRange { + offset: view.start, + length: view.end - view.start, + }); + } + } + if useful >= info.length.div_ceil(2) || ranges.len() > 4 { + Ok(PackReadChoice::Whole) + } else { + Ok(PackReadChoice::Ranges(ranges)) + } + } +} From 97169f06abf6496218d0b6058fc5476a09ae85fd Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:53:28 +0800 Subject: [PATCH 15/31] perf(gc): PostgreSQL Phase A reads each batch from an indexed copy set (layerfs-project#41) Neither the Store's nor the scratch `object_location` has a `pack_id` index, so every Phase A batch scanned the whole source table and the whole visited set three times and the growing scratch table four times (EXPLAIN ANALYZE on PostgreSQL 17.11: a batch holding 16,161 of 290,561 rows still read all 290,561 source rows and all 258,561 visited rows per statement). The copy cost batches times objects. Phase A now makes one scan of the source into a session temporary table of the visited location rows (`pg_temp.gc_copy`), indexed on `pack_id`. Each batch reads its own range through that index and takes its row counts from the two INSERTs. The copy set must hold exactly one row per visited object, and the rows copied by all batches must add up to it, so a visited object in a pack outside the live list is still refused. The final scratch check (count = count(DISTINCT object_id) = |visited|, pack count) is unchanged and is now one statement. The retained canonical total is read from the copy set instead of a second semi-join scan of the source. Kept as before: one transaction per batch, the 1 GiB batch bound, and the own-range DELETE of spec section 8 step 3. That DELETE still scans the scratch `object_location` once per batch (it has no `pack_id` index and must not gain one, because it becomes the Store's table); it is the only per-batch scan left. Build phase, one sample per case, local PostgreSQL 17.11 in Docker, 1-minute load 4.0-4.5, fixture of 1 KiB whole files (4 projects): 72,641 location rows, 73.9 MB of packs 1 batch (product bound) 349.3 -> 326.2 ms 45 -> 38 statements 92 -> 80 round trips 4 batches (16 MiB bound) 408.9 -> 336.4 ms 75 -> 53 statements 155 -> 113 round trips 290,561 location rows, 295.5 MB of packs 1 batch (product bound) 1886.1 -> 1134.3 ms 45 -> 38 statements 92 -> 80 round trips 16 batches (16 MiB bound) 2654.9 -> 1240.0 ms 195 -> 113 statements 407 -> 245 round trips Per additional batch: 10 -> 5 statements, 21 -> 11 round trips, and at 290,561 rows +51.3 -> +7.0 ms. Four times the rows and batches cost 6.5x before and 3.7x after. Limits of these numbers. The multi-batch rows were taken with the batch bound lowered to 16 MiB in an uncommitted build, because no fixture I may build has more than one 1 GiB batch; no Store with several real batches was measured. For one batch the wall time is not a demonstrated gain: a second baseline sample of the 290,561-row case read 1338.3 ms, so the baseline's own spread is as large as the difference. The statement and round-trip counts are exact. Cost: the copy set is temporary space the run holds until its session ends, about 92 to 97 bytes per retained object row (6.3 MB for 64,641 retained rows and 23.8 MB for 258,561). It is added to the reported peak temp. Test: `a_duplicate_retained_location_row_refuses_the_build` pins that a second location row for a retained object is an `Integrity` refusal before the swap, with the Store left at G and the op row naming the leftover scratch schema. It passes on the parent too (there the refusal is "compaction batch count", here "compaction copy set"), so it characterizes kept behaviour; it is not a failing-on-parent regression test. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 4, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4) pass; cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,428 -> 188,405 (delta -23) core 40,589 -> 40,589 (0) engine 90,239 -> 90,216 (-23) layerfs-persistence -23 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of parent c1c2f98198 and of this commit, recounted after the rebase onto c1c2f98198. The measurements above were taken before the rebases, on parent d19d98f138. Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/compact/build.rs | 191 +++++++++--------- .../tests/postgresql_compact.rs | 50 +++++ 2 files changed, 144 insertions(+), 97 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/build.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/build.rs index 9f15900dd3..f366c7e101 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/build.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/build.rs @@ -16,36 +16,55 @@ use layerfs_history::error::HistoryError; use layerfs_history::error::HistoryResult; use std::sync::Arc; -pub(crate) fn retained_totals( - session: &Arc, - schema: &str, -) -> HistoryResult<(i64, i64, i64)> { +/// The stored and metadata bytes of the packs the mark retained. +fn retained_pack_totals(session: &Arc, schema: &str) -> HistoryResult<(i64, i64)> { stage(session, |tx| { - let canonical = scalar( - tx, - &format!( - "SELECT COALESCE(SUM(l.canonical_length)::bigint,0) FROM {schema}.object_location l \ - WHERE EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=l.object_id)" - ), - Vec::new(), - )?; - let stored = scalar( - tx, + let rows = tx.query( &format!( - "SELECT COALESCE(SUM(p.length)::bigint,0) FROM {schema}.pack p \ + "SELECT COALESCE(SUM(p.length)::bigint,0),\ + COALESCE((SUM(p.length) FILTER (WHERE p.domain=0))::bigint,0) \ + FROM {schema}.pack p \ WHERE EXISTS (SELECT 1 FROM pg_temp.gc_live_pack v WHERE v.pack_id=p.pack_id)" ), Vec::new(), )?; - let metadata = scalar( - tx, - &format!( - "SELECT COALESCE(SUM(p.length)::bigint,0) FROM {schema}.pack p \ - WHERE p.domain=0 AND EXISTS (SELECT 1 FROM pg_temp.gc_live_pack v WHERE v.pack_id=p.pack_id)" - ), + let row = rows + .first() + .ok_or(HistoryError::Integrity("compaction retained totals"))?; + Ok((row.get(0)?, row.get(1)?)) + }) +} + +/// The copy set: every `object_location` row of a visited object, in a session +/// temporary table indexed on `pack_id`. +/// +/// The Store's `object_location` has no `pack_id` index, so reading one batch's +/// rows from it scans the whole table, and the copy would cost batches times +/// objects. This is the one scan of the source; each batch then reads its own +/// `pack_id` range through the index. Returns the set's row count, the +/// retained canonical bytes and the set's own size, which is temporary space +/// this run holds until the session ends and so part of its peak. +fn copy_set(session: &Arc, schema: &str) -> HistoryResult<(i64, i64, u64)> { + stage(session, |tx| { + tx.batch(&format!( + "DROP TABLE IF EXISTS pg_temp.gc_copy;\ + CREATE TEMP TABLE gc_copy ON COMMIT PRESERVE ROWS AS \ + SELECT l.* FROM {schema}.object_location l \ + WHERE EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=l.object_id);\ + CREATE INDEX gc_copy_pack ON pg_temp.gc_copy (pack_id);\ + ANALYZE pg_temp.gc_copy;" + ))?; + let rows = tx.query( + "SELECT count(*),COALESCE(SUM(canonical_length)::bigint,0),\ + pg_total_relation_size('pg_temp.gc_copy'::regclass)::bigint \ + FROM pg_temp.gc_copy", Vec::new(), )?; - Ok((canonical, stored, metadata)) + let row = rows + .first() + .ok_or(HistoryError::Integrity("compaction copy set"))?; + let bytes = u64::try_from(row.get::(2)?).unwrap_or(0); + Ok((row.get(0)?, row.get(1)?, bytes)) }) } @@ -183,8 +202,10 @@ fn peak_sample(session: &Arc, scratch: &str, wal_start: &str) -> Histor Ok(scratch_bytes.saturating_add(u64::try_from(wal_bytes).unwrap_or(0))) } -/// Phase A: copy retained rows in pack-id batches, each its own transaction; -/// a retried batch first deletes its own range, so the copy is idempotent. +/// Phase A: copy retained rows in pack-id batches, each its own transaction. +/// A batch first deletes its own range, so the copy is idempotent. The +/// location rows come from the copy set, so a batch reads its own rows through +/// an index instead of scanning the Store's table and the visited set. pub(crate) fn build( session: &Arc, schema: &str, @@ -222,82 +243,62 @@ pub(crate) fn build( .get(0) .map_err(Into::into) })?; + let visited = mark.objects().len() as i64; + let (copy_rows, canonical, copy_bytes) = copy_set(session, schema)?; + // One location row per visited object, or the source holds a duplicate or + // lacks a locator; the scratch table's own distinct count follows the copy. + if copy_rows != visited { + return Err(HistoryError::Integrity("compaction copy set")); + } let mut peak_temp = 0u64; + let mut copied_rows = 0i64; for batch in byte_batches(&live_packs) { let lo = batch[0].0; let hi = batch[batch.len() - 1].0; let expected_packs = batch.len() as i64; - stage(session, |tx| { + let (pack_rows, objects) = stage(session, |tx| { + // §8 step 3: a batch first deletes its own range. The scratch + // `object_location` has no `pack_id` index (the Store's has none, + // and the scratch table becomes the Store's), so this is one scan + // of the rows copied so far. tx.batch(&format!( "DELETE FROM {scratch}.pack WHERE pack_id BETWEEN {lo} AND {hi};\ DELETE FROM {scratch}.object_location WHERE pack_id BETWEEN {lo} AND {hi};" ))?; - tx.query( - &format!( - "INSERT INTO {scratch}.pack SELECT p.* FROM {schema}.pack p \ - WHERE p.pack_id BETWEEN $1 AND $2 \ - AND EXISTS (SELECT 1 FROM pg_temp.gc_live_pack v WHERE v.pack_id=p.pack_id)" - ), - vec![Param::I64(lo), Param::I64(hi)], - )?; - tx.query( - &format!( - "INSERT INTO {scratch}.object_location SELECT l.* FROM {schema}.object_location l \ - WHERE l.pack_id BETWEEN $1 AND $2 \ - AND EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=l.object_id)" - ), - vec![Param::I64(lo), Param::I64(hi)], - )?; - // Assert this batch's own pack_id range: the scratch range holds - // exactly the live packs and visited objects of the source range, - // and no object is duplicated. + // Each insert reports its own row count: this batch's range holds + // exactly the live packs and the copy set's rows of that range. let pack_rows = scalar( - tx, - &format!("SELECT count(*) FROM {scratch}.pack WHERE pack_id BETWEEN {lo} AND {hi}"), - Vec::new(), - )?; - let objects = scalar( - tx, - &format!( - "SELECT count(*) FROM {scratch}.object_location WHERE pack_id BETWEEN {lo} AND {hi}" - ), - Vec::new(), - )?; - let distinct = scalar( - tx, - &format!( - "SELECT count(DISTINCT object_id) FROM {scratch}.object_location WHERE pack_id BETWEEN {lo} AND {hi}" - ), - Vec::new(), - )?; - let source_objects = scalar( tx, &format!( - "SELECT count(*) FROM {schema}.object_location l \ - WHERE l.pack_id BETWEEN {lo} AND {hi} \ - AND EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=l.object_id)" + "WITH copied AS (INSERT INTO {scratch}.pack SELECT p.* FROM {schema}.pack p \ + WHERE p.pack_id BETWEEN $1 AND $2 \ + AND EXISTS (SELECT 1 FROM pg_temp.gc_live_pack v WHERE v.pack_id=p.pack_id) \ + RETURNING 1) SELECT count(*) FROM copied" ), - Vec::new(), + vec![Param::I64(lo), Param::I64(hi)], )?; - let source_distinct = scalar( + let objects = scalar( tx, &format!( - "SELECT count(DISTINCT l.object_id) FROM {schema}.object_location l \ - WHERE l.pack_id BETWEEN {lo} AND {hi} \ - AND EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=l.object_id)" + "WITH copied AS (INSERT INTO {scratch}.object_location \ + SELECT c.* FROM pg_temp.gc_copy c WHERE c.pack_id BETWEEN $1 AND $2 \ + RETURNING 1) SELECT count(*) FROM copied" ), - Vec::new(), + vec![Param::I64(lo), Param::I64(hi)], )?; - if pack_rows != expected_packs - || objects != source_objects - || distinct != source_distinct - || objects != distinct - { - return Err(HistoryError::Integrity("compaction batch count")); - } - Ok(()) + Ok((pack_rows, objects)) })?; - peak_temp = peak_temp.max(peak_sample(session, scratch, &wal_start)?); + if pack_rows != expected_packs { + return Err(HistoryError::Integrity("compaction batch count")); + } + copied_rows = copied_rows.saturating_add(objects); + peak_temp = + peak_temp.max(peak_sample(session, scratch, &wal_start)?.saturating_add(copy_bytes)); + } + // Every row of the copy set was copied once: a visited object in a pack + // outside the live list would be left out of every range. + if copied_rows != visited { + return Err(HistoryError::Integrity("compaction batch count")); } stage(session, |tx| { tx.batch(&format!( @@ -321,24 +322,20 @@ pub(crate) fn build( if !failures.is_empty() { return Err(HistoryError::Integrity("compaction scratch contract")); } - peak_temp = peak_temp.max(peak_sample(session, scratch, &wal_start)?); + peak_temp = + peak_temp.max(peak_sample(session, scratch, &wal_start)?.saturating_add(copy_bytes)); let (objects, distinct, pack_rows) = stage(session, |tx| { - let objects = scalar( - tx, - &format!("SELECT count(*) FROM {scratch}.object_location"), - Vec::new(), - )?; - let distinct = scalar( - tx, - &format!("SELECT count(DISTINCT object_id) FROM {scratch}.object_location"), - Vec::new(), - )?; - let pack_rows = scalar( - tx, - &format!("SELECT count(*) FROM {scratch}.pack"), + let rows = tx.query( + &format!( + "SELECT count(*),count(DISTINCT object_id),(SELECT count(*) FROM {scratch}.pack) \ + FROM {scratch}.object_location" + ), Vec::new(), )?; - Ok((objects, distinct, pack_rows)) + let row = rows + .first() + .ok_or(HistoryError::Integrity("compaction object count"))?; + Ok((row.get::(0)?, row.get::(1)?, row.get::(2)?)) })?; if distinct != mark.objects().len() as i64 || objects != distinct @@ -346,7 +343,7 @@ pub(crate) fn build( { return Err(HistoryError::Integrity("compaction object count")); } - let (canonical, stored, metadata) = retained_totals(session, schema)?; + let (stored, metadata) = retained_pack_totals(session, schema)?; let dead = DeadTotals { canonical: tuple.canonical - canonical, stored: tuple.stored - stored, diff --git a/engine/crates/layerfs-persistence/tests/postgresql_compact.rs b/engine/crates/layerfs-persistence/tests/postgresql_compact.rs index c309fe53eb..53ded67ff5 100644 --- a/engine/crates/layerfs-persistence/tests/postgresql_compact.rs +++ b/engine/crates/layerfs-persistence/tests/postgresql_compact.rs @@ -2602,3 +2602,53 @@ fn the_collected_projects_lineage_goes_with_the_project() { assert_eq!(lineage_orphans(&schema), (0, 0, 0, 0), "no orphan lineage"); clean(&schema.open()); } + +/// §8 step 3: Phase A copies exactly one location row per visited object. A +/// second row for a retained object is refused as `Integrity` before the swap: +/// the Store stays at `G` with its tables untouched, and the op row still +/// names the scratch schema the failed build left (§9). +#[test] +fn a_duplicate_retained_location_row_refuses_the_build() { + let Some(url) = url() else { return }; + let schema = Schema::new(&url, "duprow"); + let handles = schema.create(); + let live = filesystem_root(&handles, 1); + initialize(&handles, 1, live); + let dead = whole_file(&handles, b"nobody reaches this object"); + let before = generation(&handles); + let objects = schema.count("object_location"); + let packs = schema.count("pack"); + let duplicated = schema + .client() + .execute( + &format!( + "INSERT INTO {s}.object_location SELECT * FROM {s}.object_location WHERE object_id=$1", + s = schema.selection.schema + ), + &[&live.as_bytes().as_slice()], + ) + .expect("duplicate one retained location row"); + assert_eq!(duplicated, 1, "exactly one retained row is duplicated"); + + let error = handles.compact(&request(91)).unwrap_err(); + assert!( + matches!(error, layerfs_history::HistoryError::Integrity(_)), + "a duplicate location row is an Integrity refusal: {error:?}" + ); + assert_eq!(db_generation(&schema), before, "nothing was installed"); + assert!( + schema.scratch_exists(), + "the failed build leaves its scratch" + ); + assert_eq!( + schema.count_where(&format!( + "SELECT count(*) FROM {{s}}.gc_op WHERE scratch_schema='{}'", + schema.scratch() + )), + 1, + "the op row names the leftover scratch schema" + ); + assert_eq!(schema.count("object_location"), objects + 1); + assert_eq!(schema.count("pack"), packs); + assert!(located(&handles, dead), "the dead object is still stored"); +} From 11242b2ce034cb72dc4c9a12c4e795afe058d223 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:53:28 +0800 Subject: [PATCH 16/31] perf(gc): PostgreSQL answers a wave's pack heads in one statement (layerfs-project#41) The mark asks for the first 1,024 bytes of up to 256 packs per call (`Mechanics::pack_prefixes`, 962acd4f09). PostgreSQL had only the default, which sends one statement per pack. `objects::read::prefixes` answers the call with one statement: the ids as an array, joined to `pack` and returned in the array's order with each pack's declared length. A pack that is absent is `Missing`, and a head whose size disagrees with the declared length is `Malformed`, as for a single prefix. Mark walk, one sample per case, local PostgreSQL 17.11 in Docker, fixtures of 1 KiB whole files (99 per Layer, 4 projects): 72,641 location rows, 1,280 live packs (640 payload) load 5.26 -> 3.53 statements 3,850 -> 3,212 round trips 3,865 -> 3,228 pack bytes received 131,893,084 -> 131,893,084 wall 2,064.8 -> 1,846.7 ms 290,561 location rows, 5,120 live packs (2,560 payload) load 5.00 -> 3.53 statements 15,370 -> 12,820 round trips 15,385 -> 12,836 pack bytes received 527,732,824 -> 527,732,824 wall 8,430.4 -> 7,779.5 ms The counts are exact: one head statement per payload pack becomes one per 256 packs (640 -> 3 and 2,560 -> 10). The wall figures were taken at a lower host load than their baselines and are one sample each, so they are not evidence of a speed-up by themselves. The bytes do not move: the heads are 1 KiB each, and what dominates is the 256 KiB head a scoped read fetches with every descriptor. These figures were taken on parent 962acd4f09, before the mark moved its payload ranges to `pack_ranges` (c1c2f98198). Test: `a_mark_reaches_every_record_of_payload_packs_larger_than_its_head` marks a Store of four trees whose payload packs (99 records, about 102 KiB, several groups) are far larger than a head, and requires every saved object and nothing else. It passes on the parent as well; it fails when the heads come back in another order (checked by reversing the statement's ORDER BY: the closure is refused as `Integrity`). It asserts no statement count, because the mark's per-pack statements are set by `history/objects/edges.rs`, which is still changing. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,405 -> 188,451 (delta +46) core 40,589 -> 40,589 (0) engine 90,216 -> 90,262 (+46) layerfs-persistence +46 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and of this commit, recounted after the rebase onto c1c2f98198 Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/mechanics.rs | 8 ++ .../src/backend/postgresql/objects/read.rs | 41 ++++++ .../tests/postgresql_gc_mark.rs | 125 ++++++++++++++++++ 3 files changed, 174 insertions(+) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs index b69124537d..56891c772e 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs @@ -100,6 +100,14 @@ impl Mechanics for Transaction<'_> { ) -> Result)>, PersistenceError> { objects::read::prefix(self, id, limit) } + fn pack_prefixes( + &self, + ids: &[i64], + limit: usize, + out: &mut Vec<(usize, Vec)>, + ) -> Result<(), PersistenceError> { + objects::read::prefixes(self, ids, limit, out) + } fn read_scoped_pack( &self, id: i64, diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs index a34c2f1ab9..4f1506cc3b 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs @@ -181,3 +181,44 @@ pub(crate) fn prefix( } Ok(Some((length, body))) } + +/// [`prefix`] for several packs in one statement, in `ids` order. A pack that +/// is absent is an error. +pub(crate) fn prefixes( + tx: &Transaction<'_>, + ids: &[i64], + limit: usize, + out: &mut Vec<(usize, Vec)>, +) -> Result<(), PersistenceError> { + if limit == 0 || ids.iter().any(|id| *id <= 0) { + return Err(PersistenceError::Malformed); + } + if ids.is_empty() { + return Ok(()); + } + let limit = i32::try_from(limit).map_err(|_| PersistenceError::Malformed)?; + let ids = ids.to_vec(); + let rows = tx.typed( + &format!( + "SELECT p.length,substring(p.body FROM 1 FOR $2) \ + FROM unnest($1::bigint[]) WITH ORDINALITY AS i(pack_id,n) \ + JOIN {}.pack p ON p.pack_id=i.pack_id ORDER BY i.n", + tx.schema + ), + &[&ids, &limit], + ids.len() as u64 * 8 + 4, + )?; + if rows.len() != ids.len() { + return Err(PersistenceError::Missing); + } + for row in &rows { + let length: i64 = row.try_get(0).map_err(|e| rows::error(&e))?; + let body: Vec = row.try_get(1).map_err(|e| rows::error(&e))?; + let length = usize::try_from(length).map_err(|_| PersistenceError::Malformed)?; + if body.len() != length.min(limit as usize) { + return Err(PersistenceError::Malformed); + } + out.push((length, body)); + } + Ok(()) +} diff --git a/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs b/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs index 926504392c..4fac961906 100644 --- a/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs +++ b/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs @@ -384,3 +384,128 @@ fn a_mark_keeps_the_saved_objects_and_reports_the_live_heads_digest() { let (commits, _layers) = counts(&mut schema.admin()); assert_eq!(commits, 1); } + +/// One Layer's tree in one Save: `files` whole files of 1 KiB under one inode +/// leaf, its directory and its filesystem root. Returns the root and every +/// object id of the Save. +fn save_tree(handles: &PostgresHandles, seed: u32, files: u32) -> (ObjectId, Vec) { + use layerfs_content::filesystem::attributes::PortableMetadata; + use layerfs_content::filesystem::directory::{encode_directory_page, DirectoryPage}; + use layerfs_content::filesystem::inode::{encode_inode_page, InodePage}; + use layerfs_content::filesystem::root::profile_id; + use layerfs_content::filesystem::{scope_for_seed, FilesystemRoot}; + use layerfs_content::inode_leaf::{InodeKind, InodeMetadata, InodeValue}; + use layerfs_content::{encode_whole_file_payload, FinalizedObject, ObjectRole}; + + let metadata = |mode| { + InodeMetadata::Inline(PortableMetadata { + mode, + mtime_seconds: i64::from(seed), + mtime_nanoseconds: 0, + }) + }; + let mut objects: Vec = (0..files) + .map(|index| { + FinalizedObject::new( + ObjectRole::WholeFile, + encode_whole_file_payload(&bytes(seed.wrapping_mul(1000) + index, 1024)).unwrap(), + ) + .unwrap() + }) + .collect(); + let directory = FinalizedObject::new( + ObjectRole::DirectoryLeaf, + encode_directory_page(&DirectoryPage::Leaf { + entries: Vec::new(), + }) + .unwrap(), + ) + .unwrap(); + let mut entries = vec![( + 1, + InodeValue { + kind: InodeKind::Directory, + namespace_ref_count: 0, + content_root: directory.id(), + metadata_root: metadata(0o755), + }, + )]; + let mut references = vec![directory.id()]; + for (offset, file) in objects.iter().enumerate() { + entries.push(( + offset as u64 + 2, + InodeValue { + kind: InodeKind::RegularFile, + namespace_ref_count: 1, + content_root: file.id(), + metadata_root: metadata(0o644), + }, + )); + references.push(file.id()); + } + let leaf = FinalizedObject::new( + ObjectRole::InodeLeaf, + encode_inode_page(&InodePage::Leaf { entries }).unwrap(), + ) + .unwrap() + .with_references(references); + let scope = scope_for_seed([9; 32]); + let root = FilesystemRoot::new(profile_id(), scope, 1, leaf.id()).unwrap(); + let root = FinalizedObject::new(ObjectRole::FilesystemRoot, root.encode().unwrap()) + .unwrap() + .with_references(vec![leaf.id()]); + let root_id = root.id(); + objects.extend([directory, leaf, root]); + let ids: Vec = objects.iter().map(FinalizedObject::id).collect(); + let storage = Storage::new(handles.storage.clone() as Arc).unwrap(); + let save = storage.begin_save().unwrap(); + for object in objects { + save.accept(object).unwrap(); + } + save.finish().unwrap(); + (root_id, ids) +} + +/// A mark reads the heads of several packs in one call and then the record +/// headers of payload packs far larger than a head, so its ranged reads start +/// past the bytes it already holds. Every object of every tree must still be +/// reached: heads handed back in another order, or a ranged read served from +/// the wrong bytes, fail the closure. +#[test] +fn a_mark_reaches_every_record_of_payload_packs_larger_than_its_head() { + let Some(url) = url() else { return }; + let schema = Schema::new(url, "mark_ranges"); + let handles: PostgresHandles = schema.create(BINDING, CURSOR); + let (genesis_root, mut saved) = save_tree(&handles, 1, 99); + let (shop, work, _genesis) = project_with_genesis(&handles.history, 1, genesis_root); + for seed in 2..=4u32 { + let (root, ids) = save_tree(&handles, seed, 99); + commit_with(&handles.history, work, root); + publish(&handles.history, shop, work); + saved.extend(ids); + } + let large: i64 = schema + .admin() + .query_one( + "SELECT count(*) FROM pack WHERE domain<>0 AND length>4120", + &[], + ) + .unwrap() + .get(0); + assert!( + large >= 4, + "each tree has a payload pack past the head: {large}" + ); + + let mark = handles + .history + .mark() + .expect("the mark reads past the head"); + for id in &saved { + assert!(mark.objects().contains(*id), "a saved object is reached"); + } + let mut distinct = saved.clone(); + distinct.sort(); + distinct.dedup(); + assert_eq!(mark.objects().len(), distinct.len(), "and nothing else"); +} From 51944d288f9f7a5a1d97c1478e5fb784e4ccc894 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 00:53:28 +0800 Subject: [PATCH 17/31] perf(gc): PostgreSQL answers a wave's pack ranges in one statement (layerfs-project#41) The mark's payload path asks for the group directories and record headers of up to 256 packs per call (`Mechanics::pack_ranges`, c1c2f98198). PostgreSQL had only the default, one scoped acquisition per pack, and a scoped acquisition here fetches the first 256 KiB of the body with its descriptor, so every payload pack of the fixtures below still crossed the wire whole. `objects::read::ranges` answers one call with one statement: the pack ids, offsets and lengths as three arrays, joined to `pack`, one `substring` per range, returned in request order. The body is stored out of line and uncompressed, so each `substring` fetches only the chunks it covers. A pack's ranges must ascend without overlap (`Malformed` otherwise); an absent pack is `Missing`; a range that reaches past its body comes back short and is `Malformed`. Mark walk, one sample per case, local PostgreSQL 17.11 in Docker, fixtures of 1 KiB whole files (99 per Layer, 4 projects), against the parent commit: 72,641 location rows, 1,280 live packs (640 payload) load 5.54 -> 4.14 statements 3,262 -> 1,996 round trips 3,278 -> 2,013 pack bytes received 136,989,034 -> 4,611,164 wall 2,518.0 -> 1,529.4 ms 290,561 location rows, 5,120 live packs (2,560 payload) load 5.25 -> 4.20 statements 13,020 -> 7,952 round trips 13,036 -> 7,969 pack bytes received 548,116,624 -> 18,605,144 wall 10,746.3 -> 6,721.3 ms The counts are exact. The wall figures are one sample each and the baseline ran at a higher host load, so the size of the speed-up is not established; that both statements and bytes fall is. What is left of the walk is per metadata pack, three statements each (small fixture, 640 metadata packs, from the server's statement log): 640 scoped reads through the decoding reader, 640 `inode_value_block` reads and 639 single-pack prefix reads of the kept-waste pass, against 64 range statements and 3 head statements for all payload packs. Those callers are in `history/objects.rs`, outside this change. Test: `a_mark_reaches_every_record_of_payload_packs_larger_than_its_head` now runs through this statement. It fails when the ranges come back in another order (checked by reversing the ORDER BY: the closure is refused as `Integrity`). The missing-pack and short-range refusals are not reachable through a public call and have no test. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,451 -> 188,518 (delta +67) core 40,589 -> 40,589 (0) engine 90,262 -> 90,329 (+67) layerfs-persistence +67 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/mechanics.rs | 7 ++ .../src/backend/postgresql/objects/read.rs | 69 ++++++++++++++++++- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs index 56891c772e..463c79af5a 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/mechanics.rs @@ -108,6 +108,13 @@ impl Mechanics for Transaction<'_> { ) -> Result<(), PersistenceError> { objects::read::prefixes(self, ids, limit, out) } + fn pack_ranges( + &self, + requests: &[(i64, Vec)], + out: &mut Vec>>, + ) -> Result<(), PersistenceError> { + objects::read::ranges(self, requests, out) + } fn read_scoped_pack( &self, id: i64, diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs index 4f1506cc3b..8f14857511 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/objects/read.rs @@ -4,7 +4,10 @@ use crate::backend::records::{pack_info, BackendError, Param, Record}; use layerfs_storage::{ location::PackInfo, policy, - port::{AcquiredPackRead, PackReadPlan, PersistedPack, PersistedPackRead, PersistenceError}, + port::{ + AcquiredPackRead, PackRange, PackReadPlan, PersistedPack, PersistedPackRead, + PersistenceError, + }, }; fn info(r: &Record) -> Result { @@ -222,3 +225,67 @@ pub(crate) fn prefixes( } Ok(()) } + +/// The bytes of selected ranges of several packs in one statement: for each +/// `(pack, ranges)` of `requests`, in order, the bytes of each range, in +/// order. A pack's ranges are sorted and disjoint. The body is stored +/// uncompressed out of line, so each `substring` fetches only the chunks it +/// covers. A pack that is absent is `Missing`; a range that reaches past its +/// body comes back short and is `Malformed`. +pub(crate) fn ranges( + tx: &Transaction<'_>, + requests: &[(i64, Vec)], + out: &mut Vec>>, +) -> Result<(), PersistenceError> { + let total = requests.iter().map(|(_, ranges)| ranges.len()).sum(); + let mut ids = Vec::with_capacity(total); + let mut from = Vec::with_capacity(total); + let mut count = Vec::with_capacity(total); + for (id, ranges) in requests { + if *id <= 0 { + return Err(PersistenceError::Malformed); + } + let mut end = 0; + for range in ranges { + let stop = range + .offset + .checked_add(range.length) + .filter(|_| range.length > 0 && range.offset >= end) + .ok_or(PersistenceError::Malformed)?; + end = stop; + ids.push(*id); + from.push(i32::try_from(range.offset + 1).map_err(|_| PersistenceError::Malformed)?); + count.push(i32::try_from(range.length).map_err(|_| PersistenceError::Malformed)?); + } + } + let rows = if total == 0 { + Vec::new() + } else { + tx.typed( + &format!( + "SELECT substring(p.body FROM r.o FOR r.l) \ + FROM unnest($1::bigint[],$2::int4[],$3::int4[]) WITH ORDINALITY AS r(pack_id,o,l,n) \ + JOIN {}.pack p ON p.pack_id=r.pack_id ORDER BY r.n", + tx.schema + ), + &[&ids, &from, &count], + total as u64 * 16, + )? + }; + if rows.len() != total { + return Err(PersistenceError::Missing); + } + let mut rows = rows.iter(); + for (_, ranges) in requests { + let mut bodies = Vec::with_capacity(ranges.len()); + for (range, row) in ranges.iter().zip(rows.by_ref()) { + let body: Vec = row.try_get(0).map_err(|e| rows::error(&e))?; + if body.len() != range.length { + return Err(PersistenceError::Malformed); + } + bodies.push(body); + } + out.push(bodies); + } + Ok(()) +} From 0bb8989529538557318b42cc53af6a4d3463d3d3 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:03:32 +0800 Subject: [PATCH 18/31] perf(gc): PostgreSQL checks the scratch rows before the Store-wide lock (layerfs-project#41) The swap transaction took ACCESS EXCLUSIVE on every Store table and then scanned the moved object table three times (count, count DISTINCT, canonical sum) and the pack table three times, and read the live-heads digest twice. Every reader and writer of the Store waited for those scans. The row counts and sums of spec section 8 step 9 are now read in one statement from the scratch tables, inside the same transaction and before the Store-wide lock. The two scratch tables are locked first (ACCESS EXCLUSIVE, waiting), so the rows counted are the rows SET SCHEMA moves. Both sides of each comparison come from the mark: the mark's object and pack counts, and the mark's totals minus the dead totals, which the tuple re-check under the lock then pins. The digest is read once under the lock; the lock is held from that read to COMMIT. `store_totals.logical` is no longer re-read: the tuple re-check has just compared it, so the history rows are summed against the mark's value. The two `gc_op` reads are one statement. Still under the lock: the tuple and digest re-check, the ACL read and replay, the drop and move, the table fingerprint, the logical sum over the history rows, the trigger/function contract check, the relation size, both updates, the scratch drop and the standby check. Measured on a local PostgreSQL 17 (Docker, loopback), one sample each, synchronous_commit=local, lock hold timed in the client from the LOCK TABLE statement to the end of COMMIT: 72,641 object rows, 1,360 packs (1-minute load 3.26 -> 3.04) lock hold 57.4 ms -> 28.5 ms statements under it 35 -> 25 swap phase 38 statements, 64 round trips, 64.8 ms -> 30 statements, 49 round trips, 58.5 ms 290,561 object rows, 5,440 packs (1-minute load 3.08 -> 3.20) lock hold 154.4 ms -> 49.7 ms statements under it 35 -> 25 swap phase 38 statements, 64 round trips, 160.3 ms -> 30 statements, 49 round trips, 150.9 ms At the larger size the COMMIT itself is 17 ms of the remaining hold. The count DISTINCT sort (77 ms at 290,561 rows) still runs, outside the lock. Changed behaviour to know about: a count or sum mismatch on the scratch tables is now refused as Integrity before the tuple is re-read, so it is reported even when the Store also moved after the mark (before, Conflict came first). The scratch-table lock waits without NOWAIT; before this change the same wait happened at ALTER TABLE ... SET SCHEMA while the Store-wide lock was held. No test reaches the count or sum refusals (none did before); the phases run back to back inside one call and there is no hook between build and swap. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,518 -> 188,502 (delta -16) core 40,589 -> 40,589 (0) engine 90,329 -> 90,313 (-16) layerfs-persistence -16 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/compact/swap.rs | 130 +++++++++--------- 1 file changed, 63 insertions(+), 67 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/swap.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/swap.rs index 544034052f..ee5a7398e1 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/swap.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/swap.rs @@ -28,10 +28,12 @@ use layerfs_history::records::GcIntentKind; use layerfs_history::records::GcScopeKind; use std::sync::Arc; -/// Phase B: lock every base table, re-check the mark tuple, swap the two -/// payload tables, bump the generation and drop the scratch schema in one -/// short transaction. `55P03` is `Busy`; any tuple change is `Conflict` with -/// no writes. +/// Phase B: check the scratch tables' rows against the mark, then lock every +/// base table, re-check the mark tuple, swap the two payload tables, bump the +/// generation and drop the scratch schema, all in one transaction. The row +/// checks scan the scratch tables, so they run before the Store-wide lock +/// and no reader or writer waits for them. `55P03` is `Busy`; any tuple +/// change is `Conflict` with no writes. pub(crate) fn swap( _handles: &PostgresHandles, session: &Arc, @@ -55,6 +57,7 @@ pub(crate) fn swap( "SET LOCAL synchronous_commit TO {}", sync_name(rank) ))?; + verify_scratch(tx, &scratch, mark, tuple, &built.dead)?; tx.batch(&format!( "LOCK TABLE {locked} IN ACCESS EXCLUSIVE MODE NOWAIT" ))?; @@ -64,6 +67,8 @@ pub(crate) fn swap( if now != *tuple { return Err(HistoryError::Conflict("Store moved after the mark")); } + // Read once: the lock is held from here to COMMIT, so the heads this + // digest covers cannot move again inside this transaction. if digest(tx)? != mark.live_heads_digest() { return Err(HistoryError::Conflict("live heads moved after the mark")); } @@ -91,14 +96,17 @@ pub(crate) fn swap( // The mark recorded both source tables' pg_attribute/attstorage/ // reloptions/tablespace fingerprint; the moved scratch tables must // carry it exactly, or the rebuild silently changed their layout. - let recorded: String = tx - .query( - &format!("SELECT COALESCE(mark_fingerprint,'') FROM {schema}.gc_op WHERE id=1"), - Vec::new(), - )? + let op = tx.query( + &format!( + "SELECT COALESCE(mark_fingerprint,''),totals_applied FROM {schema}.gc_op WHERE id=1" + ), + Vec::new(), + )?; + let op = op .first() - .ok_or(HistoryError::Integrity("compaction fingerprint row"))? - .get(0)?; + .ok_or(HistoryError::Integrity("compaction fingerprint row"))?; + let recorded: String = op.get(0)?; + let applied: i64 = op.get(1)?; let moved = format!( "{}|{}", super::op::table_fingerprint(tx, schema, "pack")?, @@ -107,7 +115,20 @@ pub(crate) fn swap( if recorded != moved { return Err(HistoryError::Integrity("compaction table fingerprint")); } - verify_swap(tx, schema, mark, tuple, &built.dead)?; + // `store_totals.logical` equals the mark's under the lock (the tuple + // re-check above), so the history rows are summed against that value. + let logical = scalar( + tx, + &format!( + "SELECT (SELECT COALESCE(SUM(logical)::bigint,0) FROM {schema}.\"commit\")+(SELECT COALESCE(SUM(logical)::bigint,0) FROM {schema}.layer)" + ), + Vec::new(), + )?; + if logical != tuple.logical { + return Err(HistoryError::Integrity( + "Store totals disagree with the history rows", + )); + } // The trigger and immutable-pack function entries need the // Store-schema function, so they are verified here, bounded to the // moved table, after the trigger exists. @@ -117,11 +138,6 @@ pub(crate) fn swap( if !failures.is_empty() { return Err(HistoryError::Integrity("compaction moved-table contract")); } - let applied = scalar( - tx, - &format!("SELECT totals_applied FROM {schema}.gc_op WHERE id=1"), - Vec::new(), - )?; if applied != 0 { return Err(HistoryError::Integrity("compaction totals already applied")); } @@ -460,74 +476,54 @@ pub(crate) fn apply_acl( Ok(()) } -pub(crate) fn verify_swap( +/// §8 step 9's row counts and sums, read from the scratch tables inside the +/// swap transaction and before the Store-wide lock. The two tables are locked +/// first, so the rows counted here are the rows `SET SCHEMA` moves; the lock +/// waits (no `NOWAIT`) because only maintenance on the scratch schema can +/// hold it, and nothing of the Store is locked yet. Both sides of each +/// comparison come from the mark, which the tuple re-check then pins. +pub(crate) fn verify_scratch( tx: &Transaction<'_>, - schema: &str, + scratch: &str, mark: &MarkResult, tuple: &GcTuple, dead: &DeadTotals, ) -> HistoryResult<()> { - let objects = scalar( - tx, - &format!("SELECT count(*) FROM {schema}.object_location"), - Vec::new(), - )?; - let distinct = scalar( - tx, - &format!("SELECT count(DISTINCT object_id) FROM {schema}.object_location"), - Vec::new(), - )?; - let packs = scalar( - tx, - &format!("SELECT count(*) FROM {schema}.pack"), + tx.batch(&format!( + "LOCK TABLE {scratch}.pack, {scratch}.object_location IN ACCESS EXCLUSIVE MODE" + ))?; + let rows = tx.query( + &format!( + "SELECT o.objects,o.distinct_objects,o.canonical,p.packs,p.stored,p.metadata FROM \ + (SELECT count(*) AS objects,count(DISTINCT object_id) AS distinct_objects,\ + COALESCE(SUM(canonical_length)::bigint,0) AS canonical \ + FROM {scratch}.object_location) o,\ + (SELECT count(*) AS packs,COALESCE(SUM(length)::bigint,0) AS stored,\ + COALESCE(SUM(length) FILTER (WHERE domain=0)::bigint,0) AS metadata \ + FROM {scratch}.pack) p" + ), Vec::new(), )?; + let row = rows + .first() + .ok_or(HistoryError::Integrity("compaction scratch counts"))?; + let objects: i64 = row.get(0)?; + let distinct: i64 = row.get(1)?; + let canonical: i64 = row.get(2)?; + let packs: i64 = row.get(3)?; + let stored: i64 = row.get(4)?; + let metadata: i64 = row.get(5)?; if objects != mark.objects().len() as i64 || distinct != objects || packs != mark.objects().packs().len() as i64 { return Err(HistoryError::Integrity("compaction swapped count")); } - let canonical = scalar( - tx, - &format!("SELECT COALESCE(SUM(canonical_length)::bigint,0) FROM {schema}.object_location"), - Vec::new(), - )?; - let stored = scalar( - tx, - &format!("SELECT COALESCE(SUM(length)::bigint,0) FROM {schema}.pack"), - Vec::new(), - )?; - let metadata = scalar( - tx, - &format!("SELECT COALESCE(SUM(length)::bigint,0) FROM {schema}.pack WHERE domain=0"), - Vec::new(), - )?; if canonical != tuple.canonical - dead.canonical || stored != tuple.stored - dead.stored || metadata != tuple.metadata - dead.metadata { return Err(HistoryError::Integrity("compaction swapped totals")); } - let logical = scalar( - tx, - &format!("SELECT logical FROM {schema}.store_totals WHERE id=1"), - Vec::new(), - )?; - let sum = scalar( - tx, - &format!( - "SELECT (SELECT COALESCE(SUM(logical)::bigint,0) FROM {schema}.\"commit\")+(SELECT COALESCE(SUM(logical)::bigint,0) FROM {schema}.layer)" - ), - Vec::new(), - )?; - if logical != sum { - return Err(HistoryError::Integrity( - "Store totals disagree with the history rows", - )); - } - if digest(tx)? != mark.live_heads_digest() { - return Err(HistoryError::Conflict("live heads moved after the mark")); - } Ok(()) } From dc6734bdfd587ca4bca3347f636e8a5195386d73 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:19:34 +0800 Subject: [PATCH 19/31] perf(gc): PostgreSQL follows the history closure by primary key (layerfs-project#41) The mark's history closure (`kept_node_mark`) ran the text both engines share. Its recursive step joins each round's new nodes to a view of every history edge, and PostgreSQL plans that as a hash of the new nodes probed by a scan of all edges. A Commit that only its parent chain reaches arrives one round at a time, so a Branch with n unpublished Commits cost n scans of the edge set: 4,000 rounds read 5.4 million buffers (EXPLAIN ANALYZE, BUFFERS). PostgreSQL now runs its own text, registered in `backend/postgresql/history/statements.rs`. The roots and the final insert are unchanged. The recursive step reads each new node's own row by primary key and emits its references from a VALUES list. Where the shared text joined to `commit` or `layer` to learn which kind a reference names, this text emits both kinds; a name that is no row of that kind has no row to follow and the final insert's join drops it. The shared file is not edited and SQLite still runs it. Measured on a local PostgreSQL 17 (Docker, loopback), one sample each, the closure statement's server time inside a `store.compact` mark: one Branch, 999 unpublished Commits 584.9 ms -> 6.5 ms (load 5.72, 6.28) one Branch, 3,999 unpublished Commits 9,386.3 ms -> 20.8 ms (load 5.72, 6.17) mark walk of the larger Store 14,752.2 ms -> 4,486.6 ms Four times the chain cost 16 times before and 3.2 times now. It is slower where every Commit is published as a Layer, because every Layer of a live Stack is a root and the old text finished in two rounds: 2 Stacks x 1,000 Layers, 2,001 kept nodes 19.9 ms -> 258.9 ms 2 Stacks x 2,400 Layers, 4,801 kept nodes 132.4 ms -> 251.1 ms About 240 ms of each new figure is the server's JIT compiler, not the statement's work: the planner's estimate for the new text passes `jit_above_cost` at about 550 root nodes, and EXPLAIN ANALYZE shows 241 ms of JIT against 10 ms of execution. The shared text passes the same threshold on larger Stores (45 ms of its 132.4 ms above). The Store session does not set `jit`; that is in `session/profile.rs`, outside this change. Test: `the_postgresql_closure_keeps_the_nodes_the_shared_statement_keeps` builds a live project (a published Commit, a fork, a merge whose second parent is a Layer, 30 unpublished Commits) and a deleted project, and requires the same `kept_node` rows from both texts. It fails when the parent-Commit edge is removed from the new text. It does not fail when the second-parent, fork or source-Branch edge is removed: every Layer of a live Stack and every live Branch is already a root in this fixture. No fixture here has a `merged_from_id` or `displaced_id` reference. The two texts also gave the same node set (count and md5 of the sorted ids) on the four measurement Stores. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 6, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,502 -> 188,569 (delta +67) core 40,589 -> 40,589 (0) engine 90,313 -> 90,380 (+67) layerfs-persistence +67 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 --- .../queries/history/kept_node_mark.sql | 64 ++++++++++++++++++ .../backend/postgresql/history/statements.rs | 6 ++ .../tests/postgresql_gc_mark.rs | 67 ++++++++++++++++++- 3 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 engine/crates/layerfs-persistence/sql/postgresql/queries/history/kept_node_mark.sql diff --git a/engine/crates/layerfs-persistence/sql/postgresql/queries/history/kept_node_mark.sql b/engine/crates/layerfs-persistence/sql/postgresql/queries/history/kept_node_mark.sql new file mode 100644 index 0000000000..3e02322c92 --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/postgresql/queries/history/kept_node_mark.sql @@ -0,0 +1,64 @@ +WITH RECURSIVE +roots(kind,node_id) AS ( + SELECT 'stack',s.layer_stack_id FROM layer_stack s + WHERE NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id=s.layer_stack_id) + UNION + SELECT 'layer',l.layer_id FROM layer l + WHERE NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id=l.layer_stack_id) + UNION + SELECT 'branch',b.branch_id FROM branch b + WHERE NOT EXISTS (SELECT 1 FROM branch_retired r WHERE r.branch_id=b.branch_id) + AND NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id=b.layer_stack_id) +), +closure(kind,node_id) AS ( + SELECT kind,node_id FROM roots + UNION + SELECT e.to_kind,e.to_id + FROM closure k + CROSS JOIN LATERAL ( + SELECT t.to_kind,t.to_id FROM branch b + CROSS JOIN LATERAL (VALUES + ('commit',b.head_commit_id), + ('layer',b.base_layer_id), + ('commit',b.forked_from_id), + ('layer',b.forked_from_id)) t(to_kind,to_id) + WHERE k.kind='branch' AND b.branch_id=k.node_id + UNION ALL + SELECT t.to_kind,t.to_id FROM layer l + CROSS JOIN LATERAL (VALUES + ('layer',l.parent_layer_id), + ('commit',l.source_commit_id), + ('branch',l.source_branch_id)) t(to_kind,to_id) + WHERE k.kind='layer' AND l.layer_id=k.node_id + UNION ALL + SELECT t.to_kind,t.to_id FROM "commit" c + CROSS JOIN LATERAL (VALUES + ('commit',c.parent_commit_id), + ('commit',c.second_parent_id), + ('layer',c.second_parent_id), + ('layer',c.base_layer_id), + ('branch',c.branch_id), + ('commit',c.merged_from_id), + ('layer',c.merged_from_id), + ('commit',c.displaced_id), + ('layer',c.displaced_id)) t(to_kind,to_id) + WHERE k.kind='commit' AND c.commit_id=k.node_id + ) e + WHERE e.to_id IS NOT NULL +) +INSERT INTO kept_node(layer_stack_id,kind,node_id,serial) +SELECT s.layer_stack_id,'stack',s.layer_stack_id,0 + FROM closure k JOIN layer_stack s ON s.layer_stack_id=k.node_id + WHERE k.kind='stack' +UNION ALL +SELECT b.layer_stack_id,'branch',b.branch_id,0 + FROM closure k JOIN branch b ON b.branch_id=k.node_id + WHERE k.kind='branch' +UNION ALL +SELECT l.layer_stack_id,'layer',l.layer_id,l.history_serial + FROM closure k JOIN layer l ON l.layer_id=k.node_id + WHERE k.kind='layer' +UNION ALL +SELECT c.layer_stack_id,'commit',c.commit_id,c.history_serial + FROM closure k JOIN "commit" c ON c.commit_id=k.node_id + WHERE k.kind='commit' diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs index 951d9d8044..7a9be93544 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs @@ -15,6 +15,12 @@ pub(crate) fn statement(key: &str) -> Result<&'static str, BackendError> { "gc_object_locations" => Ok( "SELECT object_id,pack_id,group_number::bigint,record_number::bigint,role::bigint,canonical_length::bigint FROM object_location", ), + // The shared closure joins each round's new nodes to every history + // edge, so a chain of n unpublished Commits costs n scans of the edge + // set. This text follows each new node's own row by primary key. + "kept_node_mark" => Ok(include_str!( + "../../../../sql/postgresql/queries/history/kept_node_mark.sql" + )), "gc_epoch_read" => Ok(include_str!( "../../../../sql/postgresql/queries/history/gc_epoch_read.sql" )), diff --git a/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs b/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs index 4fac961906..8759419e3f 100644 --- a/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs +++ b/engine/crates/layerfs-persistence/tests/postgresql_gc_mark.rs @@ -17,7 +17,8 @@ use layerfs_storage::{port::PackPersistence, Storage}; use layerfs_telemetry::timer::Timing; use std::sync::Arc; use support::pg::{ - commit, commit_with, counts, project, project_with_genesis, publish, url, Schema, DEFINITION, + commit, commit_with, counts, merge_request, project, project_with_genesis, publish, url, + Schema, DEFINITION, }; const BINDING: &[u8] = b"postgresql-gc-mark"; @@ -334,6 +335,70 @@ fn a_mark_records_the_history_closure_on_postgresql() { assert_eq!(kept_rows(&mut schema.admin()), rows); } +/// The closure statement both engines shared. PostgreSQL now runs its own +/// text, which follows each node's row by primary key instead of joining each +/// round to every history edge. +const SHARED_MARK: &str = include_str!("../sql/sqlite/queries/history/kept_node_mark.sql"); + +#[test] +fn the_postgresql_closure_keeps_the_nodes_the_shared_statement_keeps() { + let Some(url) = url() else { return }; + let schema = Schema::new(url, "mark_variant"); + let handles = schema.create(BINDING, CURSOR); + // A live project with every edge kind this fixture can make: a published + // Commit, a fork, a merge whose second parent is a Layer, and a chain of + // unpublished Commits that only the Branch head reaches. + let (shop, work, genesis) = project(&handles.history, 1); + let first = commit(&handles.history, work, 5); + publish(&handles.history, shop, work); + handles + .history + .merge_branch(&merge_request( + &handles.history, + shop, + work, + first.id, + genesis, + 60, + )) + .unwrap(); + let chain: Vec<_> = (10..40) + .map(|tag| commit(&handles.history, work, tag).id) + .collect(); + // A deleted project: neither text may keep one of its rows. + let (gone, gone_work, _) = project(&handles.history, 3); + commit(&handles.history, gone_work, 7); + publish(&handles.history, gone, gone_work); + handles.history.mark_stack_deleted(gone, 2_000).unwrap(); + + let kept = handles.history.mark_history_closure().unwrap(); + for id in &chain { + assert!(kept.contains(KeptNodeId::Commit(*id)), "{id:?} is retained"); + } + let variant = kept_rows(&mut schema.admin()); + assert_eq!(variant.len(), kept.len()); + assert!(variant + .iter() + .all(|(stack, ..)| stack.as_slice() != gone.as_slice())); + + let mut admin = schema.admin(); + let mut tx = admin.transaction().unwrap(); + tx.execute("DELETE FROM kept_node", &[]).unwrap(); + tx.execute(SHARED_MARK, &[]).unwrap(); + let shared: Vec<(Vec, String, Vec, i64)> = tx + .query( + "SELECT layer_stack_id,kind,node_id,serial FROM kept_node \ + ORDER BY layer_stack_id,kind,node_id", + &[], + ) + .unwrap() + .into_iter() + .map(|row| (row.get(0), row.get(1), row.get(2), row.get(3))) + .collect(); + tx.rollback().unwrap(); + assert_eq!(variant, shared); +} + #[test] fn a_mark_keeps_the_saved_objects_and_reports_the_live_heads_digest() { let Some(url) = url() else { return }; From 305007fa5edc7aeb424b8fe6f97d66e7e99b7a15 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:25:04 +0800 Subject: [PATCH 20/31] fix(gc): PostgreSQL records a dead digest root once (layerfs-project#41) The mark's dead-set insert selected `('digest', fd.root)` for every `file_digest` row whose root is unvisited. `file_digest` is keyed by `(digest, length)`, so two rows can name one root, and `gc_dead` is keyed by `(kind, id)`: the second copy of the root broke the key and `store.compact` failed in the mark with `Integrity("persistence")`. Phase C already expects such rows (F50-2 deletes by root "even when a different digest names it"). The digest arm now selects each root once. Every other arm of the insert selects its own table's key and cannot repeat. Test: `two_digest_rows_of_one_dead_root_are_both_collected` records two digest rows for one dead root through `record_file_digests`, compacts and requires both rows gone and the op settled. Before this change it failed at the `compact` call with `Integrity("persistence")`. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 8, _compact_units 8, _gc_mark 6, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,569 -> 188,569 (delta 0) core 40,589 -> 40,589 (0) engine 90,380 -> 90,380 (0) control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/compact/op.rs | 6 ++- .../tests/postgresql_compact_fixtures.rs | 43 +++++++++++++++++++ 2 files changed, 47 insertions(+), 2 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs index dad3c92cf7..a156ec21b5 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs @@ -365,7 +365,9 @@ pub(crate) fn load_visited( /// delete each project closure in its per-project transaction instead of /// scanning the derived tables for orphans in the final settle. The /// run/depth and key/x primary keys are persisted as the concatenated bytes -/// run || int8send(depth) and key || int8send(x). +/// run || int8send(depth) and key || int8send(x). A digest root is recorded +/// once however many `file_digest` rows name it; every other arm selects its +/// table's own key. pub(crate) fn persist_mark( session: &Arc, schema: &str, @@ -394,7 +396,7 @@ pub(crate) fn persist_mark( SELECT 'signature',int8send(cs.slot)||int8send(cs.stamp) FROM {schema}.content_signature cs \ WHERE NOT EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=cs.object_id) \ UNION ALL \ - SELECT 'digest',fd.root FROM {schema}.file_digest fd \ + SELECT DISTINCT 'digest',fd.root FROM {schema}.file_digest fd \ WHERE NOT EXISTS (SELECT 1 FROM pg_temp.gc_visited v WHERE v.object_id=fd.root) \ UNION ALL \ SELECT 'layer_lineage',ll.layer_id FROM {schema}.layer_lineage ll \ diff --git a/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs b/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs index c0e1576f81..a0bb779edd 100644 --- a/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs +++ b/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs @@ -1777,6 +1777,49 @@ fn a_save_after_the_swap_survives_the_resumed_cleanup() { assert!(reopened.history.layer_stack(stack.id).unwrap().is_some()); } +/// Two `file_digest` rows can name one root (F50-2 above collects such a +/// row). When both exist at the mark and the root is dead, the persisted dead +/// set holds the root once: `gc_dead` is keyed by `(kind, id)`, and a second +/// copy of the root refused the whole mark. +#[test] +fn two_digest_rows_of_one_dead_root_are_both_collected() { + let Some(url) = url() else { return }; + let f = Fixture::new(&url, "twins"); + let live = tree(&f, whole_file(&f, b"twins live content"), 1); + let _stack = initialize(&f.handles, 1, live); + let dead = saved_whole_file(&f.handles, b"a root two digests name"); + f.handles + .history + .record_file_digests(&[ + FileDigest { + root: *dead.as_bytes(), + length: 131_072, + digest: *blake3::hash(b"twin one").as_bytes(), + }, + FileDigest { + root: *dead.as_bytes(), + length: 131_073, + digest: *blake3::hash(b"twin two").as_bytes(), + }, + ]) + .unwrap(); + let schema = &f.schema; + let twins = format!( + "SELECT count(*) FROM {{s}}.file_digest WHERE root=decode('{}','hex')", + hex(dead.as_bytes()) + ); + assert_eq!(schema.scalar(&twins), 2, "both rows exist at the mark"); + let before = generation(&f.handles); + + let receipt = f.handles.compact(&request(0x41)).unwrap(); + + assert_eq!(receipt.path, CompactPath::Rebuild); + assert_eq!(receipt.generation_after, before + 1); + assert!(!located(&f.handles, dead)); + assert_eq!(schema.scalar(&twins), 0, "both rows went with their root"); + assert_phase_c_settled(schema, 0x41); +} + /// AC-21, AC-24, AC-39: the PostgreSQL `project_epoch` fence. Commits and /// Layers leave it alone; a marker write bumps it once; a repeated marker is /// `UpToDate` and does not bump it; compact copies it; a missing row refuses From 86f5a4a5a5a63ee066f12373e3b80d65d2fde183 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:28:35 +0800 Subject: [PATCH 21/31] perf(gc): PostgreSQL deletes dead digest rows by primary key (layerfs-project#41) Phase C deleted `file_digest` rows whose root is in the dead set 1,000 at a time. Each batch joined `gc_dead` to `file_digest` on `root`, which has no index, once to pick the batch and once to delete it: two scans of the whole table per 1,000 rows. Four times the dead rows cost 17.8 times the server time (EXPLAIN shows a hash join over a sequential scan of `file_digest` in both halves). One scan now lists the keys of the rows to delete in a numbered session table. Each batch deletes its own range of that list by `(digest, length)`, the table's primary key, and checks the root against `gc_dead` again, so membership in the persisted dead set is still the filter. Batches are still 1,000 rows, one transaction each. A crash loses the session table; the resumed call lists what is left. Measured on a local PostgreSQL 17 (Docker, loopback), one sample each, synchronous_commit=local. Server time of the digest statements, and the whole cleanup phase: dead rows digest statements cleanup phase load 50,000 710.0 ms -> 201.2 ms 1,078.3 -> 483.7 ms 4.42, 7.29 200,000 12,622.2 ms -> 2,443.5 ms 14,284.4 -> 3,984.1 ms 4.42, 7.29 800,000 not run -> 7,965.7 ms not run -> 11,468.6 ms 6.68 Statements and round trips are nearly unchanged (200,000 rows: 227 -> 229 statements, 672 -> 677 round trips); the gain is in what each batch reads. Per batch: 3.4 ms at 50,000 rows, 11.4 ms at 200,000, 9.3 ms at 800,000. Up to 200,000 rows the planner still answers a batch with one scan of `file_digest` (cheaper by its own cost model than 1,000 index probes), so the cost per batch grows until the table is large enough that it switches to the primary key; from there it is level. The old text at 800,000 rows was not run (its measured slope puts it far past the two-minute limit). What is left of the phase at 200,000 rows: 1,175 ms in 211 COMMITs, one per batch. Larger batches would cut that; the batch size is unchanged here. The session table holds one row per dead digest row (about 70 bytes plus its index) until the last batch. It is not counted in `peakTempBytes`, which the swap records before Phase C starts. A `file_digest` row with a dead root that is inserted after the list is made is now left in place; the old loop would have deleted it in a later batch. Such a row can only come from a save at the new generation. Tests: `dead_digest_rows_beyond_one_batch_are_all_collected` records 2,500 dead digest rows and one retained row, compacts and requires only the retained row left; it fails when a batch drops its last row (checked with `b.n < $2`). The existing F50-2 case (a dead-root row inserted before a resumed cleanup) passes unchanged. No test deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 9, _compact_units 8, _gc_mark 6, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. The timings were taken on this cleanup.rs one commit earlier (before the dead-root dedup); the file is byte-identical. Production LOC: 188,569 -> 188,590 (delta +21) core 40,589 -> 40,589 (0) engine 90,380 -> 90,401 (+21) layerfs-persistence +21 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/compact/cleanup.rs | 57 ++++++++++++++----- .../tests/postgresql_compact_fixtures.rs | 43 ++++++++++++++ 2 files changed, 85 insertions(+), 15 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/cleanup.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/cleanup.rs index 6ab9d893c4..a7929e599c 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/cleanup.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/cleanup.rs @@ -544,29 +544,56 @@ fn delete_dead_signatures( } } +/// Rows one `file_digest` delete transaction removes. +const DIGEST_BATCH: i64 = 1000; + /// Deletes the `file_digest` rows whose root object is in the persisted dead /// set, in bounded batches. The dead set holds roots, not digests, so a row -/// with a dead root is removed even when a different digest names it. The -/// `gc_dead` rows stay until [`settle`], so a crash here resumes them. +/// with a dead root is removed even when a different digest names it. +/// `file_digest` has no index on `root`, so one scan first lists the keys of +/// the rows to delete in a numbered session table; each batch then deletes +/// its own key range by primary key and checks the root against `gc_dead` +/// again. The `gc_dead` rows stay until [`settle`], so a crash here resumes +/// them: the next run lists what is left. fn delete_dead_digests(session: &Arc, schema: &str) -> HistoryResult<()> { - loop { - let deleted = stage(session, |tx| -> HistoryResult { - let rows = tx.query( + let total = stage(session, |tx| -> HistoryResult { + tx.batch(&format!( + "DROP TABLE IF EXISTS pg_temp.gc_dead_digest;\ + CREATE TEMP TABLE gc_dead_digest ON COMMIT PRESERVE ROWS AS \ + SELECT row_number() OVER () AS n,fd.digest,fd.length \ + FROM {schema}.file_digest fd \ + WHERE EXISTS (SELECT 1 FROM {schema}.gc_dead d \ + WHERE d.kind='digest' AND d.id=fd.root);\ + CREATE UNIQUE INDEX gc_dead_digest_n ON pg_temp.gc_dead_digest (n);\ + ANALYZE pg_temp.gc_dead_digest;" + ))?; + scalar( + tx, + "SELECT count(*) FROM pg_temp.gc_dead_digest", + Vec::new(), + ) + })?; + let mut done = 0; + while done < total { + stage(session, |tx| { + tx.query( &format!( - "WITH batch AS (SELECT d.id FROM {schema}.gc_dead d \ - JOIN {schema}.file_digest fd ON fd.root=d.id \ - WHERE d.kind='digest' ORDER BY d.id LIMIT 1000) \ - DELETE FROM {schema}.file_digest fd USING batch b \ - WHERE fd.root=b.id RETURNING fd.digest" + "DELETE FROM {schema}.file_digest fd USING pg_temp.gc_dead_digest b \ + WHERE b.n > $1 AND b.n <= $2 \ + AND fd.digest=b.digest AND fd.length=b.length \ + AND EXISTS (SELECT 1 FROM {schema}.gc_dead d \ + WHERE d.kind='digest' AND d.id=fd.root)" ), - Vec::new(), + vec![Param::I64(done), Param::I64(done + DIGEST_BATCH)], )?; - Ok(rows.len()) + Ok(()) })?; - if deleted == 0 { - return Ok(()); - } + done += DIGEST_BATCH; } + stage(session, |tx| { + tx.batch("DROP TABLE IF EXISTS pg_temp.gc_dead_digest")?; + Ok(()) + }) } /// The final Phase C transaction: clear the Store-global advisory hint block, diff --git a/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs b/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs index a0bb779edd..dbf7faca07 100644 --- a/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs +++ b/engine/crates/layerfs-persistence/tests/postgresql_compact_fixtures.rs @@ -1820,6 +1820,49 @@ fn two_digest_rows_of_one_dead_root_are_both_collected() { assert_phase_c_settled(schema, 0x41); } +/// Phase C deletes dead digest rows in batches of 1,000 by a numbered key +/// list. More dead rows than two batches hold, with one retained row among +/// them: every dead row goes, across the batch boundaries, and the retained +/// row stays. +#[test] +fn dead_digest_rows_beyond_one_batch_are_all_collected() { + let Some(url) = url() else { return }; + let f = Fixture::new(&url, "digests"); + let content = whole_file(&f, b"digests live content"); + let live = tree(&f, content, 1); + let _stack = initialize(&f.handles, 1, live); + let kept = *blake3::hash(b"a digest of retained content").as_bytes(); + let mut rows = vec![FileDigest { + root: *content.as_bytes(), + length: 131_072, + digest: kept, + }]; + // Roots no object carries: unvisited, so each row is a dead-set member. + rows.extend((0..2_500u32).map(|index| FileDigest { + root: *blake3::hash(&[b"dead root".as_slice(), &index.to_le_bytes()].concat()).as_bytes(), + length: 131_072 + u64::from(index), + digest: + *blake3::hash(&[b"dead digest".as_slice(), &index.to_le_bytes()].concat()).as_bytes(), + })); + f.handles.history.record_file_digests(&rows).unwrap(); + let _dead = saved_whole_file(&f.handles, b"a pack the rebuild drops"); + let schema = &f.schema; + assert_eq!(schema.count("file_digest"), 2_501); + let before = generation(&f.handles); + + let receipt = f.handles.compact(&request(0x42)).unwrap(); + + assert_eq!(receipt.path, CompactPath::Rebuild); + assert_eq!(receipt.generation_after, before + 1); + assert_eq!( + schema.count("file_digest"), + 1, + "only the retained row is left" + ); + assert_eq!(digest_rows(schema, &kept), 1); + assert_phase_c_settled(schema, 0x42); +} + /// AC-21, AC-24, AC-39: the PostgreSQL `project_epoch` fence. Commits and /// Layers leave it alone; a marker write bumps it once; a repeated marker is /// `UpToDate` and does not bump it; compact copies it; a missing row refuses From 9dcecc8be7eb3ccf6dd7a70897c97d8dd2dbdccb Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:31:47 +0800 Subject: [PATCH 22/31] perf(gc): PostgreSQL loads the visited set without a second copy (layerfs-project#41) `load_visited` first copied every visited object id into a `Vec>` (one heap allocation per object) and the live pack ids into a second `Vec`, then streamed those to COPY. The copy lived until the load finished, beside the mark's own set. Each COPY row is now written straight from the mark's ids. The statements, their order and the rows sent are unchanged. Not measured: the saving is one 24-byte `Vec` header and one 32-byte heap block per visited object (arithmetic, before allocator overhead), and the fixtures at hand are too small to show it in peak memory. The change is confirmed by reading and by the suites below, not by a memory measurement. No test added or deleted. Checks before the rebase onto c15826e291 (this change on c1c2f98198): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 9, _compact_units 8, _gc_mark 6, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,590 -> 188,584 (delta -6) core 40,589 -> 40,589 (0) engine 90,401 -> 90,395 (-6) layerfs-persistence -6 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/backend/postgresql/compact/op.rs | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs index a156ec21b5..03a860db26 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/compact/op.rs @@ -311,13 +311,6 @@ pub(crate) fn load_visited( schema: &str, mark: &MarkResult, ) -> HistoryResult<()> { - let objects: Vec> = mark - .objects() - .objects() - .iter() - .map(|id| id.as_bytes().to_vec()) - .collect(); - let packs: Vec = mark.objects().packs().to_vec(); stage(session, |tx| { tx.batch( "DROP TABLE IF EXISTS pg_temp.gc_visited;\ @@ -327,14 +320,15 @@ pub(crate) fn load_visited( )?; // A large visited set is a bulk load, not a parameter array: COPY // streams it without one round trip per row and without SQL text that - // grows with the mark. The ids were bound in memory for the array form; - // here each row is written straight from the mark. + // grows with the mark. Each row is written straight from the mark's + // own ids, so the load holds no second copy of the set. tx.copy_in( "COPY pg_temp.gc_visited(object_id) FROM STDIN (FORMAT binary)", &[Type::BYTEA], |writer| { - for id in &objects { - writer.write(&[id])?; + for id in mark.objects().objects() { + let id: &[u8] = id.as_bytes(); + writer.write(&[&id])?; } Ok(()) }, @@ -343,7 +337,7 @@ pub(crate) fn load_visited( "COPY pg_temp.gc_live_pack(pack_id) FROM STDIN (FORMAT binary)", &[Type::INT8], |writer| { - for pack in &packs { + for pack in mark.objects().packs() { writer.write(&[pack])?; } Ok(()) From 9da60389e6bdcbfb43b5e2949fb1871ec700c804 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:42:41 +0800 Subject: [PATCH 23/31] perf(gc): kept-waste reads pack directory prefixes a batch per call The kept-waste pass reads the directory prefix of every retained pack the walk did not lay out itself: the metadata packs, which the walk reaches through the decoding reader. It asked for them one pack per call: on SQLite a length statement and a BLOB handle each, on PostgreSQL one statement and round trip each (2,399 of the 7,219 statements of a mark over 5,440 packs in the PostgreSQL agent's count). `edges::Prefixes` hands the same prefixes out in pack order and reads them through `Mechanics::pack_prefixes`, 256 packs per call: one BLOB handle reused on SQLite, one statement on PostgreSQL. gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs, 96 metadata packs), release, warm second mark, one sample, load 3.8: statements of a mark 298 -> 203 BLOB handles opened 229 -> 135 wall 30.2 ms (31 ms mean before, under load 7.6) Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30) pass; Clippy, fmt and the product boundary guard clean. Production LOC: 188,488 -> 188,527 (delta +39) core 40,589 -> 40,589 (0) engine 90,299 -> 90,338 (+39) layerfs-persistence +39 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent d60042965f and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../src/history/objects.rs | 19 ++++---- .../src/history/objects/edges.rs | 48 ++++++++++++++++++- 2 files changed, 58 insertions(+), 9 deletions(-) diff --git a/engine/crates/layerfs-persistence/src/history/objects.rs b/engine/crates/layerfs-persistence/src/history/objects.rs index b4dfadd207..e723cdeff9 100644 --- a/engine/crates/layerfs-persistence/src/history/objects.rs +++ b/engine/crates/layerfs-persistence/src/history/objects.rs @@ -34,7 +34,7 @@ use layerfs_storage::{ }, location::{ObjectLocation, SignatureRow}, pack::layout::{self, GroupCodec}, - port::{AcquiredPackRead, PACK_READ_PREFIX_BYTES}, + port::AcquiredPackRead, source::Source, StorageCapacities, StorageError, }; @@ -669,7 +669,15 @@ fn waste( closure: &Closure, ) -> HistoryResult { let mut dead_in_retained_packs_bytes = 0u64; - for pack_id in closure.packs() { + let packs = closure.packs(); + // Packs the walk did not lay out have their directory prefix read here: + // a bounded batch of packs per call, not one call per pack. + let mut prefixes = { + let layouts = source.layouts.borrow(); + let unread = packs.iter().filter(|pack| !layouts.contains_key(pack)); + edges::Prefixes::new(source, unread.copied().collect()) + }; + for pack_id in packs { let groups = located.inventory.get(&pack_id); // The walk already read this pack's directory when it read its // record headers; only a pack it reached another way is read here. @@ -680,12 +688,7 @@ fn waste( // The pack row may reserve more capacity than it uses; the header names // the used length. Everything below parses against `used`, never the // reserved capacity. A prefix that already covers `used` is the body. - let Some((capacity, prefix)) = source - .pack_prefix(pack_id, PACK_READ_PREFIX_BYTES) - .map_err(storage)? - else { - return Err(HistoryError::Integrity("mark pack directory")); - }; + let (capacity, prefix) = prefixes.take(pack_id)?; if prefix.len() < layout::USED_OFFSET + 4 { return Err(HistoryError::Integrity("mark pack length")); } diff --git a/engine/crates/layerfs-persistence/src/history/objects/edges.rs b/engine/crates/layerfs-persistence/src/history/objects/edges.rs index 39401c203e..00ab566351 100644 --- a/engine/crates/layerfs-persistence/src/history/objects/edges.rs +++ b/engine/crates/layerfs-persistence/src/history/objects/edges.rs @@ -25,7 +25,7 @@ use layerfs_storage::{ source::Source, StorageError, }; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, VecDeque}; /// After the head: a wider directory, then record headers. No pack needs more. const ROUNDS: usize = 4; @@ -422,3 +422,49 @@ pub(super) fn wave_edges( } Ok(edges) } + +/// The directory prefixes of the packs a kept-waste pass still has to read, +/// handed out in pack order and read a bounded batch of packs per call. +pub(super) struct Prefixes<'a, 'b> { + source: &'a ClosureSource<'b>, + packs: Vec, + next: usize, + held: VecDeque<(usize, Vec)>, +} + +impl<'a, 'b> Prefixes<'a, 'b> { + /// `packs` in the order [`Self::take`] will be asked for them. + pub(super) fn new(source: &'a ClosureSource<'b>, packs: Vec) -> Self { + Self { + source, + packs, + next: 0, + held: VecDeque::new(), + } + } + + /// The declared length and directory prefix of `pack_id`, which must be + /// the next pack of the list. + pub(super) fn take(&mut self, pack_id: i64) -> HistoryResult<(usize, Vec)> { + if self.packs.get(self.next) != Some(&pack_id) { + return Err(HistoryError::Integrity("mark pack directory")); + } + if self.held.is_empty() { + let batch = &self.packs[self.next..(self.next + HEAD_PACKS).min(self.packs.len())]; + let mut read = Vec::with_capacity(batch.len()); + self.source + .mechanics + .pack_prefixes(batch, PACK_READ_PREFIX_BYTES, &mut read) + .map_err(StorageError::from) + .map_err(storage)?; + if read.len() != batch.len() { + return Err(HistoryError::Integrity("mark pack directory")); + } + self.held = read.into(); + } + self.next += 1; + self.held + .pop_front() + .ok_or(HistoryError::Integrity("mark pack directory")) + } +} From 58aa0628774137a8d97315895d608cb01af10131 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 01:56:31 +0800 Subject: [PATCH 24/31] docs(gc): record the PostgreSQL swap order as built The review of the PostgreSQL rebuild commits found two texts that no longer described the code: - Spec section 8, Phase B: the swap now locks the two scratch tables and takes the row counts and sums before the Store-wide lock (0bb8989529). An "as built" note says so, including that a scratch mismatch is reported before a Conflict. - `backend/postgresql/history/statements.rs`: the module comment said only statements with SQLite-only syntax are replaced. The history closure is replaced too; the comment now says why and that both files must follow the same edges. No behaviour change. Production LOC: 188,683 -> 188,683 (delta 0) core 40,589 -> 40,589 (0) engine 90,494 -> 90,494 (0) control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent f22faa77fe and the staged tree Co-Authored-By: Claude Fable 5.2 --- core/docs/issues/project-41/LayerFS-GC-spec.md | 11 +++++++++++ .../src/backend/postgresql/history/statements.rs | 6 +++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/core/docs/issues/project-41/LayerFS-GC-spec.md b/core/docs/issues/project-41/LayerFS-GC-spec.md index 7be922c696..9f1f2f2397 100644 --- a/core/docs/issues/project-41/LayerFS-GC-spec.md +++ b/core/docs/issues/project-41/LayerFS-GC-spec.md @@ -834,6 +834,17 @@ reported as `markBytes`; the Phase A table copy itself moves no pack bytes. **Phase B — swap (one short transaction).** +> **As built (2026-11-18, `0bb8989529`).** The row counts and sums of step 9 +> are taken at the start of this transaction, before step 5: the transaction +> first takes `ACCESS EXCLUSIVE` on the two scratch tables (a waiting lock, +> not `NOWAIT`; nothing else writes them) and counts and sums them in one +> statement, then takes the Store-wide lock of step 5. The scratch lock is +> held to COMMIT, so the rows counted are the rows moved, and the Store-wide +> lock is no longer held across those scans. A count or sum mismatch is +> therefore reported (`Integrity`) before the tuple re-check of step 6 can +> report `Conflict`. The fingerprint, trigger and `logical` checks and the +> single `live_heads_digest` read stay under the Store-wide lock. + 5. `LOCK TABLE` every base table of the Store schema (from `pg_class`, including `store_lane`, `store_totals`, `history_meta`, `content_signature`, the history tables and `gc_intent`/`gc_op`) in one diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs index 7a9be93544..38fb642912 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/history/statements.rs @@ -1,6 +1,10 @@ //! History statement texts on PostgreSQL. The shipped SQLite statements are //! portable SQL (positional `$n` parameters, `RETURNING`, `ON CONFLICT`) and -//! run unchanged; only those using SQLite-only syntax are replaced here. +//! run unchanged. Those using SQLite-only syntax are replaced here, and so +//! is the history closure (`kept_node_mark`), whose shared text PostgreSQL +//! plans as one scan of every edge per recursion round. The replacement +//! must follow exactly the edges the shared text follows: add an edge to +//! both files or to neither. //! Array forms bind whole columns, so many rows cost one round trip. use crate::backend::records::BackendError; From e180a41970332fe6dba7e4dfd581933fae7fa535 Mon Sep 17 00:00:00 2001 From: YifanXu1999 Date: Wed, 18 Nov 2026 02:11:18 +0800 Subject: [PATCH 25/31] feat(gc): pg_gc_profile, a counted mark or compaction on PostgreSQL `gc_bench` and `gc_profile` are SQLite only, so PostgreSQL GC had no committed way to repeat a measurement. `pg_gc_profile` is the probe the PostgreSQL rebuild agent measured its commits with, cleaned up: pg_gc_profile build DB a Store in its own database pg_gc_profile mark DB WORK clone, run the mark, count its SQL work pg_gc_profile run DB WORK clone, run store.compact, count it pg_gc_profile seed|sizes|drop It prints wall time, the compaction's phases and the session's statements, round trips, rows and bound bytes. A diagnostic: no receipt and no claim. Documented in tools/ci/gc-bench.md. First use, PostgreSQL 17 in Docker on loopback, release, one sample each, on a Store of 72,641 objects in 1,360 packs (4 Stacks x 160 Layers x 99 files of 1 KiB, 8,000 dead objects): 6bca86167f 58aa062877 (parent) mark 3,875.3 ms 1,727.2 ms mark statements 4,489 1,359 store.compact 5,574.0 ms 2,936.0 ms compact statements 4,627 1,484 1-min load 4.2 to 4.3 7.3 An example and a doc only: no product source changes. Production LOC: 188,683 -> 188,683 (delta 0) core 40,589 -> 40,589 (0) engine 90,494 -> 90,494 (0) control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) Method: tools/production_loc.py --detail on parent 58aa062877 and the staged tree Co-Authored-By: Claude Fable 5.2 --- .../examples/pg_gc_profile.rs | 663 ++++++++++++++++++ tools/ci/gc-bench.md | 26 + 2 files changed, 689 insertions(+) create mode 100644 engine/crates/layerfs-persistence/examples/pg_gc_profile.rs diff --git a/engine/crates/layerfs-persistence/examples/pg_gc_profile.rs b/engine/crates/layerfs-persistence/examples/pg_gc_profile.rs new file mode 100644 index 0000000000..394913576e --- /dev/null +++ b/engine/crates/layerfs-persistence/examples/pg_gc_profile.rs @@ -0,0 +1,663 @@ +//! A PostgreSQL garbage-collection fixture and one counted `store.compact` +//! or mark over it: the PostgreSQL counterpart of `gc_bench` and +//! `gc_profile`, which are SQLite only. A diagnostic, not a measurement +//! harness: it writes no receipt and claims nothing. +//! +//! ```text +//! LAYERFS_TEST_POSTGRES_URL="host=... dbname=postgres ..." \ +//! pg_gc_profile build # creates database with one Store +//! pg_gc_profile run