sexp-store

sexp-store persists portable Common Lisp data as readable top-level forms. It provides atomic single-form snapshots and crash-tolerant append logs.

Snapshots

(sexp-store:snapshot-write #P"state.sexp" '(:state :version 1))

(multiple-value-bind (form sole-form-p)
    (sexp-store:snapshot-read #P"state.sexp")
  (values form sole-form-p))

Supply :grammar with a sexp-config source grammar and :maximum-octets for bounded data-only snapshot reads. That mode requires exactly one form and wraps grammar diagnostics in store-error. snapshot-read-record forwards both options along with the complete record-check schema.

Snapshot publication writes a complete temporary file beside the target and renames it over the target. The default Unix mode is 0600 on supported POSIX environments. Pass :mode nil when permissions are managed separately.

Use snapshot-write-text for exact UTF-8 text and snapshot-write-octets for an unsigned-byte vector. Both use exclusively created staging files beside the target, private unless :mode nil, and accept :require-absent. Create-only writes signal publication-conflict if any target entry already exists. Replacement writes require caller-side writer serialization, as with snapshot-write.

Append logs

(sexp-store:log-append
 #P"events.sexp"
 '(:event :id 1)
 :initial-forms '((:events :version 1)))

• log-read returns the complete forms and a second value for an incomplete final form
• log-append repairs such a tail atomically before publishing the new form
• A malformed complete form signals store-error
• log-map visits one complete form at a time and returns a file position that a later call may use as :start-position
• log-write-streaming publishes a whole log atomically from forms a producer emits one at a time, so rebuilding a large log never holds it in memory
• A form tagged :sexp-store-batch is a transaction batch, the one form a store transaction appends for several records. Readers visit its records in order, and a batch cut short by a crash shows none of them. The tag is reserved for this library.

Applications that have already verified the tail may pass :repair-tail-p nil for constant-time appends to a large log. Request repair again after loading the log following an interrupted write.

All reads bind *read-eval* to nil. Logs are for one serialized writer and any number of readers.

Property records

Validate tagged forms shaped (TAG . PROPERTIES) with record-check. Supply :versions for versioned records shaped (TAG :version VERSION . PROPERTIES). Field descriptions replace hand-written validators:

(sexp-store:snapshot-write
 #P"job.sexp"
 (sexp-store:make-record :job 2 :name "sync" :retries 3))

(sexp-store:snapshot-read-record
 #P"job.sexp"
 :tag :job
 :versions '(1 2)
 :fields (list (list :indicator :name
                     :validate (lambda (value)
                                 (and (stringp value)
                                      (plusp (length value))))
                     :required t)
               (list :indicator :retries
                     :validate #'integerp
                     :required '(2))))

• :required lists the versions that must carry the property, or t for all
• :validate is applied to present values only
• :allow-other-keys nil rejects undescribed fields; :version is implicitly allowed only when :versions is supplied
• record-check validates an in-memory form and returns a reason on failure
• Omit :versions to validate unversioned records. An explicit :versions nil accepts no version.
• Pass :properties-p t to validate a property list directly, without a tag.
• :keyword-keys-p t requires keyword indicators; :maximum-length bounds plist elements, counting both indicators and values.
• record-version, record-property, and record-property-present-p read validated forms without getf pitfalls on malformed input

For a plist alone, plist-schema-problem returns a problem keyword and the field involved, or two nil values. Supply :allowed-keys, :required-keys, :keyword-keys-p, :maximum-length and :allow-duplicate-keys as needed. An omitted allowed-key list permits any key; an explicit empty list permits none. plist-schema-p returns the corresponding boolean. With a length bound, validation stops at that bound before scanning the rest of the spine.

Transactions

Construct a log-store or snapshot-store with explicit :pathname, :lock-pathname, an :initial-state factory and a :validator predicate. Use store-read for validated fresh state, or store-transact for one locked read/modify/publish operation. Locks exclude local threads and cooperating processes through ls-flock.

For a log-store, supply :header, :header-validator, and a :reducer of state and record. An optional :finalizer converts folded state to its public value. For a snapshot-store, supply :decoder and :encoder callbacks. The updater receives fresh public state and returns changes, a result, and whether to write. Log changes are a list of records; snapshot changes are replacement state. A log batch is validated, then appended: one record as itself, several as one batch form, so the log grows by the change alone and a crash leaves all of it or none of it. Only a log with an incomplete final form is rewritten, once, which repairs it. A transaction reads the file once; the updater’s state is folded from a copy of what was read. Existing empty or malformed files are errors, rather than missing stores.

Call store-compact to rewrite a log once it has grown. Give the log-store a :compactor, a function of folded state returning the fewest records that fold to it, and the log becomes its header followed by those records. Without one, compaction keeps every complete record, flattening batches and dropping an incomplete final form. The records are validated and folded first, and the log is replaced atomically. It returns the record count and committed state.

A process that reads one log often keeps a log-follower, made by log-follower-create with a :copier, a function returning a copy of folded state that the reducer may change. log-follower-read folds only the records appended since the follower last read, and log-follower-transact works like store-transact but appends and folds just its change. A follower never changes a state it has returned, so callers may keep one while it moves on. A log that was replaced, as by compaction, shortened or removed is folded again from its start. A log with an incomplete final form, or no log yet, takes the store-transact path, which repairs or creates it. Callers serialize their own use of one follower; the store’s lock serializes processes.

A snapshot-store may supply :recover-read-error, a function of store-error that returns fresh fallback state. Recovery during a read or read-only transaction leaves the original bytes untouched. A writing transaction validates its replacement and overwrites the unreadable snapshot. Without this callback, read failures propagate. Recovery callback errors propagate unchanged. The callback may run separately for the committed base and the updater’s private working state within a transaction; return a fresh independent value each time.

(sexp-store:store-transact
 store
 (lambda (current)
   (declare (ignore current))
   (values (list '(:item :version 1 :id "a" :value 42)) :updated t))
 :publish (lambda (committed) (setf cached-state committed)))

The optional :publish callback runs after successful persistence. Keep working state private until then. A publication-callback failure occurs after commit and cannot roll back the file. Return values are the updater result and committed public state. A read-only transaction also returns fresh state.

record-finite-p validates acyclic portable data while allowing shared substructure. record-shape-p additionally requires a keyword-tagged property record. Stores reject duplicate keys by default; :duplicate-keys :first enables explicit first-value compatibility. record-check accepts :allow-duplicate-keys t for the same policy. Domain callbacks own field and version validation.

Segmented logs

Use segment-map for one header-prefixed segment and segments-map for an ordered list of physical paths. Supply :header-function, which receives a pathname and header and returns the positive start sequence and whether an empty segment is allowed. Supply :record-sequence to extract each record’s sequence. Optional :validate-record and :validate-first-record callbacks express the record schema and checkpoint policy.

Replay requires contiguous sequences within and between segments. Torn final forms are ignored, including in sealed segments; complete malformed forms are errors. segment-map returns byte position, tail flag, record count, start sequence and next sequence. segments-map returns active tail flag, total count and next sequence. Callback conditions propagate unchanged.

segment-publish atomically creates a header and first record. Supply :lock-pathname or hold an external writer lease. An optional zero-argument :occupied-p predicate includes related product storage in the occupancy check. A preexisting segment is a conflict, even when identical. If publication signals after creating the file, exact readback permits recovery. Adopt active state only after return. log-repair-tail removes only an incomplete final form under the writer lock and returns whether replacement was needed.

Create-only publication uses an atomic hard link on the supported SBCL POSIX backend, rejecting any occupied target entry, including dangling symlinks. Replacements use same-directory rename. Operations flush Lisp output and support process-crash recovery; power-loss persistence through filesystem synchronization barriers is outside this contract.

Derived snapshots

Use revision-write before authoritative source mutations under the writer lock. Hold that same lock across cache reconstruction and publication. Otherwise a reader can stamp old source data with the writer’s new revision while the writer is between invalidation and mutation. revision-read returns zero for absent or malformed revisions. Supply the product’s tag and optional version to both calls.

Use sidecar-rebuild with :lock-pathname naming the source writer lock and a zero-argument :build callback that reads source data and its token under that lock. Supply :encode, :source-token and :value-token as for sidecar-write. The lock covers reconstruction through publication. A nil build result skips publication; lock, build and publication errors propagate.

sidecar-read accepts :decode, a zero-argument :source-token callback and a :value-token extractor. Tokens are compared with equal before and after reading. Malformed, stale or concurrently invalidated cache data returns nil. Optional :validate adds domain constraints. Under the shared-lock protocol, cache hits need no lock. sidecar-write accepts a prebuilt value with :encode and the same token callbacks, declining stale publication. Hold the source lock from before constructing that value until publication completes, or build from an immutable committed snapshot with its matching token.

file-revision returns pathname, byte size and write date. Combine these with an explicit revision and source locking, since size and date cannot distinguish same-size edits within one clock tick.

Multi-file publication

files-publish publishes several files as one transaction:

(sexp-store:files-publish
 (list (list :pathname #P"state/a.sexp" :octets new-a :expected old-a)
       (list :pathname #P"state/b.sexp" :octets new-b :expected nil))
 :check (lambda (pathname) (refuse-unless-inside-root pathname)))

Every replacement and a copy of every previous content are staged beside their targets first. Each target is compared with its :expected octets, or must be absent when that is nil, immediately before it is published, and a changed target signals publication-conflict. On any failure the targets already published are restored or removed and new directories are removed. A target that changed again before it could be restored is left alone with its copy kept, and publication-rollback-failed lists those copies.

:check sees every target before staging, publishing and restoring. New files are created with :mode (default 0600) and new directories with :directory-mode (default 0700).

Development

./script/bootstrap
./script/check
./script/check --fix

./script/bootstrap installs the dependencies the qlfile pins into .qlot/. ./script/check runs the tests, then checks the source with cheque as cheque.sexp configures it. --fix skips the tests and repairs what cheque can repair. cheque and its tools are development dependencies: checks/qlfile lists them, and the check installs them on first use into ~/.cache/cheque/tools/.

The check runs the same tests as this form in a REPL:

(asdf:test-system :sexp-store)

Part of the Lambda Symbolics library shelf.