This document is the source of truth for the on-disk format. Code must follow it; changes need a format version bump.
All integers are little-endian. varint is unsigned LEB128 (7 bits per byte, high bit = more).
string is varint length + UTF-8 bytes. Id is 32 raw bytes, written as 64 lowercase hex chars
in file names.
<repo>/
config plaintext header + password-wrapped master key
recovery optional: the master key wrapped with the recovery key
data/<xx>/<pack-id> pack files (xx = first two hex chars of the pack id)
index/<index-id> sealed index files
snapshots/<snapshot-id> sealed snapshot manifests
locks/<lock-id> sealed lock files of the runs using the repository
status/<computer-id> optional: sealed status record of each computer backing up here
status/requests/<computer-id> optional: sealed request to that computer to back up now
parity/percent optional: repair data is on, at this percentage ("5\n"), see below
parity/<xx>/<pack-id> optional: the repair data of a pack (see "Recovery data")
settings/compression optional: the zstd level for new data ("12\n", 1 to 19), see below
settings/compression (Vaultik 0.12 and later) overrides the zstd level stored in config for
data written from now on. Every zstd level decompresses the same way, so nothing else changes:
versions before it ignore the file and keep compressing at the level in config, and read all
data as before. Removing the file returns to the level in config.
Files are written to a temporary name and renamed into place, so a reader never sees a partial
file. In S3-compatible cloud storage (s3:https://<endpoint>/<bucket>[/<folder>]) each file is
one object named <folder>/<path> with the same layout; object uploads are atomic, so there are no
temporary names. The cloud drives (pcloud:, onedrive:, gdrive:, mega: followed by a
folder) hold the same layout as real folders and files; an upload either replaces a file whole or
leaves it as it was. On WebDAV (webdav:) a file is uploaded as <name>.tmp-<16 hex> and moved
into place with MOVE (Overwrite: T). On SFTP (sftp:) it is written as <name>.tmp-<16 hex> and
renamed over the name; a server that cannot rename over an existing file gets the old file moved
aside as <name>.old first, removed once the new one is in place, and readers take <name>.old
for the name while the name itself is missing (a replace cut short in between). Listings leave out
names containing .tmp- (left by an upload cut short; they can be deleted by hand) and list
<name>.old as the name. A repository folder copied to or from a bucket or drive stays valid. Write order during a backup is: packs, then the index files that reference them, then the
snapshot. A crash leaves at worst unreferenced packs, never a snapshot pointing at missing data;
the next prune removes them. Lock files (see below) were added in Vaultik 0.2 without a format
version change: older versions ignore them.
The master key is 64 random bytes: enc_key (32, AES-256-GCM) followed by id_key (32, keyed
BLAKE3). It never touches disk in plaintext.
The password is stretched with PBKDF2-HMAC-SHA256 (kdf_iterations, 32-byte salt) into a
32-byte key-encryption key, which seals the master key with AES-256-GCM. The AAD of that seal is
the full config header (everything before the wrapped key), so tampering with any parameter fails
authentication.
Changing the password rewrites only config, with a new salt; the master key, and with it every
other file, stays the same.
The optional recovery file (added in Vaultik 0.3 without a format version change; older
versions ignore it) wraps the same master key with a recovery key: 20 random bytes, shown to the
user as 32 characters of Crockford base32 (0-9 A-Z without I L O U) in groups of 4. Input is
read case-insensitively with O as 0 and I/L as 1. The raw 20 bytes go through
PBKDF2-HMAC-SHA256 (iterations, salt below) to the key-encryption key. Making a new recovery
key replaces the file.
magic 8 bytes "VKRECOV\0"
version u32 1
repo_id 16 bytes the repo_id from config
iterations u32 100000; 1000 to 20000000, others are refused
salt 32 bytes
--- end of header (AAD) ---
wrapped_len u32 92
wrapped_key nonce(12) || ciphertext(64) || tag(16)
The AAD binds the file to its repository: a recovery file copied from another repository fails authentication.
magic 8 bytes "VAULTIK\0"
version u32 1
repo_id 16 bytes random
kdf u32 1 = PBKDF2-HMAC-SHA256
kdf_iterations u32 default 600000; 1000 to 20000000, others are refused
salt 32 bytes
chunk_min u32 default 262144 (256 KiB)
chunk_avg u32 default 1048576 (1 MiB, power of two)
chunk_max u32 default 4194304 (4 MiB)
zstd_level u32 default 3
--- end of header (AAD) ---
wrapped_len u32 92
wrapped_key nonce(12) || ciphertext(64) || tag(16)
Every encrypted object (blob, index file, snapshot) uses the same envelope:
sealed = nonce(12) || AES-256-GCM(enc_key, nonce, plaintext, aad) || tag(16)
plaintext = compression u8 (0 = none, 1 = zstd) || varint raw_len || payload
Nonces are 96 random bits per object. With 1 MiB chunks, the birthday bound for nonce collision reaches 2^-32 only after about 2^32 objects (around 4 PiB), far beyond a personal repository.
Compression is applied only when it makes the payload smaller.
AAD values bind each object to its role and name:
| Object | AAD |
|---|---|
| blob | type u8 |
| pack header | "VKPKHDR1" |
| index file | "VKINDEX1" |
| snapshot | "VKSNAP01" |
| lock | "VKLOCK01" |
| status record | "VKSTAT01" |
| backup request | "VKREQ001" |
A blob is either file content (type 0, data) or a directory listing (type 1, tree). The blob id
is BLAKE3-keyed(id_key, type u8 || plaintext). Keying hides content fingerprints from whoever
stores the repository (for example a cloud provider); the type prefix keeps data and tree blobs in
separate id spaces, so a file chunk that happens to equal a tree encoding is never confused with it. Identical plaintext always gets the same id, which is
what makes deduplication work across files, folders, snapshots and machines that share a password.
File content is split with FastCDC (normalized chunking, level 2):
- Rolling hash:
h = (h << 1) + gear[byte]over 64-bit values. - Bytes before
chunk_minare skipped; a cut is never made beforechunk_minor afterchunk_max. - Between
chunk_minandchunk_avga cut is made whenh & mask_s == 0; afterchunk_avgwhenh & mask_l == 0. Withb = log2(chunk_avg),mask_sis the topb + 2bits andmask_lthe topb - 2bits of a 64-bit word. gearis 256 values of splitmix64 starting from seed0x5641554C54494B31("VAULTIK1"):x += 0x9E3779B97F4A7C15; z = x; z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9;z = (z ^ (z >> 27)) * 0x94D049BB133111EB; gear[i] = z ^ (z >> 31).
Chunking restarts at every file, so a file smaller than chunk_min is a single chunk.
blob_0 .. blob_n-1 sealed blobs, back to back
header sealed (AAD "VKPKHDR1"), plaintext = varint count, then per blob:
type u8, id (32), varint offset, varint length, varint raw_len
header_len u32
magic 4 bytes "VKPK"
The pack id is the unkeyed BLAKE3 hash of the whole pack file. Packs are closed at about 32 MiB. The header makes every pack self-describing, so the index can be rebuilt from packs alone.
A pack may hold blobs of both types. While the storage keeps file data in an archive storage
class (S3 x-amz-storage-class, the job's "Storage class"), data blobs and tree blobs go to
separate packs: packs holding only data blobs, and their parity/<xx>/<pack-id> files, are
uploaded in that class; packs holding tree blobs and every other file stay in the standard class,
so the index, snapshots and folder listings can be read without a thaw. The format is the same
either way, and a reader does not need to know which packs are archived (reading one that is
not thawed fails until it is).
Sealed with AAD "VKINDEX1" || index id. Plaintext:
version u32 1
packs varint
per pack: pack id (32), varint count, then per blob the same entry layout as the pack header
A tree blob lists one directory. Entries are sorted by name, bytewise on UTF-8, with no duplicates. Writers produce version 3 when any node has extras (see below) and version 2 otherwise, so trees without extras stay readable by Vaultik 0.5 and older; readers accept versions 1 to 3.
version varint 2 or 3
count varint
per node:
name string
kind u8 0 = file, 1 = directory, 2 = metadata only (no content), 3 = symlink
attributes u32 Win32 file attributes (0 when backed up on Linux)
mode u32 POSIX permission bits, 07777 (0 when backed up on Windows)
uid u32 POSIX owner (0 on Windows)
gid u32 POSIX group (0 on Windows)
ctime u64 Windows: creation time. Linux: status change time. FILETIME units
(100 ns since 1601-01-01 UTC)
mtime u64 last write time, FILETIME units
size u64 file size in bytes (0 for everything but files)
file: varint chunk count, then chunk ids (32 each), in file order
directory: subtree id (32)
symlink: string link target, exactly as stored in the link
extras: version 3 only: varint count, then per extra: tag u8, varint length, that many bytes
Extras (readers skip tags they do not know):
| Tag | Content | Meaning |
|---|---|---|
| 1 | id (32) | a Data blob holding the Windows security descriptor (owner, group, access list; self-relative). Identical descriptors are one blob. |
| 2 | name (string), size (varint), chunk count (varint), chunk ids (32 each) | an NTFS alternate data stream of the file ("file:name") |
| 3 | reparse tag (u32), flags (u8, bit 0 = relative), target (string) | the target of a Windows symlink (0xA000000C) or junction (0xA0000003) on a kind 2 node; absolute targets without the \??\ prefix |
| 4 | count (varint, 2 or more) | the file had this many names (hard links) when it was read. A Windows folder listing shows a hard-linked file's size and times as of the last change through that name, so the next backup asks the file itself before reusing its content. Older readers skip it. |
Every blob an extra names counts as in use for prune, is copied with the version, and is checked by verify like file content.
Version 1 (written by Vaultik 0.1, Windows only) has the same layout without mode, uid and
gid, and only kinds 0 to 2.
Kind 2 is used for Windows symlinks, junctions and cloud placeholders (OneDrive files that are not on the disk), and for Linux devices, sockets and pipes. Restores recreate symlinks and junctions that have a link extra (without following them) and skip the rest.
ctime is only compared for change detection between backups; restores set it on Windows and ignore it on Linux, where it cannot be set.
The snapshot root tree is virtual. On Windows its children are drive letters (C) or UNC for
network paths; on Linux they are the top-level folders (home, etc, ...). Below those come
the path components down to each backed-up source. Virtual directories have zero attributes,
mode and timestamps.
Two more children hold virtual machines backed up through their export tools (Windows; never a
drive letter, so they cannot clash with backed-up paths): WSL holds one file <distro>.tar per
WSL distribution, the output of wsl --export <distro> - (a tar archive; its node's mtime and
ctime are the snapshot's time, its attributes 0x20, mode 0100644), and Hyper-V holds one folder
per Hyper-V machine (its name with <>:"/\|?* and control characters as _), the folder
Export-VM wrote: Virtual Machines (the .vmcx configuration and its state files), Virtual Hard Disks and, when it has checkpoints, Snapshots. A snapshot's paths list only the
backed-up folders and files, not these.
Sealed with AAD "VKSNAP01" || snapshot id. The snapshot id is 32 random bytes. Plaintext:
version u32 1
time u64 FILETIME UTC
hostname string
username string
paths varint count, then strings (full source paths as given)
tree Id root tree
has_parent u8 0 or 1, followed by the parent snapshot id (32) when 1
tags varint count, then strings
stats varint files, dirs, bytes, new_bytes, stored_bytes, errors, skipped
Tags starting with vaultik: are reserved: they carry fields added after version 1 without a
format change, so older Vaultik versions still read the snapshot (and show them as tags):
vaultik:pinned: retention and size limits never remove the snapshot.vaultik:note=<text>: the user's description of the snapshot.vaultik:changes=<new>,<changed>,<removed>: files new and changed since the parent snapshot, and files or folders of the parent that are gone (Vaultik 0.4 and later).vaultik:copy-of=<64 hex>: in a second copy, the id of the snapshot it was copied from.vaultik:apps=<64 hex>: the id of a Data blob listing the programs installed when the snapshot was made (Vaultik 0.7 and later; see below).vaultik:image: a disk image (added in Vaultik 0.8.0; see "Disk images"). Older versions see its partitions as folders of large files.
The program list blob is stored like file content (so an unchanged list is one blob across snapshots). Its plaintext is UTF-8 JSON, sorted by name:
{"version":1,"apps":[{"name":"7-Zip 24.08 (x64)","version":"24.08","publisher":"Igor Pavlov",
"id":"7zip.7zip","source":"winget"}, ...]}
name is required; version, publisher, id (a winget package id: 1 to 128 characters of
A-Z a-z 0-9 . _ - +, starting with a letter or digit) and source (winget or msstore) are
left out when unknown. Readers drop an id or source that does not match these rules. Prune
counts the blob as in use, copy copies it with the snapshot, verify reports it when missing, and
repair saves the snapshot again without the field when the blob is lost. Versions before 0.7 do
not know the tag: their prune removes the blob, and the snapshot then reads as having no list.
Unknown reserved tags are ignored. Pinning or describing a snapshot rewrites its file under the same id while holding a shared lock, and never recreates a snapshot that was removed.
An image snapshot (the reserved tag vaultik:image, added in Vaultik 0.8.0 without a
format version change) holds whole disks.
Its paths are image:<disk id> per disk, the disk id being gpt-<disk GUID> or
mbr-<signature, 8 hex digits>. Its tree is an ordinary tree (version 2), so prune, verify,
copy and repair need nothing new:
disk0/ one folder per disk, in the order backed up
layout.json the disk layout (below), as file content
r0, r1, ... the stored regions of the disk outside its partitions, as files
p0, p1, ... per partition of the layout (by its index), unless it is an MBR container
g0000/ groups of 64 segments
s000000 the partition's bytes in segments of 1 GiB (the last one shorter)
All nodes have zero attributes, mode and times, so an unchanged segment or group gives the same
tree blob as before. A segment's size is its length; its chunks, concatenated, are its bytes.
Partitions read "used" (NTFS) store unused clusters as zeros: runs shorter than 16 MiB inside
the data stream (chunked with it), longer runs ("holes") as chunks of chunk_max zero bytes and
one shorter one. The clusters of pagefile.sys, hiberfil.sys and swapfile.sys (as the volume
read, the shadow copy, has them) count as unused, but the first 1 MiB of each (rounded out to
whole clusters) is never part of a hole, so a restore onto a disk with old data writes it: a
hibernation file left there could otherwise be resumed over the restored volume. A restore recognises zero chunks by their id (the id of that many zero bytes,
looked up through the blob's raw length in the index) and need not load or write them on a new
virtual disk, nor in a hole on a disk being overwritten.
layout.json (UTF-8 JSON; readers ignore unknown fields):
{"version": 1, "style": "gpt"|"mbr", "sector_size": 512|4096, "physical_sector_size",
"size": <bytes>, "disk_guid": "<GUID>" (gpt), "signature": <u32> (mbr), "gpt_entries": 128,
"number", "model", "serial", "system": <Windows ran from it>, "id": "<disk id>",
"partitions": [{"number", "offset", "length" (bytes, sector multiples),
"type", "guid", "name", "attributes" (gpt), "mbr_type", "active" (mbr),
"filesystem", "label", "mount", "serial", "cluster_size", "windows",
"method": "used"|"raw", "shadow_copy": <bool>, "used_bytes",
"holes": [[offset, length], ...] (from the partition start, "used" only)}],
"regions": [{"offset", "length"}]}
Regions are the first 1 MiB (at most) of every gap outside the partitions (MBR containers do not count as used): the MBR with its boot code, the GPT, boot loaders; on MBR disks also up to 1 MiB from the start of every container inside a gap, where the EBRs of logical partitions sit. Regions never overlap and are sorted (images made before this have only the gap starts). A restore first makes the partition tables (refusing a layout they cannot be made for before writing anything), then writes, in this order: zeros over the first and last 1 MiB of a disk that is not empty, the regions with the partition tables they hold cleared (the MBR's four entries, and for GPT the original disk's primary and backup headers and entries; an MBR gets the layout's signature at offset 440), every partition, and last the partition tables: for GPT a new protective MBR, primary and backup GPT computed for the target's size (same disk and partition GUIDs unless new ones are asked for), for MBR the stored first sector as it was. Until then the disk shows no partitions. Readers reject a layout version other than 1, sector sizes other than 512 and 4096, partitions or regions outside the disk, and regions over 16 MiB.
A restore onto a disk smaller than the layout needs (no format change) places the partitions
first: NTFS partitions read "used" from a shadow copy can shrink to just past the last cluster
their volume bitmap (read from the stored $Bitmap) marks in use, rounded up to whole MiB, with
1 GiB more free on the partition marked windows. The last partition that can shrink is cut
first, by whole MiB and only as far as needed; every partition and stored region after a cut
one moves down by as much. A cut partition gets its stored bytes up to its new length, then the
changed metadata: the boot sector's total sectors (and hidden sectors when it moved), the backup
boot sector in the new last sector, $Bitmap (size, runs cut to what the new size needs, the
freed bits cleared, the bits past the last cluster set as the volume had them), the $Bad
stream of $BadClus cut to the new size, $LogFile filled with 0xFF, the dirty flag in
$Volume, the copy of $Bitmap's sizes in its file name and root index entry (when they were
current), and the changed records in $MFTMirr too when it mirrors them. A partition that only
moves gets its NTFS boot sectors' hidden sectors changed. MBR entries of cut and moved
partitions get their new start and length. Every file record in use is checked first; data past
the new end refuses the restore before anything is written.
Every run that reads or writes snapshot data holds a lock file. The lock id is 32 random bytes;
the file is sealed with AAD "VKLOCK01" || lock id (never compressed). Plaintext:
version u32 1
exclusive u8 0 = shared, 1 = exclusive
time u64 FILETIME UTC of the last refresh
pid u32 process id of the holder
hostname string
username string
command string "backup", "prune", ...
Backup, restore, ls, verify, a second copy (on both repositories), the repair data setting and the other reading commands take a shared lock; forget, prune and repair take an exclusive one, because they delete or replace objects. A run writes its own lock first and then lists the others; on a conflict it removes its lock again and waits or gives up. Holders rewrite their lock every 5 minutes. A lock is stale when it is held by a process that no longer runs on the same computer, when a lock of the same computer was not refreshed for 30 minutes, or when another computer's lock was not refreshed for 2 hours (its time is that computer's clock, judged by the reader's, so clocks up to about an hour and a half apart cannot remove each other's live locks). A stale lock is ignored and deleted by whoever finds it. Whether a lock is still there is asked of the storage itself, never answered from a listing the run made earlier (Google Drive, MEGA).
Several computers can back up into one repository. After each run of a saved job (backup, health
check, second copy, test restore) the computer writes its status record, so any of them can show
how every computer's backups are doing (vaultik computers). Added in Vaultik 0.8.0 without a
format version change: older versions ignore the status/ folder.
The computer id is BLAKE3-keyed(id_key, "VKHOST01" || repo_id (16) || machine id), where the
machine id is the UTF-8 text of Windows' HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid
(/etc/machine-id on Linux; host:<name> when neither exists). It stays the same when the
computer is renamed, and since it is keyed and includes the repository id, the records of one
computer in two repositories cannot be linked. The file status/<computer id as hex> is sealed
like every object (AAD "VKSTAT01" || computer id; compressed when that helps), so a record moved
to another computer's name fails authentication. Its plaintext is UTF-8 JSON; readers ignore
fields they do not know and treat missing ones as empty or 0:
{"format": 1, "host": "<computer name>", "version": "<Vaultik version>",
"updated": <FILETIME>, "jobs": [
{"id", "name", "schedule": "<in words>",
"last_run": <FILETIME the newest run of any kind ended>,
"last_kind": "backup|check|full_check|copy|restore_test",
"last_result": "success|partial|failed|cancelled", "last_message",
"files", "bytes", "new_bytes", "errors": <counts of that run>,
"last_backup": <FILETIME>, "last_backup_result",
"last_success": <FILETIME the newest backup that saved a version ended>,
"next_run": <FILETIME of the next scheduled run, 0 for a manual job>,
"damage": <true when the newest health check or test restore found damage>,
"warnings": ["<drive health, ransomware warning, ...>"]}]}
A writer reads its own record, replaces the entry of the job that ran (keeping its other jobs'), and writes the whole file back in one put (atomic, like every file). Two runs of one computer ending together could each write back what they read, losing one entry, so a writer reads the record back afterwards and, when its entry is missing or different, merges and writes again (up to 5 times, after a random pause of 50 to 300 ms). Writing it is best effort: a run never fails because its record could not be written. Prune, repair and verify do not touch the folder; S3 Object Lock leaves status records unlocked (they are rewritten after every run), and a cloud rollback does not bring back old ones. A job is stale when it has not run for 3 days past its recorded next run.
A computer that no longer backs up here can be removed (vaultik computers --forget): its status
record and a request addressed to it are deleted; its snapshots stay. A computer never removes its
own record.
A computer can ask another to back up now: it writes status/requests/<computer id of the asked computer>, sealed like a status record but with AAD "VKREQ001" || that computer id, plaintext
UTF-8 JSON:
{"format": 1, "from": "<asking computer's name>", "created": <FILETIME>,
"jobs": ["<job id>", ...], "state": "pending|running", "started": <FILETIME taken up, or 0>}
jobs are ids from the asked computer's status record. One request per computer: a new one
replaces the old. The asked computer looks for its request when one of its jobs on the
repository runs and when its frontend polls (every 15 minutes for folders and network shares,
every hour for cloud storage); it runs those of the listed jobs it has that back up to this
repository, sets state to running, removes each job from the list after it ran (whatever its
result) and deletes the file when the list is empty. A request older than 7 days, or one naming
none of its jobs here, is deleted unanswered. Requests ask only for saved jobs to run: nothing in
them is a command, a path or a setting. S3 Object Lock and cloud rollback treat them like status
records (unlocked, never brought back).
Prune holds the exclusive lock and works in this order, so a crash at any point leaves a valid
repository: (1) collect every blob reachable from any snapshot, stopping without changes if a
snapshot or folder listing cannot be read; (2) write new packs with the blobs still in use from
packs that are mostly unused (sealed blobs are copied as they are, after authenticating them);
(3) write fresh index files for every pack that stays; (4) delete the old index files; (5) delete
the packs nothing uses any more. Packs that no index file lists are read through their own
header: kept and indexed again if they hold blobs in use, deleted otherwise. A pack an index file
lists but the listing of data/ does not show is checked directly (a listing can come back short);
when it is really gone it is left out of the new index files (its blobs are lost, which vaultik verify reports and a repair cleans up). When more than one, and more than a tenth, of the indexed
packs are gone, prune stops without changing anything: the destination is more likely wrong or
unreadable than the data lost.
Repair (vaultik repair) holds the exclusive lock and opens the repository without reading the
index files. In this order, so a crash at any point leaves a repository that opens: (1) read
every pack's header and (unless --quick) decode and hash-check every blob; the first intact
copy of each blob goes into the new index, packs that cannot be read are left in place and not
indexed. A pack the old index files list but the listing does not show is checked directly first;
when more than one, and more than a tenth, of them are gone, repair stops without changing
anything unless it is run with --accept-missing-packs; (2) write new index files for that index, then delete every old index file; (3) delete
snapshot files that cannot be opened or whose root tree is lost; for the others, rewrite every
tree that references a lost blob (dropping the file nodes whose chunks are missing and the
directory nodes whose tree is missing), save the snapshot again with the new root tree and a note
"(repaired: N file(s) and M folder(s) were lost)", then delete the old snapshot file.
While parity/percent exists (a decimal 1 to 50), every program that writes a pack also writes
parity/<xx>/<pack-id>: the pack's recovery data (below) at that percentage, from the pack as
written. It is plaintext: it is computed over the pack file, which is ciphertext already, and the
hashes in it are of that ciphertext. Writing it is best effort (a pack without it is valid); a
health check that reads an intact pack without it adds it. Prune deletes it with its pack. Repair
rebuilds a pack that cannot be read whole, whose content does not hash to its id, or whose header
cannot be read, from its repair data, and when the result hashes to the pack id it writes the
pack back at once (the content its id names, so nothing that refers to it changes). Removing
parity/percent turns it off for new packs; existing repair data stays usable. Versions before
0.7 ignore the parity/ folder.
Recovery data lets a damaged file (usually a pack file) be rebuilt, like a .par2 file. The file
of data_size bytes is cut into k data shards of shard_size bytes, back to back, the last one
zero-padded; m parity shards are computed with a systematic Reed-Solomon code over GF(2^8)
(polynomial 0x11D). Any k intact shards out of the k + m rebuild the file, and since the
shards are contiguous, a run of bad sectors damages only the few shards it touches. Little-endian:
magic 4 bytes "VKRD"
version u32 1
data_size u64 size of the original file
shard_size u32
data_shards u32 k
parity_shards u32 m
data_hash 32 BLAKE3 of the whole original file
shard_hashes 16 * (k + m) first 16 bytes of BLAKE3 of each data shard (padded), then of each
parity shard
header_hash 16 first 16 bytes of BLAKE3 of every byte above
parity m * shard_size parity shard 0 .. m-1, back to back
- Parity shard
iis the sum (XOR) over data shardsjofC[i][j] * shard_j, byte by byte, with the Cauchy matrixC[i][j] = 1 / ((k + i) xor j)in GF(2^8). - The writer takes a percentage
p(1 to 50) and chooses:k_max= the largest value up to 240 withk_max + ceil(k_max * p / 100) <= 255;shard_size = max(ceil(data_size / k_max), min(data_size, 4096));k = ceil(data_size / shard_size);m = max(1, ceil(k * p / 100)). An empty file hask = 1andshard_size = 0. - Readers reject (as "not recovery data") a wrong magic or version, a header checksum mismatch,
kormof 0,k + m > 255,data_sizeabove 1 GiB, and any shard layout the writer cannot produce: fordata_size > 0,0 < shard_size <= data_sizeand(k - 1) * shard_size < data_size <= k * shard_size. Parity shards that fail their hash or lie past the end of the recovery data are not used. - Repair hashes every data shard of the damaged file (bytes past the end of a cut-short file read
as zeros, bytes past
data_sizeare dropped), rebuilds the shards that do not match from the intact data and parity shards, and keeps the result only if its BLAKE3 equalsdata_hash.