Repository navigation
feat: build only what changed, and read history in place - #49
Merged
Merged
Conversation
Surviving Lines held the text of every head file a person touched in one walk. Each walk now holds at most BATCH_BYTES of file text (always at least one file) and the rest get walks of their own, so the memory has a ceiling however large the repository. The counts are the same.
They were drawn to SVG and PNG on every request and never stored. They are now kept in a 32 MB least-recently-used cache for as long as stored Cards are, still never written to R2, where any of millions of colours could fill the bucket. The cache that answers used moves to its own module so both share it.
…t a time - The engine cache of a public repository lives in work/caches, moved out of its clone's folder where older Builders kept it, and is pruned under its own CACHE_BUDGET_GB: pruning a clone no longer loses the history read from it. - Engine runs share ENGINE_SLOTS (1) in the order they asked, so Builds, large Builds and quiet Builds never run the engine side by side on mercury's two cores; git and the network still run at once. - A quiet Build that counts the lines a big repository has left gets MORE_LINES_BUDGET_SECONDS (600), so one run publishes what three did.
Pruning walked every file of every kept clone and cache after every Build. Sizes are now kept in work/sizes.json with the folder's time, and only a folder used since it was measured is walked again; a Build touches its clone's folder when it ends, so what it fetched is counted.
Counting the lines of a commit already loaded rewrote its whole index_commits row (subject, files, changes), which left 1.37M dead rows and about 10 GB of writes per fill-in on linux. Lines Changed now go to index_commit_lines (si, sha, added, removed): the sums, since Postgres only ever read the sums. A backfill migration fills it from the arrays, and the trigger that deletes a Stored Index's rows covers it. The arrays stay in index_commits, no longer written, so a Site still running during a deploy keeps working; they go with the per-file arrays. linux: a Build that fills in lines went from 194 s of Postgres to 9 s, and index_commits from 953 MB to 510 MB.
…he last load A warm export resent every commit's Mainline and every counted commit's Lines Changed: 104 MB for a linux Build that counted 30 s more lines. Each export now leaves a mark in its manifest: the head its Mainline was worked out from, how far the line store's log had got with a hash of it, and a fingerprint of what else the rows depend on (the format, the Bulk Commit size, the commits blame passes through, a cut-short history). The Site keeps it on stored_indexes.mark and the Builder hands it back with --since. While it holds, mainline.tsv holds only the commits whose Mainline moved since that head, and lines.tsv only the commits counted into the log since; anything that does not match sends everything as before. Full exports no longer write mainline.tsv, which the loader never read. Export format 4. linux: 104 MB to 8 MB (the lines counted in that run) and 9 s to 4.9 s of Postgres; react with nothing new: 5.6 MB to nothing, 1.0 s to 0.17 s.
A Build whose Report is whole (its pack, every Mainline line counted and none left, built within a week) now asks the remote where the default branch's head is with one `git ls-remote`. When it is still the head the Report was built from, the Build ends there: no clone, no engine run, no load and no upload. repositories.checked_at records when a Report was found current, and the Site counts a Report checked within a day as fresh. Quiet Builds and retries skip the check, and a Report a week old is built again so the time Windows never drift further than that. Seeds follow the same rule: their heads are asked of GitHub's GraphQL API a hundred repositories to a request, an unmoved one is only marked checked, and a moved one is queued when someone viewed it since its last Build, or once its Report is a week old. A Build with nothing new (react, linux; this box, 2 cores): before, a fetch, a warm engine run (linux ~20 s, 1,082 MB anon), a Postgres load, a pack upload and an Atlas on the Site; after, 1.0-2.3 s wall, all of it the round trip to GitHub, about 40 ms of Node CPU and 30 ms of git.
…from new commits Every Build walked git twice over the whole history: once from HEAD for the Mainline, once from the last export's frontier for the commits the Site already has. The engine cache now keeps the commit graph beside the Index: each commit's parents by its place in the Index, and the Mainline of the head it was last worked out for. A Build lines it up with the Index, asks git only for the parents of the commits added since, walks only the new part of the first-parent line when HEAD went forward along it, and works the rest out in memory. A history written again, a cut-short one or a graph that no longer lines up is made again from git, as before. The export takes the commits the frontier reaches as places in the Index. linux on 2 cores, commit-graph present as on the Builder, no line pass: - nothing new: Mainline and frontier 4.2 s -> 91 ms, the run 9.7 s -> 5.1 s, peak anon 599 -> 500 MB (the first run after this change makes the graph once, 2.6 s; the file is 25.7 MB) - 337 new commits: 3.6 s -> 191 ms, the run 11.5 s -> 6.0 s, peak anon 754 -> 607 MB; the export and the pack are byte for byte the same
The Site made each pack's Universe Atlas itself, asking the engine service for every row as one JSON answer and packing it twice with brotli: on linux a 1.16 GB spike and 35 CPU-s on venus for every pack. `commitscape build` now writes the rows beside the pack as JSON lines (`universe.jsonl`, streamed, one commit, file or person a line); the rows code moves from the engine service to commitscape-report so both use it. Before a pack is published the Builder makes its Atlas from those rows with the Site's own atlasOf, encodeAtlas and encodeNames, in a process of its own (`node builder.mjs atlas`) that waits for an engine slot, packs it with brotli at ATLAS_QUALITY (5) and stores it under the key the Site reads first, now shared as atlasKey. The Site sends those bytes; it makes an Atlas only for a pack the Builder did not, as when the Builder's step fails. linux (2 cores): the Site's 1.16 GB and 35 CPU-s per pack -> 0; the Builder writes the rows in 0.9 s (170 MB, no change to the engine's peak) and makes the Atlas in 3.9 s wall, 4.8 CPU-s, 635 MB peak in a short-lived process; map 7.8 MB (11 would give 6.6 MB for 35 CPU-s). react: rows 49 ms (7 MB), Atlas 0.28 s, 64 MB, map 260 KB.
…disk A Repo State is the engine cache of a public repository: its Index, the Lines Changed counted and its HEAD's files. Losing it from disk meant reading the whole history again: linux, 185 s without lines and about 1,500 CPU-s with them. With STATE_KEY set, after a Build publishes, the Builder uploads the cache to R2 as 8 MB chunks named by their content and sealed with a key made from STATE_KEY (zstd, then AES-256-GCM), so no email address the cache holds leaves the Builder readable. Only chunks R2 does not hold yet are sent, so an append sends its tail; then the manifest naming them, then the chunks nothing names any more are deleted. A Build whose disk has no cache brings it back before the engine's first run, checking each chunk against its name and moving the files into place only once all are whole; anything missing or changed means a clean Build, as before. The commit graph and blame caches stay behind: they are made again. repo_states records when each State was saved and used; tidy deletes a State from the disk after STATE_DISK_DAYS (7) unused, even within CACHE_BUDGET_GB, and from R2 after STATE_KEEP_DAYS (180), once a day. Without STATE_KEY nothing changes. linux on 2 cores against the local S3 stand-in: first save of a 217 MB State sends 128 MB (3.5 CPU-s); a save with nothing new reads nothing (40 ms); after 337 new commits 15 MB (0.6 CPU-s); a restore takes 9.4 s and 2.6 CPU-s, 128 MB peak, files byte for byte the same, and the engine then loads it warm (0.4 s, plus 2.6 s to make the graph again) instead of reading the history for 185 s.
…ines Surviving Lines were blamed person by person, for the 30 people with most commits of each seed head and for anyone asked for, nothing kept from one head to the next: on linux 1.7-2.1 GB and 60 s a person, most over budget. The engine now works out the head's Ownership of lines: which commit each line of each Source file came from, as blame says. The walk is blame's own (the same diffs, renames, merges and lines passed through), also tracking where each line sits at the head. It is kept in the cache as runs of lines and carried from head to head: when the new head went forward along its first-parent line, a file whose path no first-parent commit since changed keeps its ownership as it was, which is what blame would give, and every other walk stops wherever it reaches the earlier head with a file it knows. Cut short by its budget it keeps what it finished and the next run goes on. Everyone's Surviving Lines are then a sum over it, with the files each person changed or that flowed from them, exactly as the per-person count chooses them. `commitscape surviving --everyone` does this. The Builder queues one count for everyone after each new Report and marks its people counting; a person asked for on the Site runs the same count. Today's blame stays the referee: a test compares everyone's counts with blame person by person on scripted histories (overwrites, renames, bulk and ignored commits, merges, a file changed and changed back between heads), on every fixture, and, set up with COMMITSCAPE_ORACLE_REPO, on a real repository: react, 1,997 people, 827,110 lines, the same. react on 2 cores: today the 30 seeds' people take 12.4 s, 20.3 CPU-s and 652 MB in one run; everyone (1,997 people) from nothing 18.5 s, 25.3 CPU-s, 508 MB; a head 20 commits later 0.3 s and 71 MB; the same head 0.19 s. The ownership file is 4 MB.
…t changed Records the head check and seed policy (D1), the kept commit graph and Mainline, the Atlas made once per pack by the Builder, the four tiers of the Repo State with its sealed copy in R2, Surviving Lines summed from the Ownership of lines once per head (D2, carried over along the first-parent line with blame as the referee), and why a forward pass along the Mainline and full clones were not kept as the plan.
…ing Lines are counted A public repository kept a full clone for every Build: linux's is 6.6 GB and receiving it again costs 579 CPU-s of index-pack. Once its Lines Changed and its Report's Surviving Lines are all counted, its Builds now read a lean clone instead: blobless, with no files written out (`clone --filter=blob:none --no-checkout`, then `reset --soft`, with a commit-graph), so a fetch brings only new commits and trees. Before the engine runs, the Builder lists the file versions the Build reads that the clone lacks (every file at HEAD, and both sides of each change in the commits past the last export's frontier, Bulk Commits left out) and fetches them by id, 20,000 to a request, each tried twice; the full clone is then deleted. git data reads the same clone (git fetches what diff and merge-tree need itself), and the count of Surviving Lines fetches every version of the files changed since its last count the same way, and any other it meets as it goes. A retry reads the full clone. The engine no longer fails on a missing file version: the line pass leaves that commit's lines to count later, never a wrong "not counted". linux, 19 days behind GitHub (17,127 commits), on 2 cores: a blobless fetch of 36 MB and 3.1 CPU-s plus the file versions those commits change, 105 MB and 9.6 CPU-s, against 214 MB and 16.6 CPU-s fetched in full; then the Build on the lean clone: 16.5 s, every new commit's lines counted (15,978, none left), 885 MB peak anon. The kept clone is 2.4 GB instead of 6.6 GB; react's 62 MB instead of 1.1 GB.
A word search of the Commit List scanned every subject of the Stored Index in Postgres, twice on a first page (the rows and the count): on linux 2.1 s warm and 11 s cold. Commits under a folder did the same scan to hand the engine service the commits whose words matched. The engine service, which already holds every subject, now matches the words itself as Postgres did: each word, lowercased, in the subject or in the name or login of who made the commit, every word needed, with the other filters (person, kind, dates, weekday, hour, merges). The new /v1/commits/search answers a page of shas newest first with the total and the next cursor, and /v1/commits/under matches words too; the Site reads only that page's rows from Postgres. A Report with no pack, or a query with no words, reads Postgres as before. linux's pack (1.5M commits) on 2 cores: "fix" 251,254 matches, "memory leak" 4,589, a login 47,714, each 0.22-0.26 s; the same question again comes from the engine's answer cache.
The Code window already reads a file's history a page at a time (/v1/code/log). What was left of plan item 2.3 was log.json: the whole history in one answer, 40 MB on linux, which the engine service made and kept and the Site packed and kept, though nothing has read it since the Universe moved to its Atlas. Anyone could still ask for it, a crawler included. The Site no longer serves it, its engine client no longer asks, and the DataSource and GitLog types lose it. The engine service's endpoint stays until its goldens are written again.
…'s words are searched DEPLOY.md gains the upgrade steps: Builder (migrations 0037 and 0038, STATE_KEY to turn on the R2 tier), engine service (commit search), Site; and what the first counts of everyone's Surviving Lines look like. docs/index-in-postgres.md says the word search now asks the engine service.
…g Lines The Builder kept at most 1 MB of what a command printed. A count for everyone prints every person: about 240 KB for react and several MB for a repository with linux's tens of thousands of people, which would have been cut short and read as a failure. A run can now ask to keep more, and the count keeps up to 256 MB. The commit graph is also made again, rather than kept with a parent left out, when a new commit's parent cannot be placed in the Index.
vercel/next.js (2.6 GB, 50,110 commits) clones in 291 s, 163 CPU-s, 0.6 GB, and its first Build counts every line in 19 s with 525 MB anon on 2 cores. With one engine slot under mercury's 3 GiB cap, 4000 MB takes next.js-sized repositories with room; linux-sized ones also need a longer clone limit and the disk for one full clone.
…rrays a Column The commits, changes and subjects of an Index are now Columns: held in memory as before, or, next, read in place from a mapped file. Their records get fixed layouts for that (CommitMeta and FileChange repr(C), the ids transparent, CommitKind one byte), and a change's Lines Changed are read and set through lines() and set_lines(). Packs and caches encode exactly as before, so old ones still read; every Rust test passes unchanged.
…the cache Each cache generation now keeps its commits, changes, subjects and the commits' settled line marks in a page-aligned column file (<gen>.cols), mapped copy-on-write by the new commitscape-map crate (the only unsafe code). A warm Build maps it instead of decoding the blocks; a Build with new commits streams the old columns and the new commits into the next generation's file; the line pass counts only the commits not settled and writes them back in place. When the files left out of Lines Changed change at a new head, only those files' changes are masked again. The file reads as absent until a save is on disk, so a Build stopped halfway reads the lines from the store again. The commit graph file is mapped the same way (format 2), the line store's since() streams, and the Atlas rows sort a small key instead of the rows. linux, production arguments, 2 threads: - warm (nothing new): 599-885 MB -> 118 MB peak anon, 5.9 s - +17,137 commits: 885 MB -> 453 MB peak anon, 16.5 s -> 12.4 s - first Build on an old cache (writes the columns once): 13 s, 498 MB react warm: 34-40 MB -> 9 MB peak anon, 0.25 s
Every git handle kept up to 64 MB of objects and 128 MB of deltas, and each walk, line and blame thread its own 48-96 MB, all filled by a long Build and held to its end. They are now 1-8 MB each; speed is within measurement noise except on a cold walk of linux. 2 threads, before -> after: - linux +17,137 commits: 453 -> 224 MB peak anon, 12.4 -> 12.8 s - linux cold, no lines: 1,100 -> 705 MB, 181 -> 200 s (same CPU time) - react cold with lines: 558 -> 169 MB, 34.1 -> 33.4 s
The summary's people now come straight from the contributors and their lines, without working out directory ownership it never shows; the contributors are counted per person, with one exact-size list of (person, day) for active days instead of a growing list of every commit; directory ownership counts pairs in a map instead of a list. Same output. linux warm (nothing new), production arguments: 118 -> 94 MB peak anon, 5.9 s.
The Ownership kept per head was decoded whole, and the next one built whole beside it. Its file now holds the commits sorted by id, the files sorted by path and each file's owners as varint runs, mapped and read in place; a new one is written file by file, owners going to disk as they come, then laid out and mapped. Carried-over files copy their runs renumbered, without hashing. The moves found on the way live in a small file beside it. Survival looks commits up by sorted id keys instead of two maps over every commit, and a blame walk holds 32 MB of head text instead of 256 MB. Same counts as before (react: 1,997 people, 827,110 lines). 2 threads, before -> after: - linux everyone from nothing: 2 runs of 30 + 29 min, 6,467 CPU-s, 1,577 MB peak anon -> 18 min, 2,078 CPU-s, 443 MB - linux everyone at a head 17,137 commits later: 327 s, 967 MB -> 305 s, 428 MB; the same head again: 4.7 s, 223 MB - react everyone from nothing: 18.5 s, 508 MB -> 16.4 s, 173 MB - linux Ownership file: 71 MB -> 57 MB, plus 2.3 MB of moves
Packs are now format 3: the bincode of everything but the Index's long columns, then the commits, changes, subjects and Mainline arrival times each as zlib of their fixed records, with a table of the sections at the end. The engine unpacks one once, streaming, into a page-aligned file in its pack folder and maps it, so the columns are read in place and the memory budget counts only what is left on the heap. Packs of format 2 already in R2 are decoded as before. linux pack, before -> after: - R2 object: 103.8 MB -> 106.9 MB; on the engine's disk 275 MB unpacked - engine anon after the first answer: 338 MB (580 MB while decoding) -> 117 MB; after the screens 463 -> 228 MB; per pack on the heap 52 MB - answer times unchanged (overview all, cold: 3.1 s both)
The code map found each folder's and file's top person from a list of every (node, commit) pair a Window's changes reach, millions on linux. It now counts (node, person) pairs in a map and keeps the best of each node, most commits first and the lower person id on a tie, as before. Same answers. linux, the standard screens of one head: 249 -> 187 MB peak anon, 3.5 s. The engine works one Window's shared answers out the same way.
…m first A Build now works out its head's standard screens ahead, in a process of its own (`commitscape screens`, reading the pack's columns in place): the stats, and the overview, activity, people and top of the map of the whole history, the last year and the last 90 days. The Builder stores them as one R2 object each under the Report's prefix before publishing it. The Site sends one of those questions from there, kept in memory like the rest and for good when pinned to the Report, and asks the engine service only for the others; a Report written before is asked once per question. The engine test checks every stored screen is the engine's own answer. linux: 13 answers, 2.1 MB of JSON, 3.5 s and 187 MB peak anon once per head; the engine needed 3.1 s and up to 168 MB more for the first of them on every cold pack.
The engine service reads every pack format a Builder of the same or an earlier release writes, and needs nothing the Builder's migrations add, so it goes first: a Builder that starts writing pack format 3 then never publishes a pack the running engine service cannot read.
How the cache keeps the Index's columns, how the engine service reads a pack in place, which screens a Build stores with its Report and how the Site sends them, the ownership counts as measured now, and the upgrade order: the engine service first, since only it reads the Builder's new packs.
Finding a commit's moves kept every deleted file's text beside its similarity spans, though only the spans are compared; on linux's large renames that was 104 MB at once. It now keeps the spans alone. A blame walk holds 16 MB of head text instead of 32 MB, finishing as many files in the same time. linux everyone from nothing, first 300 s, 2 threads: median 260 -> 235 MB anon, peak 389 -> 364 MB, 30,775 -> 31,041 files. The whole count still peaks near 440 MB, somewhere else in the walk.
The line pass never counts a file a repository leaves out (lockfiles, Generated Files) and the line store keeps no lines for it. When such a file stopped being left out (its attributes changed, or a new classifier), the kept columns' flip restored its lines from the store, which had none, and a pass without columns took the store's record as it was: the file's changes kept zero lines for good. The store now keeps the paths it was counted without beside it (lines.left-out); the kept pass marks a commit whose newly kept file was never counted as not settled and counts it again, and the pass without columns counts again every commit the store left uncounted for a file it now keeps. Tests: a file whose linguist-generated mark is dropped has its lines counted on the next warm build as a cold one counts them (3 -> 6); the kept-columns oracle, which flips c.rs to generated and back, agrees.
The Builder made each pack's Universe Atlas in Node, from 170 MB of rows `build` wrote beside linux's pack, taking 635 MB: a second copy of the encoder outside the engine. `commitscape atlas PACK --out DIR` now encodes the Atlas column by column from the pack read in place, as the Site's atlasOf/encodeAtlas/encodeNames do, and packs it with brotli (quality ATLAS_QUALITY, window 24). The Builder runs it under the memory gate instead of `node builder.mjs atlas`, and `build` no longer writes universe.jsonl. The browser's decoder and the Site's fallback for packs without an Atlas stay in TypeScript. linux (1.37M commits on the Mainline, 146k files): 635 MB / 4.8 CPU-s -> 98 MB / 2.1 CPU-s (2.9 s), the same 7.8 MB map; react 13 MB, 0.2 s. The estimate the memory gate reserves goes from 640 to 108 MB. Tests: on every fixture the map and names are byte for byte the TypeScript encoder's, and the packed map unpacks to them; checked by hand on react (470 KB map) and linux (15.9 MB map), identical.
Three places held memory no gate counted: - A lean clone's prefetch kept up to 512 MB of git's output and split it into lines; it now reads each line as git writes it, holding only the blob ids it keeps. - `git clone` and `git fetch` ran beside the engine whatever they took. They now say what they take and wait in ENGINE_MEMORY_MB's line: a whole clone 64 MB + 0.3 MB per MB of repository (react's index-pack measured 82-92 MB for a 150 MB pack; linux's estimate, 1.8 GB, runs alone), a blobless one a third of that, a fetch 64 + 0.02 per MB. rev-parse and reset do not wait. - The Site's cache of answers read from R2 counted their JSON text but held them parsed, several times larger; it now holds the bytes it counts and parses them on each read, so a reader can no longer change what the next one gets either. Tests: a clone, a fetch and a blobless clone say their memory and the other git steps do not; an answer read twice is fetched once and comes back unchanged after the first reader changed its copy.
A cold Build published its quick pack (without lines) and then made its Atlas and standard screens, which the full pack made again minutes later. The quick publish now skips both, so the first Report shows that much sooner (linux: about 2.9 s of Atlas and 3.5 s of screens), and the Site never makes an Atlas for a quick pack either: it says the Universe is drawn once the Build has read the whole history, rather than reading the pack's rows to make one itself. A quick pack's screens are the engine service's answers, as before. Tests: a cold Build makes the Atlas and the screens once, from the full pack; the Site neither warms nor makes a quick pack's Atlas and does not ask the engine service.
A save sealed and uploaded its chunks one after another, and a restore fetched them one after another, so every chunk paid R2's round trip (about 350 ms from venus to the bucket in Asia-Pacific) in turn. Up to four now go at once; a restore still checks each chunk against its name and writes them in order, and holds at most four. Estimate for linux's 504 MB State, about 200 chunks: a restore of about 70 s of round trips becomes about 18 s, a first save the same. Test: with storage that answers in random order, at most four chunks move at once (and more than one), and the files come back the same.
When several requests asked at once for an answer the Site had not read from R2 yet (a page's screens, a shared Report opened by many), each read it from the bucket and unpacked it. Readers now wait on the read already in flight and share its bytes. Test: three readers at once cause one read of the bucket.
A Report's standard screens were 13 R2 objects under its prefix, read by the Site from the bucket in Asia-Pacific (about 350 ms a read from venus) and kept by the Site itself. The Builder now stores them as one gzipped object beside the full pack (<pack>.screens); the engine service fetches it with the pack, keeps it on its volume next to the unpacked pack, deletes it with the pack, and answers those questions from it without working them out, cut by `top` as any answer is. The Site asks the engine service for every screen, with the pack's ETag, so a standard screen no longer costs the Site an R2 read, and the Site's own path for stored screens is gone. Tests: the engine service answers a stored overview and a stored people list cut to `top` from the object beside the pack, works out what is not stored, and keeps the object on its volume; the Builder stores one gzipped object; a pack published again at the same head is answered from the new pack with a new ETag.
Every insert into the Site's size-bounded caches measured every value it held again to know whether to drop one, so a full cache of a few thousand answers cost a few thousand measurements per insert. The caches now keep each value's size and a running total; a value changed in place (an answer once packed) is measured again on its own, and every value is measured again once every 256 inserts, for packed bodies that gain another encoding later. Tests: values go least recently used first once past the budget, a refreshed value is counted again, deleting one takes its size off, and one value larger than the budget stays until another comes.
When the kept columns could not take a Build's new commits, the Index merged them in memory, and `Column::vec_mut` copied every mapped column onto the heap without a word: on linux the whole history, several hundred MB, in a run sized for a mapped one. The merge now runs inside a `history_copied` span with the commits and changes it copies, which the Builder's events carry to its traces, and `vec_mut` says what it costs on a mapped column.
DEPLOY.md gains a table of what each server holds at most, row by row, for mercury's 3 GiB Builder container (the Builder, the runs ENGINE_MEMORY_MB lets in, git data and the head check: about 1.5 GB) and for venus (the Site, its caches and Atlases, the engine service, Postgres: about 1.7 GB), measured on linux where it could be and marked as estimates where not. ENGINE_MEMORY_MB's figures are now the estimates the Builder uses, Atlas, screens, fetches and clones included. ADR-0031 lists the readers kept for this release only so the deploy and the one rebuild work (format 2 packs, Reports in R2 without packs, the Site's Atlas fallback, the cache folder move, the per-file arrays of index_commits), which go with step 3.2. docs/working.md says the landing page is a Cloudflare Worker.
The Repo State in R2 can lag Postgres: Postgres loads each export as soon as the engine has run, the State is saved after both phases have published, and a failed save is only logged. A cache brought back from that older State kept its lineage, so the next Build appended onto the rows Postgres held; but it handed out signature and file ids in its own walk order, and the loader replaces index_people while keeping the old commit rows, so commits showed the wrong people without an error. Each cache now keeps the generations it passed through (index.generations, beside index.current), and an export's mark names the generation it was read from. A run appends only when its cache descends from the generation of the mark it is handed; any other cache, or a mark without one, writes a full export. Caches and marks from before this release write one full export, then append as before. Test: the critic's repro (build, keep the cache, commits by pat then quinn, bring the kept cache back, one more commit): "feat: p" showed quinn; it now shows pat, the fourth export is full and the next one appends again.
The column files, the commit graph and packs were read and written at offsets with std::os::unix::fs::FileExt, unguarded, so the CLI no longer compiled for Windows, where CI tests it. Positional reads and writes now go through commitscape-map (FileExt on Unix, seek_read and seek_write on Windows, every caller either positional only or seeking before reading in sequence). Windows neither renames nor deletes a file while a view of it is mapped, and a Build replaces files it still has mapped (the previous Ownership, the commit graph, a merged column file). On Windows commitscape-map therefore reads each stretch into memory instead of mapping it, and a column file's trailer no longer resizes the file when its length does not change. Checked: cargo check and clippy -D warnings for x86_64-pc-windows-msvc on every crate that does not pull in ring (core, map, index, metrics, report, forge, xtask); the CLI and the engine service stop only at ring's C build, which this machine cannot compile for MSVC. Test: bytes written at an offset read back from it, and a read past the end fails.
A full load pointed repositories.index_id at the new Stored Index, which had no pack yet, and deleted the old one, and the Builder deleted the old pack from R2 right after the load. Until the new pack was published the Report was blank, and if the publish failed there was no Report at all; every private Build is a full load. A full load now leaves its Stored Index ready but not live. The one publish transaction records the pack, makes that Stored Index live (makeLive moves index_id and deletes the repository's other Stored Indexes with their rows) and writes the Report; the old packs, screens and Atlases are deleted from R2 only after it has committed. Appends still go into the live Stored Index as before. Tests: a full Build whose pack cannot be uploaded leaves the old Report, Stored Index and pack as they were (with the old behaviour the test finds two Stored Indexes and the old Report gone), and the next Build replaces them whole; against Postgres, a full load is not live until makeLive, which returns the replaced pack.
When the Builder's Atlas failed, the Site made it itself from rows the engine service read from the pack: about 1.16 GB on the Site and 450 MB in the engine service, on the server that serves pages. A Build whose Atlas fails now publishes with the Atlas pending and queues an Atlas-only job (`atlases`), which fetches the pack from R2 and runs `commitscape atlas` under ENGINE_MEMORY_MB like a Build; the next full Build makes its own anyway. The Site only sends stored Atlases: for a pack without one, such as one published before this release, the Universe says the history is read at the next Build and the Site queues the same job, once per pack. The engine service's /v1/universe/rows, which nothing else needs, is gone, and its questions are taken out of the goldens with every other answer left as it was. Tests: a failed Atlas is queued with its Stored Index and pack; the job makes the Atlas of the live pack once and leaves alone a pack no longer live or a quick one; the Site sends a stored Atlas without asking the engine service, and for a pack without one answers 404 and queues the job once.
…tory Each Build worker ran tidy (prune, sweep, and since the last round a repack) before taking its next Build, so a long repack held that lane. Tidy now runs on its own: after a Build without being waited for, on its timer, one at a time. prune deletes a repository's clone or cache only while holding that repository, and leaves any a job holds, and each write of the record of folder sizes has a temporary file of its own. Test: a held repository's clone and cache are left while the others go, and are pruned once it is let go; two prunes at once leave no temporary file behind.
The first Build of each repository after the upgrade that brought column files writes its Index's column file whole (linux: 498 MB), and the deploy's `rebuild` sends every repository through it, but the memory gate reserved it as a warm Build (linux: 228 MB). A cache whose current generation has no column file yet is now reserved as its own kind: 64 MB + 0.3 MB per thousand commits (linux: 514 MB). Tests: a Build of such a cache asks for that estimate, one whose column file is there does not; the estimate is above the 498 MB measured.
A count of Surviving Lines that met one file version its blobless clone lacked was queued again on a full clone, which for linux means cloning everything again; it now fetches the versions it names and counts once more first. A count reads the lean clone only once a lean Build has deleted the full one, so a lean clone made early is never read before its history's versions were fetched. A cold Build keeps its quick blobless clone as the repository's lean clone instead of throwing it away and cloning blobless again when it turns lean. Tests: a count lacking one version fetches it and records on the second try without a full clone; a lean clone beside a full one is not counted from; a cold Build leaves the quick clone as lean and a later lean Build fetches into it, while a private one keeps nothing.
A Builder deployed before its engine service wrote packs of format 3, which the earlier engine service cannot read: the deploy order was a rule in DEPLOY.md and nothing more. The engine service's health answer now lists the pack formats it reads, and a Builder given ENGINE_URL asks it (at most every five minutes) before each engine run and has `commitscape build --pack-format 2` write format 2 while the engine service reads no newer one or cannot be asked. Format 2 is the bincode of the whole pack, as the engine service of the release before reads it; the Pack struct is the same as that release's. Tests: a pack written in format 2 reads back as the same pack and is refused by unpack as format 2; the engine service's health answer lists 2 and 3; the Builder writes format 2 when told so and format 3 otherwise, and asks for 3 only when the engine service says it reads it.
ADR-0031 listed what is kept only so the deploy and the rebuild after it work, and said it goes "in the next release". It now lists each reader with where it lives: packs of format 2 (read, and written for an older engine service), Reports stored in R2 before Stored Indexes had packs, caches in a clone's folder and older Builders' folders, export marks without a cache generation, and the per-file arrays of index_commits. Each is kept only in the first release after 0.4.0 and removed in the release after it, with step 3.2.
In production the Builder runs on another server than the engine service, which is reachable only inside its own Docker network, so the Builder had no ENGINE_URL that worked, and an engine service it could not ask made it write format 2: packs of the older format for good, and none of the gains of reading packs in place. When it starts, the engine service now writes engine/formats.json beside the packs (to R2, or into the folder it reads packs from) with the pack formats it reads, its version and when. The Builder writes the newest format the engine service reads, worked out at most every five minutes: PACK_FORMAT when set overrides everything; then the engine service's health answer, where ENGINE_URL reaches it (one that lists no formats is of the release before, so format 2); then engine/formats.json; else format 3, also when R2 cannot be read. An engine service that cannot reach it or reaches it unanswered no longer means format 2. The Builder logs the format and what decided it whenever either changes. Tests: the engine service writes the marker with formats [2, 3] and its version; the format comes from PACK_FORMAT over the health answer over the marker over 3, R2 missing or failing gives 3, a marker listing only formats the Builder cannot write gives 3, and an unreachable engine service leaves the marker to decide.
… change Which files' Lines Changed count (not generated, vendored or lockfiles, as the head says) can change on any commit: a .gitattributes change, or a file that gains a generated header. The cache and the pack then hold new line counts for commits Postgres already has, but the mark held, so an append export sent only the lines counted since, and the loader ignored lines for a commit it already had (ON CONFLICT DO NOTHING). The Site kept the old totals until the next full export. The mark's fingerprint now covers the files left out, by id and path (a generated file renamed away and another made at its path is a change), so when they change the mark no longer holds and the export sends every commit's lines again, as it does for a lost mark; the loader replaces the totals it is sent, counting only those that differ. Found by the differential test of incremental against clean builds (seeds 4 and 158). Tests: an append after a file turns generated sends the earlier commit's lines again, (5, 0) -> (2, 0), and fails without; the fingerprint changes with the files left out and a file's id; lines sent again replace what Postgres held, and the same lines change nothing (real Postgres).
The line store keeps the paths it was counted without (lines.left-out), and a commit whose change of a now-kept file it holds as uncounted is counted again. It matched only a file's current path, so a vendored file moved out of vendor/, or a file whose move into vendor/ a force push dropped, kept no lines in the cache, the pack and the Report for good, whenever the Build did not read the kept columns (which track files by id). It now records every path a left-out file has had, and matches any path a kept file has had; a wrong match only counts a commit again. Found by the differential test (seeds 186 and 313). Tests: a vendored file moved out, and one whose move into vendor/ was dropped by a force push, have their lines counted as a cold build counts them, (3, 0) instead of none; both fail without.
When the commit classifier changes, the cache classifies its commits again, but kept its lineage, so the next export appended: it never sends rows of commits Postgres already holds, and their kinds stayed the old ones until a full export. A reclassified cache now takes a new lineage, so the export after it is full, once per classifier change. Test: a cache classified by older rules is classified again without walking, under a new lineage that the next warm load keeps.
Each run left a folder of about 49 MB in the system's temporary folder: 4.6 GB on the development box after a day of runs.
A differential test of the incremental path. It scripts random histories (commits by six people with an alias and a bot, renames, merges, force pushes, .gitattributes and .mailmap changes) and random operations (a build killed partway, the cache brought back from any earlier Repo State), loads every export as the Builder does into a model of Postgres (appends need the live lineage, line totals replaced, Mainline updated), counts everyone's Surviving Lines, and compares the loaded rows, the pack, meta.json and Surviving Lines with a clean build of the same history, anchored at a fixed time. Ids are handed out in the order a cache first met them and must stay fixed within a lineage, so a clean build numbers them differently; the test compares them by what they name (a signature by its name and first commit, a person by its signatures, a file by its path and birth), and when they come out the same the pack and meta.json must be the same bytes. Folders no file is in, which nothing reads, and colour ties, broken by person id, are not differences. 8 seeds of 30 steps run with cargo test in 6 s; the ignored test runs SEEDS (200) from SEEDS_FROM (100), 40 steps each, in about 4 minutes. Seeds 100-499, 1000-1199 and 2000-2199 pass. Its git runs without automatic maintenance (gc.auto=0, maintenance.auto=false), as the Builder's do: git 2.54 otherwise repacks in the background after commits and objects go missing under the test.
1.7.6 is open to two advisories: OAuth state used as a magic link to sign in as another user (GHSA-965c-763c-88jm, critical) and the OAuth proxy letting an attacker sign in as another user (GHSA-r4xp-prcw-77qf, high). Both are fixed from 1.7.7.
The blobless clone test checks its clone out, which fetches the missing file versions from the fixture. On Windows that fetch's upload-pack did not get the setting the clone was given, so it refused the filter. The tests' git now hands uploadpack.allowFilter and allowAnySHA1InWant to every git it starts.
On Windows, git does not fetch a blobless clone's missing file versions from a file:// remote, with or without the remote's filter settings, so the two tests that check a blobless clone out cannot set themselves up there. Blobless clones are the Builder's, which runs on Linux; the tests still run on Linux and macOS. The environment the tests' git was given did not help (git does not pass it to another local repository) and is gone again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Build now costs what changed since the last one, not how long the history is, and nothing the Builder runs on linux goes above 256 MB. ADR-0030 and ADR-0031 have the design;
DEPLOY.mdhas every setting and the upgrade steps.What changes
git ls-remote, and ends there when nothing moved. Seeds ask GitHub's GraphQL API for a hundred heads at a time, and are built only when their head moved and someone viewed them since, or weekly.CACHE_BUDGET_GB. WithSTATE_KEYset, it is also kept sealed in R2: zstd, AES-256-GCM, chunks cut by their content, only new chunks uploaded. A lost disk restores it instead of reading the history again.index_commit_lines), so filling them in never rewrites commit rows.ENGINE_MEMORY_MB) instead of one at a time;index_commit_lines(with a backfill),stored_indexes.markandbuild_format,repositories.checked_at,report_headandreport_refs, andrepo_states.Measured (two cores, peak anonymous memory)
ls-remote, ~40 ms CPUChecks
cargo fmtand clippy clean;cargo test --workspace: 338 passed.packages/serveragainst Postgres and S3.Deploying
node builder.mjs rebuildonce, so Reports from earlier releases are built again.No setting is required.
STATE_KEYturns on the R2 tier for Repo States;PACK_FORMAT=2rolls back to the old pack format.The readers kept only for this upgrade (format 2 packs, R2-stored Reports, the old cache folders, the unwritten per-file arrays) go in the release after this one; ADR-0031 lists them.