Skip to content

README.md

Latest commit

 

History

History
374 lines (263 loc) · 22.8 KB

README.md

File metadata and controls

374 lines (263 loc) · 22.8 KB
Raw

Backscroll

CI Go License: Apache 2.0

Backscroll turns your coding-agent sessions into a permanent, searchable record of what happened.

Every command, error and decision — searchable across every assistant, and still there after the session files expire.

Your agents write down everything they do, then throw it away. Claude Code prunes its session files after about a month; each assistant stores them in its own format, in its own directory, with no way to search across them. The work is recorded and unreachable at the same time.

Backscroll indexes those sessions into SQLite and keeps them. Ask what a command returned six weeks ago, which errors keep recurring, or where a decision was made — and get an answer whose source file no longer exists.

Table of Contents

Installation

Backscroll ships as a single self-contained binary: pure Go, built with CGO disabled, so there is nothing to install alongside it. (Whether that binary ends up statically linked depends on the platform, so this makes no promise about linkage.) Runtime input manifests are separate user configuration files loaded from <config_dir>/backscroll/inputs/*.inputs.toml.

Install Script (Recommended)

curl -fsSL https://raw.githubusercontent.com/pablontiv/backscroll/master/install.sh | bash

Detects your platform (Linux x86_64 / macOS aarch64), installs the binary to ~/.local/bin/, and runs backscroll presets install. The native command installs the shipped Claude, Pi, OMP, decisions, OpenCode, and Codex presets without overwriting existing manifests. The script passes BACKSCROLL_CONFIG_DIR through --config-home when the variable is set. Otherwise the native command uses XDG_CONFIG_HOME or $HOME/.config.

Windows (PowerShell):

irm https://raw.githubusercontent.com/pablontiv/backscroll/master/install.ps1 | iex

Installs the binary to %LOCALAPPDATA%\backscroll\bin\, adds it to your PATH, and runs backscroll presets install. The native command installs the shipped Claude, Pi, OMP, decisions, OpenCode, and Codex presets without overwriting existing manifests. The Windows installer passes BACKSCROLL_CONFIG_DIR through --config-home when set and otherwise uses <user-home>\.config. Compatible with Windows PowerShell 5.1+.

Install input presets

Backscroll incorporates the Claude, Pi, OMP, decisions, OpenCode, and Codex input presets. Install missing presets with the native command:

backscroll presets install

The command writes to <config-home>/backscroll/inputs/. It preserves every existing file by default. Use --config-home /absolute/path to select a config home. Use --overwrite only when you intend to replace the six managed names.

backscroll presets install --config-home "$HOME/.config" --overwrite
backscroll validate
backscroll config

The command uses XDG_CONFIG_HOME when it is set and absolute. Otherwise it uses $HOME/.config. BACKSCROLL_CONFIG_DIR remains the runtime application override. The Unix bootstrap translates that variable to --config-home for compatibility. Pi subagent sessions remain opt-in; see the Pi input contract. OMP has a dedicated native v3 reader and excludes nested child sessions by default; see the OMP input contract. For Codex records and limits, see the Codex input contract.

backscroll presets install --config-home (Join-Path $HOME ".config")
backscroll validate
backscroll config

From Source

go install github.com/pablontiv/backscroll/cmd/backscroll@latest

To install the exact merged local commit, rather than a published version, use the project-local install and build-revision verification. This does not create a release or publish artifacts.

Proof

# Index everything (runs automatically before any query)
backscroll status

# What did we decide about the migration?
backscroll search --text "migration plan" --all-projects

# What did that command actually return?
backscroll search --text "go test ./..." --all-projects --content-type tool

# Search supplies the selectors for exact context
backscroll search --text "$QUERY" --all-projects --json --fields minimal

# Use a non-null UUID; for uuid:null use the exact source path and ordinal
if [ "$UUID" != "null" ] && [ -n "$UUID" ]; then
  backscroll context --uuid "$UUID" --before 5 --after 5
else
  backscroll context --source-path "$SOURCE_PATH" --ordinal "$ORDINAL" --before 5 --after 5
fi

# Which errors keep coming back?
backscroll patterns --kind templates --min-support 5 --all-projects

That last one answers a question search cannot. Search finds what you can already name; the census counts what recurs across every session you have:

Found 20 templates (min_support=5):

1. [36b55122]
   Text: error: Exit code <*>
   Occurrences: 344
   Projects: [backscroll rootline picokit crossbeam dotfiles ...]

Core Concepts

The database is the record, not a cache. Session files expire; indexed sessions do not. Once a file is gone from disk, sync never walks it again, so everything indexed from it stays — that is what makes the record outlive its source.

How a file is re-synced depends on whether its messages carry identity. Sessions whose messages have a uuid — Claude Code, from schema v8 onward — sync append-only: rows already keyed by uuid are left alone and their ids stay stable. The one exception is a one-time cleanup. If the same file was indexed before v8, those older rows carry no uuid, and re-parsing would duplicate the whole file; they are deleted once, on the first re-sync after the upgrade, and never again. Sessions without one, which today is most of the corpus, are wiped and reloaded on every re-sync, so their row ids are not stable and edits to a live file replace its rows wholesale. Either way, lifecycle replacement only occurs while the source exists; retained records have no deletion path.

Unchanged bytes are skipped only while their versioned ingestion-semantics fingerprint also matches. Changing the decoder (including its replay epoch), manifest source, or reasoning option schedules an available file for a bounded startup reparse. Expired or unavailable files and their indexed rows remain untouched.

Every assistant, one index. Claude Code, Pi, OMP, OpenCode, and Codex each store sessions differently. A dedicated reader per format normalizes them behind one schema, so you search content, not file layouts. Input manifests select a registered reader and its discovery roots; adding a new provider format requires a reader implementation, not only a manifest. Message origin is likewise parser-backed: Backscroll uses native structured actor and tool boundaries, never message text or a historical stored role. An indexed message without sufficient evidence is unknown. Startup reparse can enrich a historical unknown only while its configured source remains available to its reader. Recovery can also enrich unknown by merging a compatible duplicate with proven origin; contradictory proven origins are rejected.

Conversation and tool activity are indexed separately, on purpose. Prose goes to an FTS5 index with a Porter stemmer, so "migrating" finds "migration". Tool text — commands, paths, errors — goes to a trigram index, where an exact substring like internal/storage/sync.go matches. An unfiltered query merges both by rank position, which is why a search never has to pick one.

Exact context, discovery, and census are different questions. context retrieves the immediate indexed neighborhood of one exact UUID or exact source-path-plus-ordinal anchor. search discovers ranked records from terms and filters. patterns computes a census: it counts commands, failures, recurring error templates, correction candidates and repeated tool sequences across the whole corpus. No individual document contains a pattern, so ranking cannot surface one.

Error-template line selection shares one policy for Bash and generic tools: trim whitespace, retain error/fail/panic/denied/fatal matches in order (including repeated matches), then include the last non-empty line if it is not already selected. Explicit Go extraction uses its separate compiler/test/panic policy.

What You Can Do

Recover what happened

Ordinary snapshot-read, metadata-read, and mutation owners validate active manifests and attempt one incremental sync before executing. Busy readers may follow the last committed snapshot; recover skips preparation/pre-sync and syncs only after installing a replacement. See startup classes and exceptions. Session, plan, and Markdown files are ingestion inputs; SQLite is the perennial record used by search, context, list, patterns, status, and validate. Use search to discover relevant records. Every search result exposes its stored path, nullable UUID, and ordinal. If uuid/UUID is not null, pass that UUID to context. If it is null, pass the exact stored path and ordinal instead.

backscroll search --text "$QUERY" --project "$PROJECT"     # this project
backscroll search --text "$QUERY" --all-projects            # everywhere
backscroll search --text "$QUERY" --content-type tool       # commands, paths, errors
backscroll list --order timestamp:desc --limit 10             # recent sessions
backscroll search --text "$QUERY" --source-path "$SOURCE_PATH" --all-projects --json --fields minimal
if [ "$UUID" != "null" ] && [ -n "$UUID" ]; then
  backscroll context --uuid "$UUID" --before 5 --after 5 --json
else
  backscroll context --source-path "$SOURCE_PATH" --ordinal "$ORDINAL" --before 5 --after 5 --robot
fi

A context UUID is opaque and is never synthesized for a result that lacks one. The alternate selector requires the exact stored source path and ordinal; duplicate rows at that coordinate fail with context_ambiguous. Defaults are five records on each side (maximum 50 each); record text is capped at 4000 Unicode code points. The complete successful payload defaults to a 2000-token budget (--max-tokens, 64–16384). See the exact context contract.

Filters worth knowing: --after / --before for a date window, --tag for auto-detected session categories (debugging, refactoring, testing…), --source-path to pin one stored input path, --source to keep one source class, and --role to filter the stored role field. The role field is not a semantic-origin guarantee.

Search is lexical: prose uses Porter-stemmed BM25, tool text uses trigram BM25, and unfiltered searches fuse those two lexical rankings with RRF over at most 200 eligible unique non-echo candidates per FTS stream. --limit/--offset page that bounded merged pool, not every match. A proven 201st eligible candidate produces robot candidate_limit=200 / candidate_limit_reached=true metadata (stderr fallback when it cannot fit); text and JSON report the cap on stderr without changing stdout. Typed --content-type searches retain direct SQL pagination. See the search contract. --relax optionally drops low-IDF terms after a strict zero-result search without changing scope or introducing semantic expansion.

Discover what recurs

backscroll patterns --kind commands     # what runs most
backscroll patterns --kind failures     # what breaks, with exit codes
backscroll patterns --kind templates    # recurring error shapes
backscroll patterns --kind sequences    # workflows that repeat
backscroll patterns --kind sequences --after 2026-01-01 --before 2027-01-01 --json
backscroll patterns --kind corrections --origin human  # where you corrected course

--trend buckets commands and failures by week. --min-support N sets how many occurrences make a pattern. Sequences also take --min-length / --max-length. Sequence --after is inclusive (>=) and --before exclusive (<): the example selects timestamped events in [2026-01-01, 2027-01-01) before mining, excluding events with unknown timestamps. See sequence date filters.

Correction candidates are detected deterministically, never by a model, and are meant to be labelled. --origin human|assistant|system|automation|unknown is an opt-in corrections-only filter; it is applied before pagination, and omitting it preserves the existing population and output shape:

backscroll patterns --kind corrections --origin human --pending --batch 50 --robot
if [ -n "${UUID:-}" ] && [ "$UUID" != "null" ]; then
  backscroll annotate --uuid "$UUID" --kind correction --label "$LABEL"
else
  backscroll annotate --path "$SOURCE_PATH" --ordinal "$ORDINAL" --kind correction --label "$LABEL"
fi

Labelled candidates drop out of --pending, so the loop resumes wherever it stopped.

Keep the index healthy

backscroll --version         # deployed version; dev builds report dev
backscroll config --json     # effective database, session_dirs, and active inputs
backscroll status            # size, counts, last sync (not build identity)
backscroll validate --json   # prepared-index integrity, not input inventory
backscroll rebuild           # re-derive search indexes from the database

For exact development-build provenance, use go version -m "$(command -v backscroll)" and inspect vcs.revision/vcs.modified; --version is not a commit identifier.

For stranded indexes, first inspect backscroll recover --from "$STRANDED_DB" --dry-run, then intentionally apply without --dry-run. The configured active database must already exist. Recovery preserves an active backup, but a post-install sync error leaves the replacement installed; it is not automatic rollback. Follow the recovery runbook, including backup restoration. Recovery apply is supported on macOS and Linux when the filesystem supports the required atomic no-clobber moves. On other platforms, including Windows, dry-run can inspect compatible inputs, but apply fails at the unsupported move before mutating the active database; the stranded input remains intact. Automatic migration snapshots are sibling *.snapshot-v<from>-<signature12>-to-v<to> files with collision suffixes and two validated snapshots retained; they are distinct from recovery backups. See snapshot retention.

rebuild operates after the mandatory root startup sync has already prepared the database. The handler does not perform a second sync: it re-derives search indexes from stored rows, re-derives templates/correction signals/tool-event satellites where possible, and re-resolves project identities. Sessions that vanished from disk survive it untouched.

Output for whoever is reading

Default output is human-readable text. Machine modes keep stdout parseable: human progress and warnings go to stderr, JSON/robot startup progress is discarded, and structured diagnostics remain parseable.

--json is available on search, context, list, patterns, status, validate, and config. JSON mode on context emits one complete envelope with anchor, records, truncated, and omitted. --robot is available on search, context, list, and patterns; context robot mode emits line-oriented anchor, envelope, and record fields. rebuild and annotate report in plain text only.

Search identity is present in every density and output mode. Text uses Path, UUID, and Ordinal; minimal JSON uses source_path, uuid, and ordinal; full JSON keeps the model casing FilePath, UUID, and Ordinal; robot output uses result_N_filepath, result_N_uuid, and result_N_ordinal for both --fields minimal and --fields full. A missing UUID is null, not a generated value.

On search, --fields minimal|full controls machine-output density, while --max-tokens N applies the truncation budget to text and robot output; JSON has no documented truncation contract. On context, --max-tokens N budgets the complete successful payload; diagnostics are exempt.

Optional: Agent Integration

Backscroll is a CLI. Nothing requires an agent, and there is no MCP server — a CLI call costs a fraction of the tokens an MCP tool schema does.

Agents use the same commands with --robot --fields minimal --max-tokens N. The discoverable .claude/skills/backscroll/SKILL.md is a minimal shim. It tells the runtime to execute backscroll --skill and follow stdout. cmd/backscroll/backscroll_skill.md is the authoritative instruction source. backscroll skill and the compatibility alias print those exact embedded instructions without ordinary startup effects.

Run backscroll skill --install to install the embedded shim for Claude, Agents/Codex, and OpenCode. The three links point to one immutable managed copy. Unknown destination content requires a reviewed preimage digest. The installer backs up replaced content and returns a durable receipt for backscroll skill --restore RECEIPT --approve SHA256. See skill installation and restoration. Git hooks do not install skills or presets.

Configuration

Backscroll separates application configuration from input configuration.

  • Application config (backscroll.toml) controls where the database lives. By default, Backscroll creates an index at ~/.backscroll.db.
  • Input config (*.inputs.toml) controls ingestion during ordinary owner startup. The canonical runtime location is <config_dir>/backscroll/inputs/*.inputs.toml, where <config_dir> is BACKSCROLL_CONFIG_DIR when non-empty, otherwise $HOME/.config on every OS (including Windows <user-home>\.config). XDG_CONFIG_HOME, macOS Application Support, and %APPDATA% do not select this directory.

Override app settings by creating ~/.config/backscroll/config.toml or backscroll.toml in the current directory:

database_path = "/home/user/.backscroll.db"

Environment variables are also supported:

export BACKSCROLL_DATABASE_PATH="/tmp/custom.db"

Input manifests are declared as:

version = 1

[[inputs]]
id = "claude"
source = "session"
active = true

[inputs.discover]
roots = ["/home/user/.claude/projects"]
include = ["**/*.jsonl"]
exclude = ["**/subagents/**"]

[inputs.decode]
format = "claude"

A manifest declares where to find inputs (discover) and how to decode them (decode.format). Built-in readers are claude, pi, omp, opencode, codex, markdown_document (one whole-document record), and markdown_sections (records split on ## headings, with whole-document fallback). The repository presets (inputs/*.inputs.toml) are examples to install into the global input directory; Backscroll does not read that repository directory at runtime. Use backscroll config or backscroll config --json for the input inventory; validate checks prepared-index integrity instead. See the authoritative Markdown input contract for manifest examples and split behavior.

See Configuration docs for the full resolution order and all options.

Documentation

Topic Description
Sync & Indexing Incremental sync, noise filtering, project detection
Search Engine BM25 ranking, output formats, token limiting
Exact Context Exact anchors, neighboring rows, output and diagnostics
Pattern Discovery The five censuses, the classification loop, calibration
Retrieval Migration Discovery-to-context flow over perennial SQLite
Configuration Config resolution, TOML format, environment variables
Input Manifest Contract Supported fields and registered decoders for global *.inputs.toml files
Session Search Research Feasibility study: axioms, evidence tables, capabilities matrix

Development

just check              # gofmt --check + go vet
just test               # Run all tests
just fmt                # Auto-format code (gofmt -w)
just build              # Build binary
just coverage-summary   # Go test coverage report
just audit              # go mod verify

Git hooks (required, one-time per clone)

The versioned hooks in .githooks/ are not active until you point git at them:

git config core.hooksPath .githooks

Without this, git uses .git/hooks/ and every push skips:

  • the just ci aggregate-coverage gate when Go files change, and
  • the AGENTS.md and documentation update validation.

Once activated, pre-push runs those gates. post-merge updates documentation aggregates after a merge. Hooks do not install the executable, skills, or presets. Install a release with install.sh. Use the native commands for skill and preset installation.

Commits follow Conventional Commits (type(scope): description).

License

Apache License 2.0 — free for commercial and non-commercial use.

Automatic updates

Autoupdate is mandatory for valid executable release commands; there is no runtime opt-out. It starts after command/flag/argument validation and before startup/index work. Invalid invocations, help, version, and --skill never apply, fetch, or wait. Plain go build produces a dev identity, exempt in the private updater backend.

Apply and completed fetch failures emit one concise warning: autoupdate: ... on stderr without failing the command or changing stdout/JSON/robot output. Successful updates are silent. After command execution, staging waits up to 10s; if it has not completed, the command returns its normal result with a timeout warning. Cancellation returns promptly without reporting a fabricated network failure; all fetch requests are context-aware and canceled on cancellation or the staging cap, with a 60s HTTP timeout per request. The private backend preserves the pinned picokit Apache v1.0.0 release format while returning errors upstream swallowed; cancellation and the cap join bounded extraction/publication cleanup before return, with unique synced temporary files and conservative byte limits; see internal/autoupdate/README.md.