Skip to content

feat(search)!: adopt truthful lexical-only contract #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pablontiv merged 1 commit into from
Oct 15, 2026
Merged

feat(search)!: adopt truthful lexical-only contract #185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .coverage-floors.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@ default = 85
packages = [
"cmd/backscroll",
"internal/autoupdate",
"internal/chunking",
"internal/config",
"internal/embedding",
"internal/hybrid",
"internal/input_config",
"internal/models",
Expand Down
16 changes: 6 additions & 10 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Backscroll is a Go CLI tool that indexes Claude Code, Pi, OMP, OpenCode, Codex C

**Status**: Go port complete — `main` branch is the active Go implementation. The Rust implementation is frozen in the `v0` branch.

Implemented: `internal/autoupdate`, `internal/config`, `internal/input_config`, `internal/models`, `internal/readers`, `internal/sync`, `internal/tagging`, `internal/plans`, `internal/sources`, `internal/storage`, `internal/services`, `internal/projects`, `internal/templates`, `internal/corrections`, `internal/categories`, `internal/sequences`, `internal/chunking`, `internal/embedding`, `internal/hybrid`, `internal/compat`, `internal/recovery`, `internal/startuplock`. CLI commands in `cmd/backscroll/` (10 v2 commands via cobra).
Implemented: `internal/autoupdate`, `internal/config`, `internal/input_config`, `internal/models`, `internal/readers`, `internal/sync`, `internal/tagging`, `internal/plans`, `internal/sources`, `internal/storage`, `internal/services`, `internal/projects`, `internal/templates`, `internal/corrections`, `internal/categories`, `internal/sequences`, `internal/hybrid`, `internal/compat`, `internal/recovery`, `internal/startuplock`. CLI commands in `cmd/backscroll/` (10 v2 commands via cobra).
github.com/pablontiv/backscroll/internal/autoupdate — Private context-aware release staging and safe replacement

Stack: cobra, go-toml/v2, goldmark, modernc.org/sqlite (pure Go, no CGO), stdlib testing.
Expand Down Expand Up @@ -43,7 +43,7 @@ Tests use stdlib `testing` + subprocess or direct `run()` invocation. Unit tests
cmd/backscroll/
├── main.go — entrypoint; run(stdout, stderr, args) for testability
├── list.go — list command (v2: --project, --all-projects, --recent N, --order, --limit, --offset, --json, --robot)
├── search.go — search command (v2: --text, --project, --all-projects, --source, --source-path, --after, --before, --role, --content-type, --tag, --fields, --max-tokens, --lexical-only, --relax, --similarity-threshold, --json, --robot)
├── search.go — search command (v2: --text, --project, --all-projects, --source, --source-path, --after, --before, --role, --content-type, --tag, --fields, --max-tokens, --relax, --json, --robot)
├── context.go — exact indexed neighborhood command (--uuid or exact --source-path + --ordinal, --before, --after, --max-tokens, --json, --robot)
├── patterns.go — patterns command (v2: --kind commands|failures|templates|sequences|corrections [--origin human|assistant|system|automation|unknown] [--pending] [--batch N] [--trend], --project, --all-projects, --tag, --min-support, --min-confidence, --min-length, --max-length, --json, --robot)
├── annotate.go — annotate command (F3b: --uuid --kind --label; validates message existence; upsert semantics)
Expand Down Expand Up @@ -71,17 +71,15 @@ internal/
├── templates/ — F2 Drain-inspired miner: Miner, ProcessLine, ExtractErrorLines, deterministic signature via SHA256
├── corrections/ — F3 correction detection: bilingual lexicon, interrupt flags, denial heuristics, rephrase-similarity; detector registry + implementations
├── categories/ — F4 category map loader (rule engine, versioning, tool→category mapping) with embedded default preset
├── chunking/ — Token-aware text chunking helpers for embedding preparation
├── embedding/ — Embedding provider interfaces, mock provider, and ONNX provider implementation
├── hybrid/ — Reciprocal Rank Fusion helpers for merged lexical/vector retrieval
├── hybrid/ — Reciprocal Rank Fusion helpers for merged lexical retrieval
├── sequences/ — F4 PrefixSpan mining (deterministic discovery of frequent tool-call sequences per session)
├── services/ — Concrete typed read-only query orchestration over storage
└── storage/ — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v17, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences)
scripts/
└── recall-eval/ — isolated legacy/synthetic recall evaluator and cohort reporter
```

Ten v2 CLI commands: `list [--project] [--all-projects] [--recent N] [--order timestamp:desc|asc] [--limit] [--offset] [--json] [--robot]`, `search [--text <query>] [--project] [--all-projects] [--source] [--source-path] [--after] [--before] [--role] [--content-type] [--tag] [--limit] [--offset] [--fields minimal|full] [--max-tokens N] [--lexical-only] [--relax] [--similarity-threshold F] [--json] [--robot]`, `context (--uuid <opaque-uuid> | --source-path <exact-path> --ordinal <n>) [--before 0..50] [--after 0..50] [--max-tokens 64..16384] [--json] [--robot]`, `patterns --kind commands|failures|templates|sequences|corrections [--origin human|assistant|system|automation|unknown] [--pending] [--batch N] [--project] [--all-projects] [--tag] [--trend] [--after] [--before] [--min-support N] [--min-confidence F] [--min-length N] [--max-length N] [--limit] [--offset] [--json] [--robot]`, `annotate --uuid <u> --kind <k> --label <l> [--path <p> --ordinal <n>]`, `recover --from <path> [--dry-run]`, `status [--json]`, `validate [--json]`, `rebuild`, `config [--json]`.
Ten v2 CLI commands: `list [--project] [--all-projects] [--recent N] [--order timestamp:desc|asc] [--limit] [--offset] [--json] [--robot]`, `search [--text <query>] [--project] [--all-projects] [--source] [--source-path] [--after] [--before] [--role] [--content-type] [--tag] [--limit] [--offset] [--fields minimal|full] [--max-tokens N] [--relax] [--json] [--robot]`, `context (--uuid <opaque-uuid> | --source-path <exact-path> --ordinal <n>) [--before 0..50] [--after 0..50] [--max-tokens 64..16384] [--json] [--robot]`, `patterns --kind commands|failures|templates|sequences|corrections [--origin human|assistant|system|automation|unknown] [--pending] [--batch N] [--project] [--all-projects] [--tag] [--trend] [--after] [--before] [--min-support N] [--min-confidence F] [--min-length N] [--max-length N] [--limit] [--offset] [--json] [--robot]`, `annotate --uuid <u> --kind <k> --label <l> [--path <p> --ordinal <n>]`, `recover --from <path> [--dry-run]`, `status [--json]`, `validate [--json]`, `rebuild`, `config [--json]`.

The `SearchEngine` interface is the port; `internal/storage` is the adapter. Database opened lazily. `OpenReadOnly()` provides read-only access for external consumers.

Expand Down Expand Up @@ -255,16 +253,14 @@ github.com/pablontiv/backscroll/internal/sources — External source parse
github.com/pablontiv/backscroll/internal/templates — F2 Drain-inspired miner: Miner, ProcessLine, ExtractErrorLines
github.com/pablontiv/backscroll/internal/corrections — F3 correction-signal detectors: lexicon (es+en), interrupt, denial, rephrase-similarity; registry pattern for deterministic, pluggable detectors
github.com/pablontiv/backscroll/internal/categories — F4 category map loader (rule engine, versioning, tool→category mapping) with embedded default preset
github.com/pablontiv/backscroll/internal/chunking — Token-aware text chunking helpers for embedding preparation
github.com/pablontiv/backscroll/internal/embedding — Embedding provider interface, mock provider, and ONNX provider implementation
github.com/pablontiv/backscroll/internal/hybrid — Reciprocal Rank Fusion helpers for merged lexical/vector retrieval
github.com/pablontiv/backscroll/internal/hybrid — Reciprocal Rank Fusion helpers for merged lexical retrieval
github.com/pablontiv/backscroll/internal/sequences — F4 PrefixSpan mining (deterministic pattern discovery per session)
github.com/pablontiv/backscroll/internal/services — Concrete typed read-only query orchestration over storage
github.com/pablontiv/backscroll/internal/storage — Database schema, migrations v1–v17, FTS5 indexes
github.com/pablontiv/backscroll/internal/projects — Project identity registry
github.com/pablontiv/backscroll/internal/readers — SessionReader interface, Registry, ClaudeReader (text+tool_use+tool_result), PiReader (text+toolCall+custom results), OMPReader (native v3 text+thinking+toolCall+toolResult; parser-backed identities and origin), OpenCodeReader (text+tool state.input+state.output), CodexReader (rollout response items; docs/input-contract.md), MarkdownDocumentReader (`markdown_document`), MarkdownSectionsReader (`markdown_sections`); toolfmt serializer
github.com/pablontiv/backscroll/internal/recovery — Stranded database recovery orchestration, durable backup, atomic replacement, and post-install sync
Implemented: `internal/autoupdate`, `internal/config`, `internal/input_config`, `internal/models`, `internal/readers`, `internal/sync`, `internal/tagging`, `internal/plans`, `internal/sources`, `internal/storage`, `internal/services`, `internal/projects`, `internal/templates`, `internal/corrections`, `internal/categories`, `internal/sequences`, `internal/chunking`, `internal/embedding`, `internal/hybrid`, `internal/compat`, `internal/recovery`, `internal/startuplock`. CLI commands in `cmd/backscroll/` (10 v2 commands via cobra).
Implemented: `internal/autoupdate`, `internal/config`, `internal/input_config`, `internal/models`, `internal/readers`, `internal/sync`, `internal/tagging`, `internal/plans`, `internal/sources`, `internal/storage`, `internal/services`, `internal/projects`, `internal/templates`, `internal/corrections`, `internal/categories`, `internal/sequences`, `internal/hybrid`, `internal/compat`, `internal/recovery`, `internal/startuplock`. CLI commands in `cmd/backscroll/` (10 v2 commands via cobra).
github.com/pablontiv/backscroll/internal/autoupdate — Private context-aware release staging and safe replacement
github.com/pablontiv/backscroll/internal/startuplock — Canonical database lock coordination via persistent OS advisory sidecar
github.com/pablontiv/backscroll/scripts/recall-eval — Isolated recall evaluation runner and cohort reporter
Expand Down
9 changes: 4 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,11 +176,10 @@ A context UUID is opaque and is never synthesized for a result that lacks one. T

Filters worth knowing: `--after` / `--before` for a date window, `--tag` for auto-detected session categories (debugging, refactoring, testing…), `--source-path` to pin one stored input path, `--source` to keep one source class, and `--role` to filter the stored role field. The role field is not a semantic-origin guarantee.

`--lexical-only` explicitly selects BM25-only retrieval. The CLI currently has no
wired embedding provider, so ordinary search also falls back to lexical retrieval.
`--similarity-threshold F` accepts a finite value in `0..1` (default `0.3`,
`0` disables a vector threshold), but applies only to vector results; it does not
filter BM25 results or `--relax`. These flags do not enable semantic expansion.
Search is lexical: prose uses Porter-stemmed BM25, tool text uses trigram BM25,
and unfiltered searches fuse those two lexical rankings with RRF. `--relax`
optionally drops low-IDF terms after a strict zero-result search without changing
scope or introducing semantic expansion.

### Discover what recurs

Expand Down
11 changes: 5 additions & 6 deletions cmd/backscroll/backscroll_skill.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Input manifests are loaded only from:
where `<config_dir>` is non-empty `BACKSCROLL_CONFIG_DIR`, otherwise `$HOME/.config`
on every OS (Windows: `<user-home>\.config`). `XDG_CONFIG_HOME`, macOS Application
Support, and `%APPDATA%` do not select this directory. The app config file is for
database and embedding settings, not ingestion sources.
database settings, not ingestion sources.

Inspect the effective database, legacy session directories, and active inputs:

Expand Down Expand Up @@ -125,11 +125,10 @@ If output is truncated, treat it as evidence that more indexed data exists. Refi

For explicit lexical term-overload recall, add `--relax` (for example `backscroll search --text '+violet handshake technique adaptation' --relax --robot --fields minimal --max-tokens 2000`); `+term`/quoted phrases stay protected, scope never widens, and relaxed results report `match_stage`/`dropped_terms`—this does not recover missing vocabulary.

`--lexical-only` explicitly requests BM25-only retrieval. The CLI currently has
no wired embedding provider and falls back to lexical retrieval even without
that flag. `--similarity-threshold F` accepts finite `0..1` (default `0.3`;
`0` disables the vector threshold) but applies only to vector results, not BM25
or `--relax`. It does not enable semantic expansion.
Search is always lexical: prose uses Porter-stemmed BM25, tool text uses trigram
BM25, and unfiltered searches fuse those lexical rankings with RRF. `--relax`
only drops low-IDF terms after a strict zero-result search; it never performs
semantic expansion.

## 5) Query patterns by use case

Expand Down
4 changes: 2 additions & 2 deletions cmd/backscroll/codex_e2e_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ format = "codex"
{"codexafterquartz", "assistant", "text"},
} {
t.Run(tc.marker, func(t *testing.T) {
out, stderr, err := runCmd("search", tc.marker, "--project", "codex-fixture-project", "--source", "session", "--source-path", source, "--role", tc.role, "--content-type", tc.kind, "--after", "2026-09-01", "--before", "2026-09-02", "--lexical-only", "--json")
out, stderr, err := runCmd("search", tc.marker, "--project", "codex-fixture-project", "--source", "session", "--source-path", source, "--role", tc.role, "--content-type", tc.kind, "--after", "2026-09-01", "--before", "2026-09-02", "--json")
if err != nil {
t.Fatalf("search: %v stderr=%s", err, stderr)
}
Expand All @@ -69,7 +69,7 @@ format = "codex"
"codexignoredinstructions", "codexignoredimage", "codexignoredencrypted",
"codexignoredcompact", "codexignoredfuture", "codexignoredmalformed", "codexreasonquartz",
} {
out, stderr, err := runCmd("search", marker, "--all-projects", "--lexical-only", "--json")
out, stderr, err := runCmd("search", marker, "--all-projects", "--json")
if err != nil {
t.Fatalf("negative search: %v %s", err, stderr)
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/backscroll/codex_lifecycle_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ func TestCodexPresetIncrementalAndPerennialRecall(t *testing.T) {
}
recall := func(marker string) {
t.Helper()
out, stderr, err := runCmd("search", marker, "--all-projects", "--lexical-only", "--json")
out, stderr, err := runCmd("search", marker, "--all-projects", "--json")
if err != nil {
t.Fatalf("recall %s: %v %s", marker, err, stderr)
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/backscroll/codex_wrapper_e2e_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ func TestCodexInjectedWrappersV1E2E(t *testing.T) {
{"codexblockkeepquartz", "user", 1},
} {
t.Run(tc.marker, func(t *testing.T) {
out, stderr, err := runCmd("search", tc.marker, "--all-projects", "--source-path", sourcePath, "--role", tc.role, "--content-type", "text", "--lexical-only", "--json")
out, stderr, err := runCmd("search", tc.marker, "--all-projects", "--source-path", sourcePath, "--role", tc.role, "--content-type", "text", "--json")
if err != nil {
t.Fatalf("search: %v stderr=%s", err, stderr)
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/backscroll/echo_shell_zero_query_gap_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ func TestZeroValuedCodexShellEchoExcludedBeforeReplay(t *testing.T) {
}
}

out := e.run("search", "--text", "violet handshake adaptation", "--all-projects", "--lexical-only", "--relax", "--robot", "--fields", "minimal", "--max-tokens", "200")
out := e.run("search", "--text", "violet handshake adaptation", "--all-projects", "--relax", "--robot", "--fields", "minimal", "--max-tokens", "200")
if !strings.Contains(out, "result_0_filepath="+filepath.Join(e.fixtures, "target.jsonl")+"\n") || !strings.Contains(out, `result_0_dropped_terms=["adaptation"]`) {
t.Errorf("zero-valued shell query echoes inverted unfiltered --relax IDF\nstdout=%s", out)
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/backscroll/echo_zero_replay_e2e_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ func TestZeroValuedCodexExecCommandEchoExcludedBeforeReplay(t *testing.T) {
t.Errorf("zero-valued Codex fallback leaks=%d want %d: %v", codexLeaks, wantCodexLeaks, queryEchoShape(got))
}

out := e.run("search", "--text", "violet handshake adaptation", "--all-projects", "--lexical-only", "--relax", "--robot", "--fields", "minimal", "--max-tokens", "200")
out := e.run("search", "--text", "violet handshake adaptation", "--all-projects", "--relax", "--robot", "--fields", "minimal", "--max-tokens", "200")
if !strings.Contains(out, "result_0_filepath="+filepath.Join(e.fixtures, "target.jsonl")+"\n") || !strings.Contains(out, `result_0_dropped_terms=["adaptation"]`) {
t.Errorf("zero-valued query echoes changed unfiltered --relax IDF\nstdout=%s", out)
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/backscroll/legacy_sources_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ func TestLegacySourcesAllowsHelpAndVersion(t *testing.T) {
}{
{name: "root help", args: []string{"--help"}, want: "Available Commands:"},
{name: "root version", args: []string{"--version"}, want: "backscroll"},
{name: "subcommand help", args: []string{"search", "--help"}, want: "Search performs a hybrid search"},
{name: "subcommand help", args: []string{"search", "--help"}, want: "Search performs lexical full-text search"},
}

for _, tt := range tests {
Expand Down
40 changes: 11 additions & 29 deletions cmd/backscroll/main_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,40 +471,22 @@ func TestSearchHelp(t *testing.T) {
if err != nil {
t.Fatalf("search --help error: %v", err)
}
for _, flag := range []string{"--project", "--source", "--after", "--before", "--role", "--limit", "--json", "--robot", "--lexical-only", "--similarity-threshold"} {
for _, flag := range []string{"--project", "--source", "--after", "--before", "--role", "--limit", "--json", "--robot", "--relax"} {
if !strings.Contains(out, flag) {
t.Errorf("search --help missing flag %q", flag)
}
}
}

func TestSearchLexicalOnly(t *testing.T) {
_, cleanup := testEnv(t)
defer cleanup()

fixture := filepath.Join(fixturesDir(), "claude.jsonl")
_, _, _ = syncForTest(t, "sync", "--path", filepath.Dir(fixture))

out, _, err := runCmd("search", "--lexical-only", "session")
if err != nil {
t.Fatalf("search --lexical-only error: %v", err)
}
_ = out
}

func TestSearchSimilarityThreshold(t *testing.T) {
_, cleanup := testEnv(t)
defer cleanup()

fixture := filepath.Join(fixturesDir(), "claude.jsonl")
_, _, _ = syncForTest(t, "sync", "--path", filepath.Dir(fixture))

// With no vectors in DB, falls back to BM25 regardless of threshold
out, _, err := runCmd("search", "--similarity-threshold", "0.5", "session")
if err != nil {
t.Fatalf("search --similarity-threshold error: %v", err)
func TestSearchRejectsRemovedSemanticFlags(t *testing.T) {
for _, flag := range []string{"--lexical-only", "--similarity-threshold"} {
t.Run(flag, func(t *testing.T) {
_, _, err := runCmd("search", "session", flag)
if err == nil || !strings.Contains(err.Error(), "unknown flag: "+flag) {
t.Fatalf("search %s error = %v, want unknown flag", flag, err)
}
})
}
_ = out
}

func TestListTextSynced(t *testing.T) {
Expand Down Expand Up @@ -1359,7 +1341,7 @@ func TestSearchFindsPionRecordTypeAfterEmptyIndex(t *testing.T) {
t.Fatalf("close seeded index: %v", err)
}

out, _, err := runCmd("search", "pion_recordtype_search_token", "--all-projects", "--lexical-only")
out, _, err := runCmd("search", "pion_recordtype_search_token", "--all-projects")
if err != nil {
t.Fatalf("search Pion recordType: %v", err)
}
Expand Down Expand Up @@ -1440,7 +1422,7 @@ func TestPiPionIngestionRegressionMatrixE2E(t *testing.T) {

searchCount := func(query string) int {
t.Helper()
out, stderr, err := runCmd("search", "--text", query, "--all-projects", "--lexical-only", "--json")
out, stderr, err := runCmd("search", "--text", query, "--all-projects", "--json")
if err != nil {
t.Fatalf("search %q: %v\nstdout=%s\nstderr=%s", query, err, out, stderr)
}
Expand Down
Loading
Loading