Skip to content

Commit 71df02e

Browse files
feat(p12.2-4): BM25 + IntentRanker (Legacy / Bm25Hybrid / LlmRerank)
Per audit semantic-rank-layer 2026-05-05 : L4 verdict WRONG : "no-bm25-only-baseline-measured (impact 5)" — the audit blocked the original dense-first plan and demanded BM25 as the measurable baseline first. This commit ships that baseline + the layered IntentRanker abstraction the audit recommended. L4 finding "llm-rerank-top-k-not-considered (impact 4)" — for SatRank's current 192-row catalogue and <10 qps regime, BM25 narrow → Claude Haiku rerank top-K is the audit-validated default. This commit ships the LlmRerankRanker as one of the IntentRanker variants, gated by RANKER_MODE env var (next commit). L2 architecture : "EmbeddingProvider/VectorIndex/IntentRanker is correctly placed seam" — IntentRanker interface is the seam; future rankers (DenseHybridRanker, CrossEncoderRanker, GbdtLtrRanker) plug in without rewriting intentService. What ships : src/services/bm25Service.ts Standard Okapi BM25 (k1=1.5, b=0.75). In-memory inverted index + English stopwords + lowercase non-alphanumeric tokenizer. Exposes : buildIndex(docs) — rebuild from catalogue score(docId, query) — single-doc score topK(query, k) — top-K matching docs explain(docId, query) — per-term contribution (audit L5 + L6) stats() — diagnostics Sub-ms p_avg at 192 docs ; ~30MB RAM at 100k docs. src/services/intentRanker.ts IntentRanker interface : rank(intentText, candidates) → RankedCandidate[] Three implementations : LegacyRanker — pre-P12 ordering (p_e2e DESC) Bm25HybridRanker — 0.5 × BM25 + 0.5 × p_e2e LlmRerankRanker — BM25 narrow top-20 → LLM rerank top-K + buildAnthropicRerankAdapter — wires LlmRerankRanker.llmCall to @anthropic-ai/sdk Messages.create with claude-haiku-4-5 default. Anthropic SDK 0.93+ added as a runtime dep (was already in scripts/sim but not in main package.json). Dependency is OPTIONAL at runtime — the IntentRanker stack defaults to LegacyRanker when RANKER_MODE is unset. Tests : - 23 BM25 tests covering tokenize / buildIndex / topK / score / explain / stats / 192-doc perf smoke (<50ms build, <10ms p_avg topK) - 18 IntentRanker tests covering LegacyRanker tiebreaks / Bm25Hybrid fallback + Sim-13-hyperdope-equivalent regression / LlmRerank LLM- error fallback / buildAnthropicRerankAdapter JSON parsing 1876 server tests total ; tsc strict + tsc-tests strict clean. Wiring into intentService + RANKER_MODE flag come in the next commit. Production behaviour is UNCHANGED until that commit lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 2b2dc1c commit 71df02e

6 files changed

Lines changed: 1070 additions & 0 deletions

File tree

‎package-lock.json‎

Lines changed: 49 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@
4646
],
4747
"license": "AGPL-3.0",
4848
"dependencies": {
49+
"@anthropic-ai/sdk": "^0.93.0",
4950
"@modelcontextprotocol/sdk": "^1.29.0",
5051
"@noble/curves": "^2.0.1",
5152
"@noble/hashes": "^1.8.0",

‎src/services/bm25Service.ts‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
// Phase 12.2 (2026-05-05) — BM25 in-process scorer.
2+
//
3+
// Per audit semantic-rank-layer 2026-05-05 (lens L4, finding "no-bm25-only-
4+
// baseline-measured", impact 5). Audit verdict was WRONG on the original
5+
// plan because dense embeddings were specced without measuring BM25 first.
6+
// On short structured intents over a 192-row catalogue with category +
7+
// modality filters, BM25 routinely hits 95%+ retrieval quality (BEIR :
8+
// BM25 beats dense on FiQA, SciFact, ArguAna). This service is the
9+
// baseline that gates whether dense embeddings are worth shipping.
10+
//
11+
// Implementation : Okapi BM25 with k1=1.5, b=0.75 (standard defaults).
12+
// Inverted index in-memory ; tokenizer = lowercase + non-alphanumeric
13+
// split + English stopwords. ~30MB RAM at 100k endpoints, sub-ms query
14+
// latency at 192. No external dependency, fully deterministic.
15+
16+
const DEFAULT_K1 = 1.5;
17+
const DEFAULT_B = 0.75;
18+
19+
// Conservative English stopword list. Drop these so common verbs don't
20+
// dominate the IDF distribution. Borrowed from Lucene's
21+
// EnglishAnalyzer.ENGLISH_STOP_WORDS_SET (subset, the high-frequency core).
22+
const STOPWORDS = new Set([
23+
'a', 'an', 'and', 'are', 'as', 'at', 'be', 'but', 'by', 'for', 'if',
24+
'in', 'into', 'is', 'it', 'no', 'not', 'of', 'on', 'or', 'such', 'that',
25+
'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was',
26+
'will', 'with', 'i', 'me', 'my', 'we', 'our', 'you', 'your', 'he', 'him',
27+
'his', 'she', 'her', 'them', 'do', 'does', 'did', 'has', 'have', 'had',
28+
'can', 'could', 'should', 'would', 'may', 'might', 'must', 'need',
29+
]);
30+
31+
export interface Bm25Document {
32+
id: number;
33+
text: string;
34+
}
35+
36+
export interface Bm25Score {
37+
id: number;
38+
score: number;
39+
}
40+
41+
interface PostingList {
42+
/** doc_id → term frequency in that doc. */
43+
postings: Map<number, number>;
44+
/** number of docs containing the term. */
45+
df: number;
46+
}
47+
48+
/** Pure tokenizer : lowercase, split on non-alphanumeric, drop stopwords +
49+
* too-short tokens. Exported for tests. */
50+
export function tokenize(text: string): string[] {
51+
return text
52+
.toLowerCase()
53+
.split(/[^a-z0-9]+/)
54+
.filter(t => t.length >= 2 && !STOPWORDS.has(t));
55+
}
56+
57+
export class Bm25Service {
58+
private invertedIndex: Map<string, PostingList> = new Map();
59+
private docLengths: Map<number, number> = new Map();
60+
private avgDocLength = 0;
61+
private docCount = 0;
62+
private k1: number;
63+
private b: number;
64+
65+
constructor(opts: { k1?: number; b?: number } = {}) {
66+
this.k1 = opts.k1 ?? DEFAULT_K1;
67+
this.b = opts.b ?? DEFAULT_B;
68+
}
69+
70+
/** Build the inverted index from scratch. Call when the catalogue
71+
* changes ; for SatRank's 192-row catalog this takes ~1ms. */
72+
buildIndex(docs: ReadonlyArray<Bm25Document>): void {
73+
this.invertedIndex = new Map();
74+
this.docLengths = new Map();
75+
this.docCount = docs.length;
76+
let totalLen = 0;
77+
for (const doc of docs) {
78+
const tokens = tokenize(doc.text);
79+
this.docLengths.set(doc.id, tokens.length);
80+
totalLen += tokens.length;
81+
const termFreqs = new Map<string, number>();
82+
for (const t of tokens) {
83+
termFreqs.set(t, (termFreqs.get(t) ?? 0) + 1);
84+
}
85+
for (const [term, tf] of termFreqs) {
86+
let pl = this.invertedIndex.get(term);
87+
if (!pl) {
88+
pl = { postings: new Map(), df: 0 };
89+
this.invertedIndex.set(term, pl);
90+
}
91+
pl.postings.set(doc.id, tf);
92+
pl.df += 1;
93+
}
94+
}
95+
this.avgDocLength = this.docCount > 0 ? totalLen / this.docCount : 0;
96+
}
97+
98+
/** Score a single doc against a query. Returns 0 if the doc is unknown
99+
* (caller passed an id that wasn't in buildIndex). */
100+
score(docId: number, query: string): number {
101+
if (this.docCount === 0) return 0;
102+
const docLen = this.docLengths.get(docId);
103+
if (docLen === undefined) return 0;
104+
const queryTerms = new Set(tokenize(query));
105+
if (queryTerms.size === 0) return 0;
106+
let score = 0;
107+
for (const term of queryTerms) {
108+
const pl = this.invertedIndex.get(term);
109+
if (!pl) continue;
110+
const tf = pl.postings.get(docId);
111+
if (tf === undefined || tf === 0) continue;
112+
// Okapi BM25 IDF : log((N - df + 0.5) / (df + 0.5) + 1).
113+
// The +1 inside the log is the BM25Plus / Lucene variant : keeps
114+
// IDF non-negative even when df > N/2.
115+
const idf = Math.log(((this.docCount - pl.df + 0.5) / (pl.df + 0.5)) + 1);
116+
const norm = 1 - this.b + this.b * (docLen / (this.avgDocLength || 1));
117+
const tfTerm = (tf * (this.k1 + 1)) / (tf + this.k1 * norm);
118+
score += idf * tfTerm;
119+
}
120+
return score;
121+
}
122+
123+
/** Top-K matching docs for a query. O(|query terms| * avg posting len)
124+
* — for SatRank scale that's microseconds. */
125+
topK(query: string, k: number): Bm25Score[] {
126+
const queryTerms = new Set(tokenize(query));
127+
if (queryTerms.size === 0 || this.docCount === 0) return [];
128+
// Aggregate scores across terms in a single pass over the matching
129+
// posting lists, instead of scoring every doc individually.
130+
const scores = new Map<number, number>();
131+
for (const term of queryTerms) {
132+
const pl = this.invertedIndex.get(term);
133+
if (!pl) continue;
134+
const idf = Math.log(((this.docCount - pl.df + 0.5) / (pl.df + 0.5)) + 1);
135+
for (const [docId, tf] of pl.postings) {
136+
const docLen = this.docLengths.get(docId) ?? 0;
137+
const norm = 1 - this.b + this.b * (docLen / (this.avgDocLength || 1));
138+
const tfTerm = (tf * (this.k1 + 1)) / (tf + this.k1 * norm);
139+
scores.set(docId, (scores.get(docId) ?? 0) + idf * tfTerm);
140+
}
141+
}
142+
return Array.from(scores.entries())
143+
.map(([id, score]) => ({ id, score }))
144+
.sort((a, b) => b.score - a.score)
145+
.slice(0, k);
146+
}
147+
148+
/** Diagnostics for the explain mode (audit L5 + L6). Returns per-term
149+
* contribution to the score so agents can audit the ranking decision
150+
* without an LLM-in-loop. */
151+
explain(docId: number, query: string): Array<{ term: string; tf: number; idf: number; contribution: number }> {
152+
const result: Array<{ term: string; tf: number; idf: number; contribution: number }> = [];
153+
if (this.docCount === 0) return result;
154+
const docLen = this.docLengths.get(docId);
155+
if (docLen === undefined) return result;
156+
const queryTerms = new Set(tokenize(query));
157+
for (const term of queryTerms) {
158+
const pl = this.invertedIndex.get(term);
159+
if (!pl) continue;
160+
const tf = pl.postings.get(docId);
161+
if (tf === undefined || tf === 0) continue;
162+
const idf = Math.log(((this.docCount - pl.df + 0.5) / (pl.df + 0.5)) + 1);
163+
const norm = 1 - this.b + this.b * (docLen / (this.avgDocLength || 1));
164+
const tfTerm = (tf * (this.k1 + 1)) / (tf + this.k1 * norm);
165+
result.push({ term, tf, idf, contribution: idf * tfTerm });
166+
}
167+
return result.sort((a, b) => b.contribution - a.contribution);
168+
}
169+
170+
stats(): { docCount: number; avgDocLength: number; vocabSize: number } {
171+
return {
172+
docCount: this.docCount,
173+
avgDocLength: this.avgDocLength,
174+
vocabSize: this.invertedIndex.size,
175+
};
176+
}
177+
}

0 commit comments

Comments
 (0)