fix(search): recency boost goes to the matched chunk, gated by the all-words grade (#507)

BACKLOG.md

@@ -19,7 +19,7 @@ note's title-only chunk outrank a note whose body matches every query word. Full
 analysis on the epic.
 
 - [x] **Index the vault in a deterministic order and break BM25 ties on a stable key** ([#506](https://github.com/CryptoJones/omind/issues/506)) — **v10.4.1** — _bug_
-- [ ] **Recency must not lift a newer note's title-only match above a note whose body matches every query word** ([#507](https://github.com/CryptoJones/omind/issues/507)) — _bug_ — after #506
+- [x] **Recency must not lift a newer note's title-only match above a note whose body matches every query word** ([#507](https://github.com/CryptoJones/omind/issues/507)) — **v10.4.2** — _bug_ — after #506
 
 ## Not planned
 
@@ -75,6 +75,17 @@ _Closed as not planned or rejected. Kept for the record; nothing here is pending
 
 ## Done
 
+### Shipped — 2026-10-31 (v10.4.2)
+
+- [x] **Recency must not lift a newer note's title-only match above a note whose body matches every query word** ([#507](https://github.com/CryptoJones/omind/issues/507)) — **v10.4.2** — _bug_ —
+  the recency boost is credited to a note's best matched content chunk (first in the keyword list,
+  else the vector list) instead of its `ord = 0` title chunk. Candidates stay the notes whose title
+  chunk a content leg returned; when any of them holds a chunk matching every query word, only those
+  receive the boost. (Epic decision 4's wider gate, and decision 7's no-gate fallback, each cost one
+  bench case on the reference vault; this rule ranks all 30 identically to 10.4.1.) Weights unchanged;
+  recency still never adds a note. `Hit.recency_rank` is filled in and `omind search --explain`
+  prints it. The strict `xfail` from v10.4.1 is removed. Last child of epic #505.
+
 ### Shipped — 2026-10-31 (v10.4.1)
 
 - [x] **Index the vault in a deterministic order and break BM25 ties on a stable key** ([#506](https://github.com/CryptoJones/omind/issues/506)) — **v10.4.1** — _bug_ —

CHANGELOG.md

@@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [10.4.2] - 2026-10-31
+
+### Fixed
+- **A newer note that only shared the query's title could outrank an older
+  note whose body matched every query word
+  ([#507](https://github.com/CryptoJones/omind/issues/507), closes epic
+  [#505](https://github.com/CryptoJones/omind/issues/505)).** The recency leg
+  ranked notes newest-first but credited each note's `ord = 0` title chunk,
+  whatever chunk had actually matched. With two same-titled notes the decoy's
+  tied title match plus the recency boost beat the real answer's full-body match.
+  - The boost now goes to a note's best-ranked content chunk: its first chunk
+    in the keyword list, otherwise in the vector list. A note's keyword credit
+    and recency credit are no longer split across two chunks.
+  - Candidates are, as before, the notes whose *title* chunk a content leg
+    returned: recency breaks ties among notes that are about the query. When any
+    candidate also holds a chunk matching *every* query word, only those
+    candidates receive the boost. That is what drops the title-only decoy.
+  - Epic #505 decision 4 wanted every matched note admitted when the all-words
+    grade was empty. Measured on the reference vault that cost a bench case (a
+    day-old note at keyword rank 88 took the rank-1 boost: recall@5 80.0 → 76.7,
+    MRR 0.6616 → 0.6604), and so did decision 7's fallback (no gate at all).
+    The narrower rule above ranks all 30 cases identically to 10.4.1.
+  - Recency still only reorders: a note no content leg returned has no best
+    chunk and cannot be added. `_W_RECENCY` and `RRF_K` are unchanged.
+  - `omind search --explain` now reports `recency=N` beside the keyword and
+    semantic ranks; `Hit.recency_rank` was declared but never filled in.
+  - The strict `xfail` that 10.4.1 left on the decoy case is gone; the case
+    passes under both indexing orders.
+
 ## [10.4.1] - 2026-10-31
 
 ### Fixed

docs/retrieval.md

@@ -37,7 +37,15 @@ Two rules keep the result honest:
 
 - **Recency re-ranks; it never adds.** A note that matched nothing cannot ride
   the recency leg into your results. (Before the index, search sorted purely by
-  date, so the newest notes came back regardless of relevance.)
+  date, so the newest notes came back regardless of relevance.) It is also
+  reserved for notes whose *title* the query matched — it breaks ties among
+  notes that are about the query, it does not reward a passing mention in a
+  long body. The boost is credited to the note's best *matched* chunk, not its
+  title chunk, and when any of those notes holds a chunk matching every query
+  word only such notes receive it — so a newer note that merely shares a title
+  cannot outrank an older note whose body answers the whole query.
+  `omind search --explain` shows the recency rank beside the keyword and
+  semantic ranks.
 - **Credential notes are de-prioritised** unless your query is itself about
   credentials — the same rule the consult gate applies. Search must never steer
   an agent toward the secrets notes.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "omind"
-version = "10.4.1"
+version = "10.4.2"
 description = "Reproduce the OMI/Obsidian memory integration for AI agents, plus a local web app to view, edit, and add memory entries."
 readme = "README.md"
 requires-python = ">=3.11"

src/omind/__init__.py

@@ -2,4 +2,4 @@
 # Copyright 2026 Aaron K. Clark
 """omind — OMI/Obsidian memory tooling for AI agents."""
 
-__version__ = "10.4.1"
+__version__ = "10.4.2"

src/omind/cli.py

@@ -1414,7 +1414,8 @@ def _run_search_explain(omi_dir: Path, args: argparse.Namespace) -> int:
         where = f"#{hit.heading}" if hit.heading else " (title/name)"
         print(
             f"{rank}. {hit.filename}{where}  score={hit.score:.5f} "
-            f"keyword={hit.keyword_rank or '-'} semantic={hit.vector_rank or '-'}"
+            f"keyword={hit.keyword_rank or '-'} semantic={hit.vector_rank or '-'} "
+            f"recency={hit.recency_rank or '-'}"
         )
         # Usefulness (item #2): always show the reads behind the weight — both
         # the counted (organic) and filtered (hook/re-read) tallies — so a

src/omind/searchindex.py

@@ -1078,7 +1078,7 @@ def search(
                 return self._listing(db, allowed, limit)
             scope = _scope(query, limit)
             expr = _fts_query(query)  # the OR form; also snips the excerpts
-            keyword = self._keyword_leg(db, query, allowed, scope.depth)
+            keyword, all_words = self._keyword_leg(db, query, allowed, scope.depth)
             vector = self._vector_leg(db, query, allowed, scope.depth)
             matched = set(keyword) | set(vector)
             if not matched:
@@ -1087,7 +1087,16 @@ def search(
             # note. Left unrestricted it made every query return the whole vault
             # newest-first — the exact failure this index exists to end.
             recency = [
-                cid for cid in self._recency_leg(db, allowed, scope.depth) if cid in matched
+                cid
+                for cid in self._recency_leg(
+                    db,
+                    allowed,
+                    scope.depth,
+                    keyword=keyword,
+                    vector=vector,
+                    all_words=all_words,
+                )
+                if cid in matched
             ]
             fused = _fuse(
                 [(_W_KEYWORD, keyword), (_W_VECTOR, vector), (_W_RECENCY, recency)]
@@ -1103,6 +1112,7 @@ def search(
             ranks = {
                 "keyword": {cid: i + 1 for i, cid in enumerate(keyword)},
                 "vector": {cid: i + 1 for i, cid in enumerate(vector)},
+                "recency": {cid: i + 1 for i, cid in enumerate(recency)},
             }
             return self._materialize(
                 db, query, expr, fused, ranks, scope.limit, scope.excerpt_chars, useful
@@ -1133,11 +1143,16 @@ def _candidates(
 
     def _keyword_leg(
         self, db: sqlite3.Connection, query: str, allowed: set[str] | None, depth: int
-    ) -> list[int]:
+    ) -> tuple[list[int], set[int]]:
         """BM25 over the FTS index, graded: chunks matching every query word
-        first, then chunks matching any of them."""
+        first, then chunks matching any of them.
+
+        Returns the ranked chunk ids and, separately, the ids of the top grade
+        (every query word matched). The recency leg uses that set as its gate
+        (#507), so it is returned from here rather than computed twice."""
         ranked: list[int] = []
         seen: set[int] = set()
+        all_words: set[int] = set()
         for require_all in (True, False):
             expr = _fts_query(query, require_all=require_all)
             if not expr:
@@ -1146,9 +1161,12 @@ def _keyword_leg(
                 if chunk_id not in seen:
                     seen.add(chunk_id)
                     ranked.append(chunk_id)
+            if require_all:
+                all_words = set(ranked)
             if len(ranked) >= depth:
                 break
-        return ranked[:depth]
+        head = ranked[:depth]
+        return head, all_words & set(head)
 
     @staticmethod
     def _bm25(
@@ -1363,17 +1381,62 @@ def _weighting(self, db: sqlite3.Connection) -> _Weights:
         return weights
 
     def _recency_leg(
-        self, db: sqlite3.Connection, allowed: set[str] | None, depth: int
+        self,
+        db: sqlite3.Connection,
+        allowed: set[str] | None,
+        depth: int,
+        *,
+        keyword: list[int],
+        vector: list[int],
+        all_words: set[int],
     ) -> list[int]:
+        """Newest-first over the notes whose *name* the content legs matched,
+        credited to each note's best content chunk.
+
+        Recency is a note-level signal, but RRF fuses chunks. The boost used to
+        land on a note's ``ord = 0`` title chunk whatever chunk had actually
+        matched, so a newer note whose *title alone* shared the query could
+        outrank an older note whose body matched every word (#507). Now:
+
+        - a note is a candidate when its title chunk was returned by a content
+          leg, exactly as before: recency breaks ties among notes that are
+          *about* the query, it does not reward every passing mention. (Letting
+          every matched note in — epic #505 decision 4 — cost one bench case:
+          a day-old note at keyword rank 88 took the rank-1 boost.)
+        - when any candidate also holds a chunk matching *every* query word
+          (``all_words``), only those candidates receive the boost. That is what
+          drops the title-only decoy.
+        - the boost goes to the note's best-ranked content chunk — its first
+          chunk in the keyword list, otherwise in the vector list — so a note's
+          keyword credit and recency credit are no longer split across two
+          chunks.
+        - it still only reorders. A note no content leg returned has no best
+          chunk and cannot be added here.
+        """
+        matched = set(keyword) | set(vector)
+        owners = self._owners(db)
+        best: dict[str, int] = {}
+        for chunk_id in (*keyword, *vector):
+            name = owners.get(chunk_id)
+            if name is not None:
+                best.setdefault(name, chunk_id)
+        if not best:
+            return []
         rows = db.execute(
             "SELECT c.id AS id, c.filename AS filename FROM chunks c"
             " JOIN notes n ON n.filename = c.filename"
             " WHERE c.ord = 0 ORDER BY n.created DESC, n.filename LIMIT ?",
             (depth * 3,),
         )
-        return [
-            int(r["id"]) for r in rows if allowed is None or str(r["filename"]) in allowed
+        newest = [
+            (int(r["id"]), str(r["filename"]))
+            for r in rows
+            if allowed is None or str(r["filename"]) in allowed
         ][:depth]
+        candidates = [name for title_id, name in newest if title_id in matched and name in best]
+        strong = {owners[cid] for cid in all_words if cid in owners}
+        graded = [name for name in candidates if name in strong]
+        return [best[name] for name in (graded or candidates)]
 
     def _weight_generated(
         self, db: sqlite3.Connection, fused: list[tuple[int, float]]
@@ -1530,6 +1593,7 @@ def _materialize(
                     agent=str(row["agent"] or ""),
                     keyword_rank=ranks["keyword"].get(chunk_id, 0),
                     vector_rank=ranks["vector"].get(chunk_id, 0),
+                    recency_rank=ranks.get("recency", {}).get(chunk_id, 0),
                     usefulness_weight=use.weight if use is not None else 1.0,
                     useful_reads=use.useful_reads if use is not None else 0,
                     filtered_reads=use.filtered_reads if use is not None else 0,

tests/test_cli.py

@@ -255,6 +255,25 @@ def test_search_finds_notes(tmp_path: Path, capsys: pytest.CaptureFixture[str])
     assert "no matches" in capsys.readouterr().out
 
 
+def test_search_explain_reports_the_recency_rank(
+    tmp_path: Path, capsys: pytest.CaptureFixture[str]
+) -> None:
+    """#507: ``--explain`` names every leg that moved a result, recency included.
+    ``Hit.recency_rank`` existed but was never filled in, so the line showed only
+    keyword and semantic ranks."""
+    from omind.store import NoteFields, OmiStore
+
+    OmiStore(tmp_path / "OMI").create_note(
+        NoteFields(title="Codeberg Note", summary="about codeberg releases")
+    )
+    args = ["search", "codeberg", "--explain", "--vault", str(tmp_path), "--folder", "OMI"]
+    assert main(args) == 0
+    out = capsys.readouterr().out
+    assert "Codeberg Note" in out
+    assert "keyword=1" in out
+    assert "recency=1" in out
+
+
 def test_dedup_hint_warns_about_a_similar_existing_note(
     tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
 ) -> None:

tests/test_name_timelines.py

@@ -567,11 +567,11 @@ def test_preflight_names_resolve_through_recall(
         ),
         # #507: the Kp12 pair again, but the NEWER off-topic decoy's filename
         # sorts first. #506 breaks the title chunks' exact BM25 tie on filename,
-        # so the decoy's title chunk takes the tied keyword rank, and the
-        # recency boost (credited to a note's title chunk) carries it to the
-        # top, whichever order the vault was indexed in. Strict: this case must
-        # keep failing until #507 moves the boost onto the matched chunk.
-        pytest.param(
+        # so the decoy's title chunk takes the tied keyword rank; the recency
+        # boost used to land on that title chunk and carry the decoy to the
+        # top. Now the boost goes to A's matched Details chunk and the decoy,
+        # which matches no chunk on every query word, gets none.
+        (
             [
                 (
                     "Kp12 deploy A.md",
@@ -588,9 +588,6 @@ def test_preflight_names_resolve_through_recall(
             ],
             "Kp12 deploy steps ansible playbook frobnicate",
             "Kp12 deploy A.md",
-            marks=pytest.mark.xfail(
-                strict=True, reason="#507: recency lifts a newer title-only decoy"
-            ),
         ),
     ],
 )

tests/test_searchindex.py

@@ -252,6 +252,74 @@ def test_ranking_is_the_same_whatever_order_the_vault_is_indexed_in(
     assert runs[0] == runs[1]
 
 
+def _twin(omi: Path, filename: str, title: str, details: str, created: str) -> None:
+    (omi / filename).write_text(
+        f"# {title}\n\n## Metadata\n- Created: {created}\n- Tags: #ops\n\n"
+        f"## Summary\n{title}\n\n## Details\n{details}\n",
+        encoding="utf-8",
+    )
+
+
+def test_recency_credits_the_matched_chunk_and_skips_a_title_only_decoy(omi: Path) -> None:
+    """#507: two notes share a title. The older one's body matches every query
+    word; the newer one is about tomatoes. The recency boost used to land on
+    each note's title chunk, so the newer decoy's tied title match plus the
+    boost beat the real answer. The boost now goes to a note's best content
+    chunk, and only to notes holding a chunk that matched every query word."""
+    _twin(
+        omi,
+        "Kp12 deploy A.md",
+        "Kp12 deploy steps",
+        "Kp12 deploy steps via ansible playbook frobnicate.",
+        "2026-09-01",
+    )
+    # Filename sorts first, so with #506's tie-break its title chunk takes the
+    # tied keyword rank — the arrangement that made the old boost decisive.
+    _twin(omi, "Kp12 deploy 0.md", "Kp12 deploy steps", "unrelated garden tomatoes.", "2026-09-30")
+    for number in range(4):
+        _note(omi, f"Filler {number}", f"plain filler {number}", ["f"])
+    idx = searchindex.SearchIndex(omi)
+    hits = idx.search("Kp12 deploy steps ansible playbook frobnicate") or []
+    assert [h.filename for h in hits][:2] == ["Kp12 deploy A.md", "Kp12 deploy 0.md"]
+    winner, decoy = hits[0], hits[1]
+    # The boost was credited to the chunk that matched, not the title chunk…
+    assert winner.heading == "Details"
+    assert winner.recency_rank == 1
+    assert winner.keyword_rank == 1
+    # …and the decoy, which matches no chunk on every word, got no boost at all.
+    assert decoy.recency_rank == 0
+
+
+def test_recency_boosts_every_name_match_when_no_chunk_matches_all_words(omi: Path) -> None:
+    """#507: the all-words gate only applies when something passed it. Two
+    notes are each *named* for one word of a two-word query and no chunk holds
+    both; both are boosted, newest first, and ``recency_rank`` reports the order."""
+    _note(omi, "Zebra", "zebra stripes", ["a"], created="2026-01-01")
+    _note(omi, "Giraffe", "giraffe spots", ["a"], created="2026-06-01")
+    _note(omi, "Other", "nothing relevant here", ["b"], created="2026-09-01")
+    idx = searchindex.SearchIndex(omi)
+    by_name = {h.filename: h for h in idx.search("zebra giraffe") or []}
+    assert set(by_name) == {"Zebra.md", "Giraffe.md"}
+    assert by_name["Giraffe.md"].recency_rank == 1
+    assert by_name["Zebra.md"].recency_rank == 2
+
+
+def test_recency_ignores_a_note_the_query_names_nowhere_in_its_title(omi: Path) -> None:
+    """Recency breaks ties among notes that are *about* the query, as it always
+    did: a body-only mention earns no boost, and ``--explain`` shows a dash for
+    it rather than a made-up position. (Admitting every matched note cost a
+    bench case — a day-old note at keyword rank 88 took the rank-1 boost.)"""
+    _note(omi, "Target", "the specific thing", ["t"], created="2026-02-02")
+    _note(omi, "Specific", "the thing itself", ["t"], created="2020-01-01")
+    idx = searchindex.SearchIndex(omi)
+    by_name = {h.filename: h for h in idx.search("specific") or []}
+    assert set(by_name) == {"Target.md", "Specific.md"}
+    assert by_name["Target.md"].recency_rank == 0  # body-only: no boost
+    assert by_name["Specific.md"].recency_rank == 1  # named for the query
+    assert by_name["Specific.md"].keyword_rank == 1
+    assert by_name["Target.md"].vector_rank == 0
+
+
 def test_recency_only_reorders_matches_it_never_adds_notes(omi: Path) -> None:
     for number in range(5):
         _note(omi, f"Filler {number}", "unrelated filler", ["f"], created=f"2026-07-0{number + 1}")