Skip to content

Fix document replacement, watcher synchronization, and search weighting - #1

Merged
lychee888 merged 1 commit into
mainfrom
fix/document-lifecycle
Sep 30, 2026
Merged

lychee888 merged 1 commit into
mainfrom
fix/document-lifecycle

Conversation

@lychee888

Copy link
Copy Markdown
Owner

Re-importing, moving, or emptying a document could leave obsolete chunks and orphaned visual captions in search results. This change replaces each source atomically, gives all chunks the same resolved source identity and page numbering, and reconciles file changes when the watcher starts.

The fixes also preserve documents during metadata migration and failed extraction/embedding, use an OS lock for watcher ownership, refresh database readers across processes, apply search_alpha to weighted rank fusion, and load complete cached embedding snapshots directly. Windows output uses UTF-8, including help output; an explicit new config path no longer falls back to the user's existing config. Source-filtered queries retain the configured search weighting. LanceDB's minimum version is raised to 0.25 for native full-text indexing.

Validation on Windows / Python 3.11:

  • 88 tests passed, 2 optional VLM caption tests skipped, 6 autostart/legacy watcher tests deselected. Real cached embeddings and English/Chinese CLI queries were tested offline using LanceDB 0.36.
  • 38 focused regressions passed against the actual minimum LanceDB 0.25.0, covering native filesystem events, concurrent watcher ownership and process exit, replacements, empty documents, failure preservation, schema migration, search weighting, and configuration/output isolation.
  • Wheel build succeeded; both installed command entry points passed smoke checks.
  • Added a Windows/Linux regression workflow without model downloads or OS startup changes.

The original polling watcher is covered by a new isolated regression test. Live OCR/VLM quality and OS autostart installation were not exercised. README documents cleanup of ambiguous relative/basename source entries left by older releases; these cannot all be safely reassigned automatically.

@lychee888
lychee888 merged commit 131b826 into main Sep 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant