Repository navigation
Preserve documents on partial extraction and keep keyword search model-free - #2
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-ingestion preserves the entire previous source snapshot when enabled visual extraction is incomplete, including visual chunks and earlier text. CLI and both watcher paths expose errors and partial status. Successfully empty extraction still removes old rows; explicitly disabling VLM opts into text-only replacement. Keyword-only queries avoid constructing an embedding engine, including source-filtered queries. README clarifies the scope of offline model-file transfer.
Visual-region discovery, PDF rendering, and DOCX image-read failures now propagate instead of being silently accepted. An embedded-image fallback is accepted only when it recovers every required visible raster image and the page has no unrecovered vectors, inline images, masks, annotations, or widgets. Blank PDF pages and short native text without visuals need no OCR/VLM; scanned content and checked form controls still use extraction fallback.
The independent outside review found inline-image, checkbox, and alpha-mask edge cases. All were fixed, along with explicit color-key masks. New real-document/LanceDB regressions cover whole-snapshot preservation through CLI and both watchers, successful opaque raster recovery, partial image recovery, DOCX image reads, blank pages, short text, separator pages, scanned content, inline images, checked widgets, and masked charts. These tests are included in the focused CI suite.
Local final validation on LanceDB 0.25.0: 69 focused tests passed. With the existing PDF image-extraction tests included, 72 passed. The broader model-backed suite is exercised by CI using a downloaded embedding model followed by offline execution. CI runs the focused suite on Windows/Linux and Python 3.11/3.13.
Live VLM/OCR quality and OS autostart installation were not exercised locally. The broader safe suite excludes autostart and legacy watcher tests that modify startup state; the isolated replacement regressions exercise both watcher implementations.
Verified CI on final head 81950fc: all 5 checks passed (run). Leave this PR open for review.