Skip to content

feat(processor): implement cascading document deletion via adelete_by_doc_id (#127) - #471

Merged
LarFii merged 6 commits into
HKUDS:mainfrom
StevenSiahaann:feat-cascade-deletion
Nov 12, 2026
Merged

LarFii merged 6 commits into
HKUDS:mainfrom
StevenSiahaann:feat-cascade-deletion

Conversation

@StevenSiahaann

@StevenSiahaann StevenSiahaann commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Add RAGAnything.adelete_by_doc_id() to delegate core graph/vector/document deletion to LightRAG and then clean project completion state, placeholders and optional parsed artifacts.

Core deletion must explicitly confirm success (or confirm absence of a cache-only orphan). Exceptions, busy/failure results, None and responses without a status retain project data. Remove basename-based directory guessing: use document-ID-linked records and exact paths, and retain outputs referenced by another indexed document. Filesystem removal errors leave media_files_deleted=False, including partial Office-output cleanup. Legacy basename-only records can still delete core data while leaving unlocated media in place.

Retain support for per-call output directories, persisted parser metadata/cache records and Office conversion artifacts. English and Chinese READMEs describe the API and cleanup-result semantics.

Validation:

  • Reproduced the unconfirmed-result and same-basename wrong-document defects before the maintainer fix: 5 failures.
  • 25 deletion regression cases cover failures/retries, unknown results, source/output collisions, shared outputs, restart metadata and read-only output directories.
  • Isolated deletion changes on main@121ef41: 1,292 tests pass; Ruff 0.6.4 lint and format pass.
  • Final tree after integrating fix(processor): cut multimodal ingest LLM call amplification (#348) #453: 1,308 tests pass; lint and format pass.
  • GitHub Python 3.10/3.11/3.12 tests and lint pass on the updated head.
  • Public deletion also passed with real Qwen3-VL and Qwen embeddings in all four extraction modes, using actual LightRAG stores; all registered chunks were removed and original source files remained intact.

Fixes #127
Addresses #332

@LarFii LarFii left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed public deletion API is useful, but the cleanup order currently causes data loss when core deletion fails.

Reproduction with a temporary parsed-output directory and the PR's processor harness: have LightRAG.adelete_by_doc_id raise before deleting the graph, then call adelete_by_doc_id(doc_id, delete_media_files=True). The result has status='failed'; the document/graph still exists, but the media directory and multimodal completion cache have already been deleted. The outer message also says 'Successfully processed deletion'. All current PR tests pass because they exercise successful core deletion.

Please delegate core deletion first and retain media/cache/placeholders unless its result confirms success; cover exceptions and unsuccessful/busy results. The fallback that deletes only doc_status/full_docs must not report a successful graph/vector deletion. Finally, artifact cleanup needs the actual parsed-output location: reconstructing it from the default basename-only file_path or config.parser_output_dir misses absolute source paths and per-call output_dir overrides. Please add regression coverage and document the new API in both READMEs. Leaving this PR open for revision.

@StevenSiahaann

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, @LarFii! Every point has been addressed in 932ffed:

  1. Core Deletion First & Failure/Busy Retention:
    • adelete_by_doc_id now snapshots the pre-deletion metadata and delegates core graph/vector/KV deletion to self.lightrag.adelete_by_doc_id(...) first.
    • If LightRAG.adelete_by_doc_id raises an exception or returns a non-success status ("fail", "failed", "not_allowed", "busy", etc.), multimodal_status_cache, parsed media directories on disk, and doc-pre-<file_name> placeholders are retained intact so retries can cleanly recover, and the outer status and message accurately reflect the failure rather than reporting "Successfully processed deletion".
  2. Removed False-Positive Partial Deletion Fallback:
    • When the underlying LightRAG instance does not provide adelete_by_doc_id, adelete_by_doc_id now fails cleanly (status="failed") instead of deleting only doc_status/full_docs and falsely reporting graph/vector deletion success.
  3. Resolved Actual Parsed-Output Locations:
    • Added _compute_parsed_output_dirs, _record_parsed_output_location, and _resolve_parsed_output_dirs_for_deletion so adelete_by_doc_id(..., delete_media_files=True) resolves the actual parsed output directory (and Office intermediate PDF conversion directory) from:
      1. In-memory locations recorded during parse_document, process_document_complete, process_document_complete_lightrag_api, and insert_content_list (handling both nested/absolute source paths under default use_full_path=False and per-call output_dir overrides).
      2. Persisted doc_status["metadata"] (raganything_output_base, raganything_parsed_output_dirs, raganything_source_file_path) and parse_cache entries across restarts, with stale parse_cache entries invalidated when media directories are deleted.
      3. Fallback to doc_status["file_path"] + config.parser_output_dir for legacy records.
  4. Tests & Documentation:
    • Expanded tests/test_document_deletion_cascade.py (17 tests) covering LightRAG.adelete_by_doc_id raising exceptions + retry recovery, unsuccessful/busy statuses (fail, failed, not_allowed, busy), missing adelete_by_doc_id without false-positive fallback, nested source paths with use_full_path=False and per-call output_dir overrides, same-basename files in different directories, persisted parse_cache resolution/invalidation across restarts, and Office document dual output directories.
    • Documented adelete_by_doc_id in both README.md (#### 9. Deleting Documents) and README_zh.md (#### 9. 删除文档).

…261112

# Conflicts:
#	README.md
#	README_zh.md

@LarFii LarFii left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved the earlier deletion-order findings and the remaining ambiguous-result / unrelated-output defects. Core exceptions, busy/failure/unknown responses retain project data. Basename guessing is removed, outputs referenced by other documents are retained, and actual filesystem failures cannot be reported as successful media removal. Full integrated suite: 1,308 passed with Ruff 0.6.4 lint/format; Python 3.10/3.11/3.12 CI passed. Four live Qwen3-VL/Qwen embedding insertion-query-public-deletion flows passed across all extraction modes, using real LightRAG stores. The rebased head has exactly the same file tree as the tested integration. Previous blocking review findings are resolved.

@LarFii
LarFii merged commit eb7a670 into HKUDS:main Nov 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Question]: Can we leverage LightRAG features to delete content from the knowledge base?

2 participants