fix(engine): drop per-bank vector indexes before bank delete data (#3485) - #3493
Open
xiaoxiaoHe-E wants to merge 1 commit into
Open
fix(engine): drop per-bank vector indexes before bank delete data (#3485)#3493xiaoxiaoHe-E wants to merge 1 commit into
xiaoxiaoHe-E wants to merge 1 commit into
Conversation
Bank deletion previously dropped a bank's per-bank vector indexes only after its delete transaction committed. With per-bank backends, every bank adds 3 partial indexes to the shared memory_units table, and Postgres plans any statement against it by locking ALL of its indexes; past a few thousand banks the lock-manager pool is exhausted and every statement against memory_units fails — including the delete DML, so the API could no longer delete banks to recover. Drop the indexes first (still CONCURRENTLY, on an autocommit connection, still retried with backoff): DROP INDEX only locks its own index plus the table, so it keeps working when the pool is exhausted, restoring an API-driven recovery path. Fact-type-scoped deletes and delete_bank_profile=False keep the indexes, as before. Refs vectorize-io#3485
koriyoshi2041
approved these changes
Aug 14, 2026
koriyoshi2041
left a comment
Contributor
There was a problem hiding this comment.
Tested the exact head 9680107 locally on macOS/Python 3.11 with embedded PostgreSQL: both HNSW delete-order tests passed, including the new before-data regression, and both bank lifecycle deadlock retry tests passed. The full-delete-only guard also preserves the scoped-delete behavior. The recovery ordering looks sound to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Short-term fix for #3485 (per-bank partial vector indexes exhaust the Postgres lock table at ~2,100 banks — recall AND bank deletion both fail cluster-wide).
delete_bankpreviously dropped a bank's 3 per-bank vector indexes only after its delete transaction committed. With per-bank backends (pgvector/pgvectorscale/vchord), every bank adds 3 partial indexes to the sharedmemory_unitstable, and Postgres plans any statement against that table by taking a lock on all of its indexes. Past a few thousand banks the lock-manager shared-memory pool is exhausted, so every statement againstmemory_unitsfails — including the delete DML. Bank deletion was therefore part of the failure, not the recovery: the API could not delete banks to escape the wall.This PR changes the order: the bank's per-bank vector indexes are dropped before the delete transaction starts.
Why this works
DROP INDEX (CONCURRENTLY)is a utility statement that only locks its own index plus the table — it still succeeds when the lock pool is exhausted, while the delete DML cannot plan.CONCURRENTLYon an autocommit connection (never inside a transaction), still wrapped inretry_with_backofffor the transient deadlock with concurrent index builds.fact_type-scoped deletes anddelete_bank_profile=Falsekeep the indexes (only the full-delete path drops them).Changes
hindsight_api/engine/memory_engine.py: fetchinternal_id, drop the 3 per-bank vector indexes on an autocommit connection, then run the delete transaction; removed the post-commit drop.hindsight_api/engine/db/ops_postgresql.py/hindsight_api/engine/retain/bank_utils.py: comments/docstrings updated to the new order.tests/test_hnsw_indexes.py: new regression testtest_delete_bank_drops_indexes_before_dataasserting the indexes are dropped while the bank's rows and profile still exist.tests/test_bank_lifecycle_deadlock_retry.py: docstring updated.Notes
./scripts/hooks/lint.shclean.Refs #3485