fix: make FTS finalization idempotent

[jackye1995]

Summary

Make distributed FTS finalization idempotent and object-store friendly. The old finalize flow was not retry-safe because it moved build-only part_<old_id>_* partition files before writing the final metadata.lance; if the process stopped in that window, a retry saw no final metadata but the original source files had already been deleted.

The new flow treats metadata.lance as the commit marker and keeps all pre-commit partition sources under stable staging/ paths. Finalization copies staged files to dense final names without deleting the staged sources or canonical final paths first, writes final metadata, then removes staging best-effort. Any leftover staging/ objects are filtered from committed FTS IndexMetadata.files, so cleanup failures do not make transient files part of the committed index.

Implementation

• Write distributed build-only partition data and per-partition metadata under stable staging/part_<old_id>_* paths.
• Keep the non-distributed builder path writing final part_<id>_* files and final metadata.lance directly, with no dense rewrite.
• Add IndexStore::copy_index_file_to(source, dest, dest_store) for source-preserving copy-to-new-name; LanceIndexStore implements it with object-store copy and supports nested relative paths like staging/....
• Copy existing root partition files into staging for the public distributed-from-existing path, from_existing_index(..., fragment_mask=Some(...)), before advertising those partitions in staged metadata.
• During merge_index_files, discover staged metadata, sort original partition IDs, map them densely to 0..N, copy staging/part_<old_id>_* to final part_<new_id>_* without pre-deleting final paths, and then write final metadata.lance.
• Delete staged partition data and staged per-partition metadata only after final metadata is written, and keep that cleanup best-effort.
• Filter leftover staging/ files from FTS segment IndexMetadata.files during segment commit.
• On retry, existing final metadata.lance makes finalize a no-op; without it, staged sources remain authoritative and any partial final files are replaceable from staging.

Added regressions for staged distributed writes, distributed-from-existing finalize, partial final-file retry without root part_* deletion, preserving staging when final metadata writing fails, and excluding stale staging files from committed metadata. Validated locally with Rust fmt/tests/clippy, segmented FTS integration tests, Python extension build, the targeted Python progress test, and uv run make lint-rust.

[codecov]

Codecov Report

❌ Patch coverage is 83.25581% with 108 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/builder.rs	83.13%	48 Missing and 51 partials ⚠️
rust/lance-index/src/scalar/lance_format.rs	64.70%	4 Missing and 2 partials ⚠️
rust/lance-index/src/scalar.rs	0.00%	2 Missing ⚠️
rust/lance/src/index.rs	88.88%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

Two comments on FTS finalization.

[Xuanwo]

This is still unsafe with concurrent finalizers. metadata.lance is only checked at entry, so another finalizer may commit it before this one reaches delete_index_file(final_path). If this task then crashes after the delete, retries will no-op on the committed metadata while a final part_* file is missing.

Can we avoid deleting canonical final files, or re-check the commit marker before any destructive final-path operation?

[jackye1995]

Fixed in e9ccb116a by removing the pre-copy delete of canonical final part_<id>_* files. I agree full atomic multi-object finalize is not possible with the current object-store abstraction, so the guarantee here is idempotence: concurrent/retry finalizers copy deterministic staged bytes to the same final names, metadata.lance remains the commit marker, and post-commit cleanup only touches staging/.

I also extended test_merge_index_files_rewrites_partial_final_files_from_staging so it still proves partial final files are repaired from staging, and now asserts merge does not delete root part_* files first.

@Xuanwo could you take another look?

[Xuanwo]

Ignoring staging cleanup errors can commit staging/... into IndexMetadata.files. The commit path finalizes, then lists the whole index dir, and that listing does not filter staging files.

Can we filter staging/, return the canonical final file list from finalize, or fail the commit when cleanup fails?

[jackye1995]

Addressed by filtering staging/ from FTS segment IndexMetadata.files after finalization. Cleanup remains best-effort, but stale staged objects are treated as transient and are not recorded in committed metadata. I also extended test_commit_existing_index_segments_finalizes_fts_segments to create a leftover staging/orphan.lance after final metadata exists and assert the committed file list excludes it.

[Xuanwo]

Thank you working for this!