perf(index): cache small-index posting, doc, and vector-probe reads

[hamersaw]

Summary

Querying a small full-text index (e.g. a mem_wal flushed-generation index) re-paid object-store IO on every query for metadata that is never cached. Three uncached read paths, each fixed for small indexes while leaving large-index behavior unchanged:

1. Posting metadata — posting_len_for_token / posting_metadata_for_token issued one tiny single-row read_range against the posting file per term per partition, never cached. For a small index (≤256Ki tokens) bulk-load the whole posting metadata once into the existing OnceCell (small_index_bulk_metadata); large indexes keep the O(1) single-row path.
2. Doc row-ids — DeferredDocSet::resolve_row_ids did targeted (uncached) row-id reads on every query. For a small partition (≤256Ki docs) load and cache the whole ROW_ID column instead.
3. Vector-index probe — the "is this a vector index?" file-existence check for indexes without files metadata issued a HEAD per generic open. Memoize per uuid in the session index cache (IsVectorIndexProbeKey).

Changes

• lance-index/.../inverted/index.rs: small_index_bulk_metadata + updated test_bm25_stats_for_terms_is_lazy.
• lance-index/.../inverted/lazy_docset.rs: small-partition row-id column caching in resolve_row_ids.
• lance/src/index.rs: memoized is_vector_index existence probe.
• lance/src/session/index_caches.rs: IsVectorIndexProbe cache key.

Validation

cargo test -p lance-index test_bm25_stats_for_terms_is_lazy passes (asserts the first stats lookup issues exactly one posting read and subsequent lookups issue none). Validated end-to-end against a WAL FTS benchmark: a warm query dropped from re-reading per-term posting offsets + doc row-ids + the vector probe each query to zero such reads (all served from cache).

🤖 Generated with Claude Code

[hamersaw]

This PR was fixing a performance issue on a path we should not have been on, closing accordingly.