Vector Set recall smoke tests

[badrishc]

What

Recall-oriented smoke tests for Garnet Vector Sets that stress a DiskANN graph across the physical storage configurations that matter for disk-tiered operation. (A CopyUpdater sizing change was briefly included and has since been reverted — see the last section.)

Smoke test coverage

• quantization mode — NOQUANT / Q8 / BIN
• larger-than-memory — records spill to the storage tier during load, or are evicted to disk after an in-memory build
• read-cache vs copy-reads-to-tail vs neither
• save → restart → recover, optionally recovering into a much smaller log

Invariant

Physical robustness: a graph that answers queries well while resident in memory must keep answering them well once the same records are served from disk or recovered from a checkpoint.

Each test builds a small, well-clustered, deterministic graph (Dim=32, 16 clusters, M=16, EF=64, fixed seed, single connection, sequential VADD), measures recall@10 against a brute-force ground truth, applies a physical stressor, and asserts recall does not collapse (MaxRecallDrop = 0.20). Spill is forced and verified with DEBUG FLUSHANDEVICT + the store head/tail addresses. Everything is single-threaded and the graphs are intentionally tiny, so the whole suite runs in ~30s.

Tests (18 cases)

Test	Coverage
RecallSurvivesFlushAndEvict(quant, {none/readcache/copytotail})	larger-than-memory served from disk × read-cache combinations
RecallSurvivesLargerThanMemoryLoad(quant)	graph built while spilling to disk (256k log / 32k pages)
RecallSurvivesSaveAndRecover(quant, recoverIntoSmallerLog)	save → restart → recover, optionally into a much smaller (disk-tiered) log

Known-failing cases (deliberate regression guards)

The Q8 cases that read from disk (FlushAndEvict, larger-than-memory load, and recover) currently fail against diskann-garnet 4.0.0. The per-dimension Q8 quantization table is native in-memory-only state that is lost when the index is recreated after eviction/recovery and cannot be rebuilt, so stored codes are decoded with a missing table and recall collapses (~1.0 to ~0.03-0.33). NOQUANT and BIN (table-free) are robust across every configuration.

Current result against main (diskann 4.0.0): 12 pass / 6 fail, where the 6 failures are exactly the Q8 disk cases:

evict         Q8 [none/readcache]              1.000 -> 0.092
evict         Q8 [copytotail]                  1.000 -> 0.045
largerthanmem Q8                               0.333
recover       Q8 [recoverIntoSmallerLog=False] 1.000 -> 0.195
recover       Q8 [recoverIntoSmallerLog=True]  1.000 -> 0.073

These are intentional regression guards for the Q8 disk-recall bug - they should start passing once the quantization table is persisted with (or rebuildable for) a recreated/recovered index. Opening as draft since these Q8 cases are expected red until that fix lands.

CopyUpdater "Destination is too short" — investigated, fix reverted

An earlier commit sized the CopyUpdater destination to max(WriteDesiredSize + ValueAlignmentBytes, existingValue.Length) to stop a rare ArgumentException ("Destination is too short") seen during a BIN larger-than-memory build at 4 KB pages. That commit has been reverted after root-cause investigation showed it guards a condition that does not occur.

Instrumented runs (per-(namespace,key) write-size tracking with liveness confirmation, plus a disk-chain-walk namespace/key match check, run isolated and across the full smoke suite at 4 KB pages under CPU stress) established:

• DiskANN RMWs never change a record's size. Exactly two copy/inplace shapes are produced and each is perfectly constant across a build (a 68 B inline NeighborList and an 8192 B overflow record). Zero size changes were ever observed for any key.
• The pending (disk) CopyUpdater always matches the correct key and namespace for DiskANN records (zero mismatches). GarnetKeyComparer's generic path compares NamespaceBytes + KeyBytes, so a small-record RMW never copy-updates a larger record.

Because the source value equals WriteDesiredSize + ValueAlignmentBytes for a correctly-matched, size-stable record, oldValueAligned.Length always equals newValueAligned.Length and the CopyTo cannot overflow in normal operation. The max(...) sizing therefore protects against nothing that happens; worse, if ValueSpan.Length were ever transiently corrupt (large) it would attempt a huge allocation (OOM) rather than fail fast. The original crash was most plausibly the separate index-record recreate path (already correctly sized in HEAD via VectorManager.IndexSize) or an extremely rare eviction/pending-IO race; either way it must be root-caused as a race/corruption, not sized around.

Notes

• Follows the existing Garnet.test.vectorset conventions (TestBase, TestUtils.CreateGarnetServer, VADD/VSIM via StackExchange.Redis, binary int element ids).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[tiagonapoli]

nit: we have DiskANNSyntheticRecallTests which has methods duplicated with some here, maybe worth evaluating if it's possible to reuse? Or updating DiskANNSyntheticRecallTests to add fixtures that will also add quantization and spilling to disk?