fix(benchmarks): make question order deterministic and load() idempotent

[Agnik47]

Fixes #75.

Problem

Question order is whatever the filesystem hands back

// src/benchmarks/longmemeval/index.ts:187
const files = readdirSync(questionsDir).filter((f) => f.endsWith(".json"))

readdirSync order is unspecified: hash-ordered on ext4 with dir_index, roughly but not reliably lexicographic on APFS/NTFS, and liable to change when files are rewritten. That order then decides which questions a limited run covers — the orchestrator slices the front of the list for --limit N (src/orchestrator/index.ts:227), for sampling.mode === "limit", and for consecutive sampling.

So run -p supermemory -b locomo --limit 50 on a maintainer's Mac and on CI's Linux box benchmark different sets of 50 questions, and the accuracy difference between them is indistinguishable from a real provider difference. For a benchmark whose published numbers are meant to be independently reproducible, that is the whole point undone.

load() is not idempotent

All three benchmarks push into this.questions (and LongMemEval into this.data) without clearing first, and sessionsMap.set overwrites. Calling load() twice on one instance duplicates every question, so getQuestions() returns each one twice — double-counting in the report and halving the effective coverage of --limit.

The CLI path builds a fresh instance per run, but the server caches benchmark instances (src/server/routes/runs.ts:16-24), so this is reachable rather than theoretical.

Fix

• .sort() the question file list in LongMemEvalBenchmark.loadQuestions, so a limited or sampled run covers the same questions on every machine. LoCoMo and ConvoMem read a single JSON file, so their order was already deterministic.
• Reset data / questions / sessionsMap at the start of the load in all three benchmarks. For ConvoMem the reset happens after JSON.parse succeeds, so a failed reload doesn't discard data that was already loaded.

No public API change, and a fresh single load() behaves exactly as before.

Tests

New src/benchmarks/question-order.test.ts (7 tests, bun test). It mocks fs to serve a deliberately unsorted directory listing — ["q3.json", "q1.json", "q10.json", "q2.json"], i.e. what ext4 hands back — so the ordering assertion is deterministic on every platform rather than depending on the developer's filesystem. The real fs is captured up front and restored in afterAll, so the mock doesn't leak into other test files.

• questions come back in stable lexicographic order regardless of directory order
• the --limit 2 slice selects the same pair on any filesystem (it would have been a different pair pre-fix)
• sessions stay attached to the right question after sorting
• loading twice does not duplicate questions — LongMemEval, LoCoMo and ConvoMem
• ground truth and haystack sessions survive a reload

5 of the 7 fail on main; all 7 pass with this change.

Left out

The issue also suggests recording the resolved targetQuestionIds in the report. That means touching src/orchestrator/phases/report.ts, which #80 is currently rewriting, so I left it out to keep this reviewable and conflict-free. Happy to add it separately once #80 lands.

bunx tsc --noEmit is clean and the new file is Prettier-clean. The three touched files already fail bun run format:check on main (along with ~69 others), so I left their pre-existing formatting alone to keep the diff scoped.