Skip to content

fix(providers): honour the search limit so every provider gets the same result budget - #65

Open
rajarshidattapy wants to merge 1 commit into
supermemoryai:mainfrom
rajarshidattapy:fix/provider-search-limit
Open

fix(providers): honour the search limit so every provider gets the same result budget#65
rajarshidattapy wants to merge 1 commit into
supermemoryai:mainfrom
rajarshidattapy:fix/provider-search-limit

Conversation

@rajarshidattapy

Copy link
Copy Markdown

Fixes #64 — Supermemory hardcoded limit: 30 while the search phase asked every provider for
10, so it received ~3x the answer-prompt context of the providers it was being compared with.

Also affected: Zep (the issue's table under-counts it)

While verifying the per-provider table I found a second violation of the same contract. Zep
searches edges and nodes as two separate queries and concatenates them:

const edgeLimit = options.limit || 20
const nodeLimit = Math.min(edgeLimit, 10)
// ...results.push(...edges); results.push(...nodes)

With limit: 10 that requests 10 of each and returns up to 20, not the 10 the issue's
table records. Fixing only Supermemory would have left Zep with roughly double its peers'
context, so the comparison would still not have been like-for-like.

Confirmed compliant: mem0 (top_k: options.limit || 30), filesystem, rag.

Changes

providers/supermemorylimit: options.limit || 30, matching how the same call already
treats options.threshold one line below. Also normalised the tab-indented searchMode/
include block, since those are the lines the fix touches and the file was the one file in this
diff that was already failing prettier --check at HEAD.

providers/zep — the limit is now split between the two scopes instead of applied to each,
via an extracted splitSearchBudget(). Kept edge-heavy at 2:1 to preserve the previous
20-edge / 10-node intent, so limit: 10 becomes 7 edges + 3 nodes rather than 10 + 10.

orchestrator/phases/search — the limit is now enforced at the point results are consumed,
not just requested. A provider that over-returns is truncated to the shared budget and logged:

WARN greedy returned 30 results for a limit of 10; truncating to keep providers comparable

This is the part that makes the fairness property hold going forward. Patching the two current
offenders fixes today's numbers; the chokepoint means a provider added next month cannot quietly
reintroduce the same advantage. The magic 10 and 0.3 are now named constants, with the
k = 10 coupling to calculateRetrievalMetrics written down.

Note the truncation had to come with the Zep fix rather than instead of it: edges are pushed
before nodes, so a blind slice(0, 10) on unfixed Zep output would have silently dropped every
node result and turned Zep into an edges-only provider.

types/provider — documented limit as a hard cap rather than a hint, including the
"split the budget across scopes" rule, since SearchOptions is where a provider author looks.
The issue suggested surfacing this as a declared capability; documenting the contract and
enforcing it centrally seemed better than adding an interface knob no provider needs yet.

Tests

src/orchestrator/search-limit.test.ts:

  • splitSearchBudget never exceeds the caller's limit across limits 1..50, always leaves at
    least one node slot, and stays edge-heavy.
  • runSearchPhase against a stub provider that returns 30 results for a limit of 10: asserts
    both the checkpoint and the persisted result file are capped at 10. The file matters
    independently — the answer phase reads context from disk, not from the checkpoint, so a fix
    that capped only one of the two would still have leaked the advantage into the prompt.
  • A compliant provider returning 4 results is passed through untouched, so the guard cannot
    quietly become a floor.

bun test 3/3, tsc --noEmit clean, prettier --check clean on all touched files.

Effect on existing numbers

Any Supermemory or Zep run recorded before this change is not comparable with runs from other
providers, and not comparable with post-fix runs of its own: accuracy, avgContextTokens, and
the MemScore token component all move. Published leaderboard entries for those two providers
should be regenerated rather than mixed with new ones.

Follow-up worth its own issue

Result count is now equal, but result size is not. Supermemory requests
include: { summaries: true, chunks: true }, so one result can carry substantially more text
than one mem0 memory or one Zep edge. contextTokens is measured per prompt so it reports this
honestly, but "10 results" still does not mean equal context across providers. Capping by
token budget rather than result count would be the real like-for-like fix, and that is a
design decision rather than a bug fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant