Skip to content

fix(models): pass maxOutputTokens so output caps actually apply on AI SDK v5 - #63

Open
rajarshidattapy wants to merge 1 commit into
supermemoryai:mainfrom
rajarshidattapy:fix/max-output-tokens
Open

fix(models): pass maxOutputTokens so output caps actually apply on AI SDK v5#63
rajarshidattapy wants to merge 1 commit into
supermemoryai:mainfrom
rajarshidattapy:fix/max-output-tokens

Conversation

@rajarshidattapy

@rajarshidattapy rajarshidattapy commented Aug 13, 2026

Copy link
Copy Markdown

Fixes #62maxTokens has been a no-op since the AI SDK v5 upgrade, so every judge, answer
and extraction call has been running with no output cap.

Verified first

grep -n "maxOutputTokens" node_modules/ai/dist/index.d.ts
20:    maxOutputTokens?: number;          # v5 CallSettings

maxTokens is not a v5 call setting, so the SDK dropped it and defaultMaxTokens: 1000 never
reached a single request.

What was hiding it

Every call site built an untyped bag and cast it, which switches off excess-property checking —
precisely the check that would have flagged the rename:

const params: Record<string, unknown> = { ..., maxTokens: ... }
await generateText(params as Parameters<typeof generateText>[0])

Each of the five sites is now a plain typed object literal, with the conditional temperature
expressed as a spread instead of post-hoc mutation. Reintroducing the old name is now a compile
error, which I confirmed rather than assumed:

src/judges/anthropic.ts(33,7): error TS2353: Object literal may only specify known
properties, and 'maxTokens' does not exist in type 'CallSettings & ...'

That's the regression guard — no test needed for it.

One judgment call beyond the issue text

Making the cap effective is not safe as a pure rename, and shipping it that way would have
traded a cost bug for a correctness bug.

For OpenAI reasoning models the SDK forwards maxOutputTokens as max_completion_tokens
(@ai-sdk/openai/dist/index.js:743-748 remaps it), and that budget covers reasoning tokens
plus visible output
. A newly-enforced 1000 could therefore be consumed entirely by reasoning,
returning empty text — and parseJudgeResponse treats unparseable output as "incorrect", so
an o3 or gpt-5 judge would have silently scored questions wrong instead of erroring. Gemini
2.5/3 bill thinking against the same ceiling.

So defaultMaxTokens is now two tiers: SHORT_MAX_TOKENS (1000) for non-reasoning models, and
ROOMY_MAX_TOKENS (25000) for reasoning/thinking models and for unrecognised aliases. The
ceiling is a runaway guard, not a budget — short verdicts still use a fraction of it, so the
cost win the issue asks for lands on the models where it's safe.

Two related guards, both cheap, both protecting against truncation that the fix newly makes
possible:

  • Empty judge response now throws (judges/base.ts) instead of scoring "incorrect". The
    evaluate phase records a real, resumable failure rather than quietly biasing accuracy.
  • Extraction cap raised 2000 -> 8000 and truncation is logged (prompts/extraction.ts). A
    truncated extraction silently shrinks the memory corpus that the filesystem and rag
    providers are scored on, which would read as a provider quality problem rather than a harness
    limit.

If reviewers would rather keep the rename minimal, the tiering and the two guards are separable
hunks — but I'd argue the rename alone is not shippable.

Also in scope, per the issue

  • Deleted the dead maxTokensParam field from ModelConfig, all 21 registry entries, and the
    five prefix-fallback branches. The SDK already normalizes the per-provider parameter name, so
    there was nothing for it to configure.
  • Updated src/judges/README.md, which documented that field to anyone adding a judge.

Tests

src/utils/models.test.ts asserts the invariant that matters — reasoning/thinking models and
unknown aliases carry enough headroom to emit a verdict, and known non-reasoning models stay
tightly capped.

It earned its place immediately: it failed on o5-mini, a hypothetical future reasoning model
that matches none of the o1|o3|o4 prefixes and so lands in the final catch-all, which I had
left at 1000. Fixed there too.

bun test 4/4 green, tsc --noEmit clean, prettier --check clean on all touched files.

Notes for the reviewer

  • src/judges/index.ts is unformatted at HEAD and I left it alone; every file I touched is
    prettier-clean.
  • I ran bun install --frozen-lockfile to verify the SDK types against the pinned version;
    bun.lock and package.json are unmodified.
  • ROOMY_MAX_TOKENS = 25000 is a deliberately conservative ceiling, not a measured figure. If
    you have real reasoning-token numbers from a run, that constant is the one knob to tune.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

maxTokens is silently ignored on AI SDK v5 — every judge and answer call runs uncapped

1 participant