fix(models): pass maxOutputTokens so output caps actually apply on AI SDK v5 - #63
Open
rajarshidattapy wants to merge 1 commit into
Open
fix(models): pass maxOutputTokens so output caps actually apply on AI SDK v5#63rajarshidattapy wants to merge 1 commit into
rajarshidattapy wants to merge 1 commit into
Conversation
…or reasoning models
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #62 —
maxTokenshas been a no-op since the AI SDK v5 upgrade, so every judge, answerand extraction call has been running with no output cap.
Verified first
maxTokensis not a v5 call setting, so the SDK dropped it anddefaultMaxTokens: 1000neverreached a single request.
What was hiding it
Every call site built an untyped bag and cast it, which switches off excess-property checking —
precisely the check that would have flagged the rename:
Each of the five sites is now a plain typed object literal, with the conditional
temperatureexpressed as a spread instead of post-hoc mutation. Reintroducing the old name is now a compile
error, which I confirmed rather than assumed:
That's the regression guard — no test needed for it.
One judgment call beyond the issue text
Making the cap effective is not safe as a pure rename, and shipping it that way would have
traded a cost bug for a correctness bug.
For OpenAI reasoning models the SDK forwards
maxOutputTokensasmax_completion_tokens(
@ai-sdk/openai/dist/index.js:743-748remaps it), and that budget covers reasoning tokensplus visible output. A newly-enforced 1000 could therefore be consumed entirely by reasoning,
returning empty text — and
parseJudgeResponsetreats unparseable output as"incorrect", soan
o3orgpt-5judge would have silently scored questions wrong instead of erroring. Gemini2.5/3 bill thinking against the same ceiling.
So
defaultMaxTokensis now two tiers:SHORT_MAX_TOKENS(1000) for non-reasoning models, andROOMY_MAX_TOKENS(25000) for reasoning/thinking models and for unrecognised aliases. Theceiling is a runaway guard, not a budget — short verdicts still use a fraction of it, so the
cost win the issue asks for lands on the models where it's safe.
Two related guards, both cheap, both protecting against truncation that the fix newly makes
possible:
judges/base.ts) instead of scoring"incorrect". Theevaluate phase records a real, resumable failure rather than quietly biasing accuracy.
prompts/extraction.ts). Atruncated extraction silently shrinks the memory corpus that the
filesystemandragproviders are scored on, which would read as a provider quality problem rather than a harness
limit.
If reviewers would rather keep the rename minimal, the tiering and the two guards are separable
hunks — but I'd argue the rename alone is not shippable.
Also in scope, per the issue
maxTokensParamfield fromModelConfig, all 21 registry entries, and thefive prefix-fallback branches. The SDK already normalizes the per-provider parameter name, so
there was nothing for it to configure.
src/judges/README.md, which documented that field to anyone adding a judge.Tests
src/utils/models.test.tsasserts the invariant that matters — reasoning/thinking models andunknown aliases carry enough headroom to emit a verdict, and known non-reasoning models stay
tightly capped.
It earned its place immediately: it failed on
o5-mini, a hypothetical future reasoning modelthat matches none of the
o1|o3|o4prefixes and so lands in the final catch-all, which I hadleft at 1000. Fixed there too.
bun test4/4 green,tsc --noEmitclean,prettier --checkclean on all touched files.Notes for the reviewer
src/judges/index.tsis unformatted at HEAD and I left it alone; every file I touched isprettier-clean.
bun install --frozen-lockfileto verify the SDK types against the pinned version;bun.lockandpackage.jsonare unmodified.ROOMY_MAX_TOKENS = 25000is a deliberately conservative ceiling, not a measured figure. Ifyou have real reasoning-token numbers from a run, that constant is the one knob to tune.