Repository navigation
fix: exclude cache-write tokens from usage.inputTokens.noCache - #620
Open
OsamaAnsar wants to merge 1 commit into
Open
OsamaAnsar wants to merge 1 commit into
OsamaAnsar wants to merge 1 commit into
Conversation
computeTokenUsage subtracted only cached_tokens from prompt_tokens, so when prompt_tokens_details.cache_write_tokens was reported, noCache still counted the cache-write tokens and noCache + cacheRead + cacheWrite exceeded total (prompt_tokens 100, cached 20, written 30 gave noCache 80, sum 130). Subtract cache writes as well, clamped at 0. This matches how @ai-sdk/openai converts the same prompt_tokens_details.cache_write_tokens field. The doGenerate and doStream paths of the chat and completion models share computeTokenUsage, so they are fixed together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
computeTokenUsagecomputesinputTokens.noCacheasprompt_tokens - cached_tokensand ignoresprompt_tokens_details.cache_write_tokens. When cache writes are reported,noCachestill contains the written tokens, sonoCache + cacheRead + cacheWriteis larger thantotal.Before (the fixture already used in
compute-token-usage.test.ts, plus the example from OpenRouter's usage-accounting docs):After:
The existing tests asserted
noCache: 80for the first case (compute-token-usage.test.ts,usage-accounting.test.ts,stream-usage-accounting.test.ts); I changed those expectations to 50.Fix
noCache = Math.max(0, prompt_tokens - cached_tokens - (cache_write_tokens ?? 0))insrc/utils/compute-token-usage.ts.doGenerateanddoStreamof both the chat and completion models go through this one function, so no other call site needed changing. Whencache_write_tokensis absent or null the result is unchanged (prompt_tokens - cached_tokens).Why this is the intended contract
LanguageModelV4Usage.inputTokens.noCacheinvercel/aiis documented as "The number of non-cached input (prompt) tokens used." (packages/provider/src/language-model/v4/language-model-v4-usage.ts). For the sameprompt_tokens_details.cache_write_tokensfield,@ai-sdk/openaicomputes (packages/openai/src/chat/convert-openai-chat-usage.ts, line 37, at vercel/ai3216f06):The
total - cacheReadformula here comes from #409, which followed@ai-sdk/openai-compatible; that converter has no cache-write field (cacheWrite: undefined), so the formula was right for it but does not carry over oncecacheWriteis populated.What I could not confirm
I assumed
prompt_tokensalready includes cache-write tokens. OpenRouter's docs describecache_write_tokensas "prompt tokens written to the cache" and their usage example (prompt_tokens: 194,cache_write_tokens: 100) is consistent with that, but I did not find a sentence stating it outright, and I have not made a live API call. If some upstream provider reportsprompt_tokensexcluding writes, this would under-countnoCachefor it.Clamping
The reference
@ai-sdk/openaiconverter does not clampnoCache(it does clamptextat 0). I clamp at 0 so inconsistent upstream numbers (cache tokens larger thanprompt_tokens) cannot produce a negative count. Happy to drop the clamp to mirror the reference exactly.Checklist
pnpm stylecheckandpnpm typecheckpnpm testand all tests pass (I ran the two underlying commands,vitest --config vitest.node.config.ts --runandvitest --config vitest.edge.config.ts --run, on Node 22: 35 files / 576 tests pass in each; not thepnpm testscript itself)Changeset
pnpm changesetto create a changeset file (I added.changeset/exclude-cache-writes-from-no-cache.mdby hand in the same format as the existing patch changesets)Tests
New cases in
src/utils/compute-token-usage.test.ts: cache write only,noCache + cacheRead + cacheWrite = total, absent/nullcache_write_tokens(unchanged behaviour), and cache tokens exceedingprompt_tokens(clamped to 0). Reverting onlysrc/utils/compute-token-usage.tsmakes 6 tests fail (4 incompute-token-usage.test.ts, plus the two updatedusage-accounting/stream-usage-accountingexpectations); restoring it makes all pass.Not tested: no live OpenRouter calls and no e2e tests (
pnpm test:e2eneeds an API key).