Skip to content

fix: exclude cache-write tokens from usage.inputTokens.noCache - #620

Open
OsamaAnsar wants to merge 1 commit into
OpenRouterTeam:mainfrom
OsamaAnsar:fix/usage-no-cache-excludes-cache-writes
Open

OsamaAnsar wants to merge 1 commit into
OpenRouterTeam:mainfrom
OsamaAnsar:fix/usage-no-cache-excludes-cache-writes

Conversation

@OsamaAnsar

Copy link
Copy Markdown

Description

computeTokenUsage computes inputTokens.noCache as prompt_tokens - cached_tokens and ignores prompt_tokens_details.cache_write_tokens. When cache writes are reported, noCache still contains the written tokens, so noCache + cacheRead + cacheWrite is larger than total.

Before (the fixture already used in compute-token-usage.test.ts, plus the example from OpenRouter's usage-accounting docs):

prompt_tokens=100 cached=20 cache_write=30 -> {"total":100,"noCache":80,"cacheRead":20,"cacheWrite":30}  noCache+read+write = 130
prompt_tokens=194 cached=0  cache_write=100 -> {"total":194,"noCache":194,"cacheRead":0,"cacheWrite":100} noCache+read+write = 294

After:

prompt_tokens=100 cached=20 cache_write=30 -> {"total":100,"noCache":50,"cacheRead":20,"cacheWrite":30}  sum = 100
prompt_tokens=194 cached=0  cache_write=100 -> {"total":194,"noCache":94,"cacheRead":0,"cacheWrite":100} sum = 194

The existing tests asserted noCache: 80 for the first case (compute-token-usage.test.ts, usage-accounting.test.ts, stream-usage-accounting.test.ts); I changed those expectations to 50.

Fix

noCache = Math.max(0, prompt_tokens - cached_tokens - (cache_write_tokens ?? 0)) in src/utils/compute-token-usage.ts. doGenerate and doStream of both the chat and completion models go through this one function, so no other call site needed changing. When cache_write_tokens is absent or null the result is unchanged (prompt_tokens - cached_tokens).

Why this is the intended contract

LanguageModelV4Usage.inputTokens.noCache in vercel/ai is documented as "The number of non-cached input (prompt) tokens used." (packages/provider/src/language-model/v4/language-model-v4-usage.ts). For the same prompt_tokens_details.cache_write_tokens field, @ai-sdk/openai computes (packages/openai/src/chat/convert-openai-chat-usage.ts, line 37, at vercel/ai 3216f06):

noCache: promptTokens - cachedTokens - (cacheWriteTokens ?? 0),

The total - cacheRead formula here comes from #409, which followed @ai-sdk/openai-compatible; that converter has no cache-write field (cacheWrite: undefined), so the formula was right for it but does not carry over once cacheWrite is populated.

What I could not confirm

I assumed prompt_tokens already includes cache-write tokens. OpenRouter's docs describe cache_write_tokens as "prompt tokens written to the cache" and their usage example (prompt_tokens: 194, cache_write_tokens: 100) is consistent with that, but I did not find a sentence stating it outright, and I have not made a live API call. If some upstream provider reports prompt_tokens excluding writes, this would under-count noCache for it.

Clamping

The reference @ai-sdk/openai converter does not clamp noCache (it does clamp text at 0). I clamp at 0 so inconsistent upstream numbers (cache tokens larger than prompt_tokens) cannot produce a negative count. Happy to drop the clamp to mirror the reference exactly.

Checklist

  • I have run pnpm stylecheck and pnpm typecheck
  • I have run pnpm test and all tests pass (I ran the two underlying commands, vitest --config vitest.node.config.ts --run and vitest --config vitest.edge.config.ts --run, on Node 22: 35 files / 576 tests pass in each; not the pnpm test script itself)
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable) (not applicable, no docs mention this)

Changeset

  • I have run pnpm changeset to create a changeset file (I added .changeset/exclude-cache-writes-from-no-cache.md by hand in the same format as the existing patch changesets)

Tests

New cases in src/utils/compute-token-usage.test.ts: cache write only, noCache + cacheRead + cacheWrite = total, absent/null cache_write_tokens (unchanged behaviour), and cache tokens exceeding prompt_tokens (clamped to 0). Reverting only src/utils/compute-token-usage.ts makes 6 tests fail (4 in compute-token-usage.test.ts, plus the two updated usage-accounting / stream-usage-accounting expectations); restoring it makes all pass.

Not tested: no live OpenRouter calls and no e2e tests (pnpm test:e2e needs an API key).

computeTokenUsage subtracted only cached_tokens from prompt_tokens, so when
prompt_tokens_details.cache_write_tokens was reported, noCache still counted
the cache-write tokens and noCache + cacheRead + cacheWrite exceeded total
(prompt_tokens 100, cached 20, written 30 gave noCache 80, sum 130).

Subtract cache writes as well, clamped at 0. This matches how
@ai-sdk/openai converts the same prompt_tokens_details.cache_write_tokens
field. The doGenerate and doStream paths of the chat and completion models
share computeTokenUsage, so they are fixed together.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant