Skip to content

fix(gateway): keep hard-budget reservations when stream usage is incomplete - #251

Open
o-mid wants to merge 3 commits into
theam:mainfrom
o-mid:fix/gateway-interrupted-stream-budget
Open

fix(gateway): keep hard-budget reservations when stream usage is incomplete#251
o-mid wants to merge 3 commits into
theam:mainfrom
o-mid:fix/gateway-interrupted-stream-budget

Conversation

@o-mid

@o-mid o-mid commented Sep 1, 2026

Copy link
Copy Markdown

Summary

Fixes #248.

When a streamed request is interrupted after Anthropic reports partial usage (for example message_start input tokens) but before the terminal message_delta, the gateway was reconciling hard-budget reservations down to that partial measured cost. That undercounts spend because the provider may have charged for additional generation whose usage never arrived.

This change tracks whether provider usage telemetry is complete separately from whether any usage was observed, and keeps hard-budget reconciliation conservative when usage is incomplete.

Changes

  • UsageTee exposes usageComplete, set when a terminal usage-bearing event is parsed
  • RequestRecord carries usageComplete through the proxy pipeline
  • resolveMeteredCost() falls back to max(measured, estimated) when a provider-charged error has incomplete usage
  • Integration test 9c reproduces client abort after message_start only and asserts the hard-budget counter stays at the reservation

Test plan

  • pnpm --filter @facility/core build && pnpm --filter @facility/db build && pnpm --filter @facility/gateway test
  • New unit tests for UsageTee.usageComplete and resolveMeteredCost
  • Existing client-abort test still passes for terminal message_delta partial usage

…mplete

Interrupted Anthropic streams can report partial input usage before the
terminal message_delta arrives. Treat that as incomplete billing telemetry
and reconcile hard budgets against the preflight estimate instead of the
partial measured cost.

@adrian-lorenzo adrian-lorenzo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

usageComplete protects the client-abort path, but resolveMeteredCost only applies the conservative fallback when record.status === "error". A 200 stream can end cleanly or emit an in-band provider error before the terminal usage frame; that leaves usageComplete false while the status remains ok, and the hard-budget reservation is still reconciled down to partial measured spend.

Please make incomplete provider telemetry conservative regardless of the transport status, and add unit and integration coverage for a 200 stream ending after message_start without message_delta. The changed files also need formatting with Biome.

Once those cases are covered, this should be ready to approve.

o-mid and others added 2 commits September 1, 2026 19:39
Address review on theam#251: apply the preflight estimate whenever provider
telemetry is incomplete, not only when the transport status is error.
Cover the 200 stream that ends after message_start without message_delta.
@o-mid

o-mid commented Sep 1, 2026

Copy link
Copy Markdown
Author

Addressed review: resolveMeteredCost now applies the conservative fallback whenever providerMayHaveCharged and usage is incomplete, regardless of HTTP status. Added test 9e for a 200 stream that ends after message_start without message_delta, plus a unit test for the ok/incomplete case. Ran Biome on changed files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway can undercount interrupted streams when partial usage is nonzero

2 participants