fix(gateway): keep hard-budget reservations when stream usage is incomplete - #251
fix(gateway): keep hard-budget reservations when stream usage is incomplete#251o-mid wants to merge 3 commits into
Conversation
…mplete Interrupted Anthropic streams can report partial input usage before the terminal message_delta arrives. Treat that as incomplete billing telemetry and reconcile hard budgets against the preflight estimate instead of the partial measured cost.
adrian-lorenzo
left a comment
There was a problem hiding this comment.
Thanks for the contribution!
usageComplete protects the client-abort path, but resolveMeteredCost only applies the conservative fallback when record.status === "error". A 200 stream can end cleanly or emit an in-band provider error before the terminal usage frame; that leaves usageComplete false while the status remains ok, and the hard-budget reservation is still reconciled down to partial measured spend.
Please make incomplete provider telemetry conservative regardless of the transport status, and add unit and integration coverage for a 200 stream ending after message_start without message_delta. The changed files also need formatting with Biome.
Once those cases are covered, this should be ready to approve.
Address review on theam#251: apply the preflight estimate whenever provider telemetry is incomplete, not only when the transport status is error. Cover the 200 stream that ends after message_start without message_delta.
|
Addressed review: |
Summary
Fixes #248.
When a streamed request is interrupted after Anthropic reports partial usage (for example
message_startinput tokens) but before the terminalmessage_delta, the gateway was reconciling hard-budget reservations down to that partial measured cost. That undercounts spend because the provider may have charged for additional generation whose usage never arrived.This change tracks whether provider usage telemetry is complete separately from whether any usage was observed, and keeps hard-budget reconciliation conservative when usage is incomplete.
Changes
UsageTeeexposesusageComplete, set when a terminal usage-bearing event is parsedRequestRecordcarriesusageCompletethrough the proxy pipelineresolveMeteredCost()falls back tomax(measured, estimated)when a provider-charged error has incomplete usage9creproduces client abort aftermessage_startonly and asserts the hard-budget counter stays at the reservationTest plan
pnpm --filter @facility/core build && pnpm --filter @facility/db build && pnpm --filter @facility/gateway testUsageTee.usageCompleteandresolveMeteredCostmessage_deltapartial usage