Gateway can undercount interrupted streams when partial usage is nonzero

[juansolana]

Summary

An interrupted provider stream can report partial usage before its terminal usage frame arrives. When that happens, Facility can treat the partial nonzero usage as the final cost and reconcile the hard-budget reservation down to it, even though the provider may have charged for additional generation whose usage was never received.

The existing conservative fallback only applies when measured cost is exactly zero.

Where

services/gateway/src/metering.ts currently falls back to the preflight estimate only when:

record.status === "error" &&
record.providerMayHaveCharged &&
measuredCost === 0

Otherwise it uses measuredCost, and reconcileBudgetReservations() releases the remainder of the reservation.

This becomes unsafe when measuredCost > 0 represents partial rather than final usage.

Anthropic streaming provides a deterministic example:

• message_start reports input/cache usage;
• the terminal usage-bearing event reports final output usage;
• if the stream is interrupted between them, UsageTee can contain valid nonzero usage while the final output count is still unknown.

The request is recorded as an error with providerMayHaveCharged: true, but because measured cost is already nonzero, Facility can reconcile the reservation as though that partial amount were the final provider cost.

Example

Suppose a request has a conservative estimated maximum cost of $4, so Facility reserves $4 against a hard budget.

During streaming, Facility receives enough usage telemetry to measure $0.30 of input cost, but the connection is interrupted before the final output-usage event arrives.

The provider may ultimately charge, for example, $2.20.

Facility only knows:

measured cost so far: $0.30
final usage: unknown
provider may have charged: true

Because measured cost is nonzero, the current fallback does not use the $4 estimate. The reservation can therefore be reconciled from $4 down to $0.30, releasing $3.70 back into the hard budget even though the final provider charge is unknown.

Repeated interrupted requests could therefore make Facility's hard-budget counter materially lower than provider spend.

Existing test gap

services/gateway/test/gateway.test.ts already has:

Client abort aborts upstream and records partial usage

which looks like a good reproduction surface.

The test verifies that partial usage is recorded and that spentCents > 0, but it does not verify whether the remainder of a hard-budget reservation is released when the received usage is incomplete.

Suggested direction

Track whether provider usage is complete separately from whether any usage was observed.

For example, UsageTee could expose a completeness/terminal signal derived from the provider's final usage-bearing event or from a fully parsed non-streamed response.

When:

• the request errored,
• the provider may have charged, and
• usage is incomplete,

hard-budget reconciliation should remain conservative rather than releasing the reservation based only on partial usage.

The request record can still preserve the usage actually observed, the important distinction is between observed usage and final billable usage known.

Acceptance criteria

• Interrupted Anthropic streams with partial usage but no terminal usage event do not reconcile hard-budget spend down to the partial measured amount.
• Complete successful streams continue reconciling reservations to actual usage.
• Failures before the provider can charge still release the reservation as today.
• Zero-usage provider-charged errors preserve the existing conservative behavior.
• The client-abort integration test asserts the resulting hard-budget counter, not merely that it is greater than zero.
• Successfully completed requests are not double-counted.

[o-mid]

Opened #251 to track stream usage completeness separately from observed usage, and keep hard-budget reconciliation conservative when an interrupted stream only reports partial telemetry.