Repository navigation
Commit 89ab11a
Report a fully streamed response as completed when the client closes early
A client that closes its socket the instant it has read the terminal frame
(`data: [DONE]`, `event: message_stop`, or an NDJSON `"done":true` frame)
cancels the request context before the ReverseProxy copy has consumed the
upstream EOF, which the llama.cpp router sends a few milliseconds later. That
cancel also propagates to the upstream request, so the copy ends with a
context error rather than EOF. Both reporters keyed purely on the cancelled
context: the disconnect watcher emitted "cancelled" at once and the deferred
finalize classified the same condition as cancelled. In a recorded eight-node
run, 21 of 120 fully delivered 200 streams were labelled cancelled; a client
that read to EOF got 24 of 24 completed.
The committed response capture now records when the body is complete: the
route's terminal frame has been written (line-anchored for SSE, so model text
quoting the marker does not count), Content-Length bytes have been written, or
the upstream body reached EOF and its bytes are out. The disconnect watcher
does not emit when the body is complete, and finalize treats a cancelled
context as "cancelled" only when it is not. A close before the last frame, a
failed write to a dead client, and a truncated upstream are classified as
before.
Tests run a real server and raw TCP clients for the OpenAI, Anthropic and
NDJSON dialects in both the late-close and mid-stream-close cases, and the
completion record's line anchoring, content length and EOF paths. spec.md
section 5.4 gains the outcome row and the reasoning.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>1 parent 9c8632c commit 89ab11a
3 files changed
Lines changed: 652 additions & 16 deletions
0 commit comments