Skip to content

Commit 89ab11a

Browse files
Report a fully streamed response as completed when the client closes early
A client that closes its socket the instant it has read the terminal frame (`data: [DONE]`, `event: message_stop`, or an NDJSON `"done":true` frame) cancels the request context before the ReverseProxy copy has consumed the upstream EOF, which the llama.cpp router sends a few milliseconds later. That cancel also propagates to the upstream request, so the copy ends with a context error rather than EOF. Both reporters keyed purely on the cancelled context: the disconnect watcher emitted "cancelled" at once and the deferred finalize classified the same condition as cancelled. In a recorded eight-node run, 21 of 120 fully delivered 200 streams were labelled cancelled; a client that read to EOF got 24 of 24 completed. The committed response capture now records when the body is complete: the route's terminal frame has been written (line-anchored for SSE, so model text quoting the marker does not count), Content-Length bytes have been written, or the upstream body reached EOF and its bytes are out. The disconnect watcher does not emit when the body is complete, and finalize treats a cancelled context as "cancelled" only when it is not. A close before the last frame, a failed write to a dead client, and a truncated upstream are classified as before. Tests run a real server and raw TCP clients for the OpenAI, Anthropic and NDJSON dialects in both the late-close and mid-stream-close cases, and the completion record's line anchoring, content length and EOF paths. spec.md section 5.4 gains the outcome row and the reasoning. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
1 parent 9c8632c commit 89ab11a

3 files changed

Lines changed: 652 additions & 16 deletions

File tree

0 commit comments

Comments
 (0)