Skip to content

Agent: an app that quits right after sending changes does not lose them (after #16) - #42

Open
Skyfuller1303 wants to merge 4 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:agent-keep-last-changes-on-quit
Open

Skyfuller1303 wants to merge 4 commits into
itrycodinganditsbad:mainfrom
Skyfuller1303:agent-keep-last-changes-on-quit

Conversation

@Skyfuller1303

Copy link
Copy Markdown

Merge after #16. The diff of this PR alone: main...agent-keep-last-changes-on-quit. This branch is based on main; it waits for #16 only because the agent's live_documents test binary runs out of memory on main until #16 is in.

An app that sends its last changes and quits right after can lose them: the agent takes in the first ones and refuses or never reads the rest. This is on main today. It shows when a live document's last keystrokes before closing the app don't reach the other machine.

Cause

Two things in the agent drop changes that the app had already sent:

  1. The hub destroys a connection whose output is closed. The docs task of an app that closes its socket enqueues its updates, the document's close and Gone, then exits and drops the receiver of the hub's output. The hub journals update A and sends A's ack to that dead receiver. tell() then calls gone() at once, which removes the connection's handles. Update B, still queued, finds no handle and is refused instead of journalled. gone() ran before the Gone that sits behind B in the queue.
  2. The socket loop gives up on the first failed write. When a send to the app fails, the loop breaks without reading the frames the app sent before. The app's teardown sends a WebSocket close and shuts the TCP connection down right after, so the acks have nowhere to go and the rest of its frames stay unread.

Updates the agent has acknowledged are durable already (the journal is synced before the ack). Only the ones still in flight at teardown were lost.

What

  • hub.rs, tell() no longer removes the connection. When the output is closed or full, the connection's sender is dropped: nothing more is sent to it, and its socket ends once it has taken what was queued. Its handles stay. They are cleaned up by the connection's own DocsCall::Gone, which is in the same FIFO behind every change the connection queued. A slow client is disconnected the same way, so that order holds there too.
  • docs.rs, serve():
    • always enqueues Gone when the connection ends, last (a dropped output used to skip it, which would now leak the connection);
    • a write to the app that fails because the app is gone (an event, or the reply to a refused frame) no longer ends the connection. The loop stops writing, keeps reading and forwarding the app's frames until its close or EOF, and gives up after 5 seconds (READ_ON). The usual frame limits apply meanwhile.
    • an app that is merely too slow (no message taken within 10 s) is still dropped at once, as before. The send helpers now say which of the two happened (Unsent::Closed or Unsent::Slow). Reading on from a socket whose write timed out half-way is not safe: my first version did, and the existing an_app_that_misbehaves_is_refused_or_dropped_and_the_rest_go_on test then ran out of memory.
    • the closing WebSocket close frame is sent under the same 10 s timeout, so an app too slow to read cannot hold the task there;
    • a muted connection drops its receiver of the hub's output, so the hub finds out at once that nothing is read any more;
    • limit: a receive error (an abrupt reset, as opposed to the app's close or EOF) still ends the connection, muted or not. Reading on after an error is not safe to rely on, and it would not help: on Linux a reset drops the frames the agent has received but not yet read, so they never reach the agent. Noted in a comment.
  • Imports. When tell() finds a connection dead, every document whose import was asked of that connection is asked of the next open waiting for it that can still be told (ask_import skips connections that take no messages), instead of waiting for the dead connection's Gone. Opens of a dead connection stay waiting until its Gone, which gives the join up if nobody else waits.
  • App library (crates/app), Documents::drop: sends the close, waits up to a second for the agent's close (the reader thread ends on it), and only then shuts the stream down. Before, it shut the stream down at once. A connection closed while the agent's answers are still unread is reset, and on Linux a reset drops frames the agent has received but not read yet, which were the last changes before the app quit (found on Linux CI: the first version of the main test dropped a raw socket at that moment and failed now and then; reproduced in WSL under CPU load, 2 failures in 40 runs, and the library test with the old drop once in 150).
  • No protocol change.
  • Test hooks, all #[doc(hidden)] in docs: FAIL_WRITES (every write to an app fails), FAIL_WRITES_OF (the same for the connections of one named app) and READ_ON_MS (how long a muted connection is read on), like record_delay and announce_delay in RunOptions.

Checks

On Windows, through a memory-limited runner:

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test -p craft-cloud-agent -p craft-cloud-app --locked --no-fail-fast
  • cargo +1.95 check --workspace --all-targets --locked (MSRV)

Merge after #16. Without it, the live_documents test binary runs out of a 16 GiB limit on main (the misbehaving-app test alone needs about 10 GiB through Loro's SnapshotAt export); #16 fixes that, and with it the binary stays under 2 GiB.

New tests in apps/craft-cloud-agent/tests/live_teardown.rs run a real agent pair and a raw WebSocket app. Each makes 60 one-letter changes, closes the document, sends the WebSocket close and (first test) waits for the agent's close before it drops the socket, as an app does that reads its answers; the other machine must then open the file with all 60. The second test makes the agent's writes fail meanwhile, starting with an over-size update whose refusal can't be sent, so the agent has to keep reading.

A third test has two apps wait for the import of one file. The first one (asked first) stops taking messages while its connection stays open and its read-on window is set to two minutes, so only the hub's own detection can help; the second must be asked within 20 s. With the new hub.rs half removed it times out (the second app was never asked for the import).

I ran the first two with the fix removed: both fail on main (the other machine never gets all changes). With only the hub.rs half removed both still fail, and with only the docs.rs half removed the second one fails. The failures are timeouts of the wait for the text, not sleeps that pass by chance.

Latest check of this head

Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and CRAFT_CLOUD_REQUIRE_BACKENDS=1 under a 16 GiB memory limit at 18e6f36: cargo fmt --all -- --check and cargo clippy --workspace --all-targets --locked -- -D warnings pass; cargo test --workspace --locked --no-fail-fast: 957 passed, 0 failed, 2 ignored. The run ends with exit code 101 without a failed test: the agent's live_documents test binary aborts (memory allocation of 786432 bytes failed) on branches based on main. It reproduces alone and is fixed by #16. craft-cloud-desktop --lib also crashes on Windows (STATUS_ACCESS_VIOLATION) on branches without the render lock of #35; pristine main crashes the same way.

Not in this PR

  • The app side (crates/app): its teardown still shuts the socket down right after the close. Changes the agent never received can't be recovered by the agent alone; the app keeps the acknowledgement-and-retry job.

Linux note

The first version of the first test dropped the socket with the agent's answers unread and failed on Linux CI (ubuntu-24.04), where it passed on Windows and macOS. The cause is the reset described above, not a bug the agent can repair. The tests now end like an app that reads its answers, and a fourth test quits through craft-cloud-app, whose drop now waits for the agent's close. In WSL (Ubuntu, 24 cores): 10 of 10 runs of live_teardown, and 60 of 60 with the test pinned to two cores and ten busy loops on the others.

… them

An app that sent its updates, closed the document and shut its socket down at once could lose its last changes.
Two things dropped them:

- The hub's `tell` removed the whole connection (its handles) as soon as the connection's output was closed or
  full. Changes the connection had queued before were then refused for having no handle. `tell` now only stops
  sending to that connection (its sender is dropped, so the socket ends); the handles stay until the connection's
  own `DocsCall::Gone`, which comes in order after everything it queued. The docs socket now always sends `Gone`
  when it ends.
- The docs socket stopped at the first failed write to the app, without reading the frames the app had sent
  before. A write that fails because the app is gone now only mutes the connection: it reads and forwards what the
  app sent, until its close or EOF, for at most five seconds. An app too slow to take a message is still dropped
  at once, as before.

Tests drive a real agent: an app sends several updates, closes the document and drops the socket; and one whose
writes fail (a hidden test hook) while its updates are still arriving. Every update must reach another machine.
…taking messages

When the hub found that a docs connection no longer takes messages, an import it had asked of that connection stayed
asked: the other opens waiting for the same document were only asked once the connection's `Gone` came, and with
the read-on window after a failed write that can be seconds. `tell` now asks the next open waiting that can still
be told, at once, and `ask_import` skips connections that take no messages. A connection that stops reading drops
its receiver of the hub's output, so the hub finds out right away.

The final WebSocket close of a connection that was dropped for being too slow is sent under the send timeout. A
receive error ends the connection, muted or not; that limit is noted in a comment.

A test has two apps wait for the import of one file while the first stops taking messages (hooks for one app's
writes and for the read-on window).
…nection is dropped

A connection closed while data of the agent's is still unread is reset instead of closed, and a reset can take
with it frames the agent has received but not read yet, which on Linux were the last changes before the app quit.
Dropping the connection now sends the close, waits up to a second for the agent's, and only then shuts the
stream down.
…one test goes through the app library

The test that dropped its socket with the agent's answers unread failed on Linux CI now and then: the socket was
reset, and the reset took frames with it that the agent had received but not read. That is a property of TCP, not
something the agent can repair, so the test now ends the way an app that reads its answers does. A new test quits
through craft-cloud-app, whose drop does the same.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant