Repository navigation
Agent: an app that quits right after sending changes does not lose them (after #16) - #42
Open
Skyfuller1303 wants to merge 4 commits into
Open
Skyfuller1303 wants to merge 4 commits into
Skyfuller1303 wants to merge 4 commits into
Conversation
… them An app that sent its updates, closed the document and shut its socket down at once could lose its last changes. Two things dropped them: - The hub's `tell` removed the whole connection (its handles) as soon as the connection's output was closed or full. Changes the connection had queued before were then refused for having no handle. `tell` now only stops sending to that connection (its sender is dropped, so the socket ends); the handles stay until the connection's own `DocsCall::Gone`, which comes in order after everything it queued. The docs socket now always sends `Gone` when it ends. - The docs socket stopped at the first failed write to the app, without reading the frames the app had sent before. A write that fails because the app is gone now only mutes the connection: it reads and forwards what the app sent, until its close or EOF, for at most five seconds. An app too slow to take a message is still dropped at once, as before. Tests drive a real agent: an app sends several updates, closes the document and drops the socket; and one whose writes fail (a hidden test hook) while its updates are still arriving. Every update must reach another machine.
…taking messages When the hub found that a docs connection no longer takes messages, an import it had asked of that connection stayed asked: the other opens waiting for the same document were only asked once the connection's `Gone` came, and with the read-on window after a failed write that can be seconds. `tell` now asks the next open waiting that can still be told, at once, and `ask_import` skips connections that take no messages. A connection that stops reading drops its receiver of the hub's output, so the hub finds out right away. The final WebSocket close of a connection that was dropped for being too slow is sent under the send timeout. A receive error ends the connection, muted or not; that limit is noted in a comment. A test has two apps wait for the import of one file while the first stops taking messages (hooks for one app's writes and for the read-on window).
…nection is dropped A connection closed while data of the agent's is still unread is reset instead of closed, and a reset can take with it frames the agent has received but not read yet, which on Linux were the last changes before the app quit. Dropping the connection now sends the close, waits up to a second for the agent's, and only then shuts the stream down.
…one test goes through the app library The test that dropped its socket with the agent's answers unread failed on Linux CI now and then: the socket was reset, and the reset took frames with it that the agent had received but not read. That is a property of TCP, not something the agent can repair, so the test now ends the way an app that reads its answers does. A new test quits through craft-cloud-app, whose drop does the same.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge after #16. The diff of this PR alone: main...agent-keep-last-changes-on-quit. This branch is based on
main; it waits for #16 only because the agent'slive_documentstest binary runs out of memory onmainuntil #16 is in.An app that sends its last changes and quits right after can lose them: the agent takes in the first ones and refuses or never reads the rest. This is on
maintoday. It shows when a live document's last keystrokes before closing the app don't reach the other machine.Cause
Two things in the agent drop changes that the app had already sent:
closeandGone, then exits and drops the receiver of the hub's output. The hub journals update A and sends A'sackto that dead receiver.tell()then callsgone()at once, which removes the connection's handles. Update B, still queued, finds no handle and is refused instead of journalled.gone()ran before theGonethat sits behind B in the queue.Updates the agent has acknowledged are durable already (the journal is synced before the ack). Only the ones still in flight at teardown were lost.
What
hub.rs,tell()no longer removes the connection. When the output is closed or full, the connection's sender is dropped: nothing more is sent to it, and its socket ends once it has taken what was queued. Its handles stay. They are cleaned up by the connection's ownDocsCall::Gone, which is in the same FIFO behind every change the connection queued. A slow client is disconnected the same way, so that order holds there too.docs.rs,serve():Gonewhen the connection ends, last (a dropped output used to skip it, which would now leak the connection);READ_ON). The usual frame limits apply meanwhile.Unsent::ClosedorUnsent::Slow). Reading on from a socket whose write timed out half-way is not safe: my first version did, and the existingan_app_that_misbehaves_is_refused_or_dropped_and_the_rest_go_ontest then ran out of memory.tell()finds a connection dead, every document whose import was asked of that connection is asked of the next open waiting for it that can still be told (ask_importskips connections that take no messages), instead of waiting for the dead connection'sGone. Opens of a dead connection stay waiting until itsGone, which gives the join up if nobody else waits.crates/app),Documents::drop: sends the close, waits up to a second for the agent's close (the reader thread ends on it), and only then shuts the stream down. Before, it shut the stream down at once. A connection closed while the agent's answers are still unread is reset, and on Linux a reset drops frames the agent has received but not read yet, which were the last changes before the app quit (found on Linux CI: the first version of the main test dropped a raw socket at that moment and failed now and then; reproduced in WSL under CPU load, 2 failures in 40 runs, and the library test with the old drop once in 150).#[doc(hidden)]indocs:FAIL_WRITES(every write to an app fails),FAIL_WRITES_OF(the same for the connections of one named app) andREAD_ON_MS(how long a muted connection is read on), likerecord_delayandannounce_delayinRunOptions.Checks
On Windows, through a memory-limited runner:
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --locked -- -D warningscargo test -p craft-cloud-agent -p craft-cloud-app --locked --no-fail-fastcargo +1.95 check --workspace --all-targets --locked(MSRV)Merge after #16. Without it, the
live_documentstest binary runs out of a 16 GiB limit onmain(the misbehaving-app test alone needs about 10 GiB through Loro'sSnapshotAtexport); #16 fixes that, and with it the binary stays under 2 GiB.New tests in
apps/craft-cloud-agent/tests/live_teardown.rsrun a real agent pair and a raw WebSocket app. Each makes 60 one-letter changes, closes the document, sends the WebSocket close and (first test) waits for the agent's close before it drops the socket, as an app does that reads its answers; the other machine must then open the file with all 60. The second test makes the agent's writes fail meanwhile, starting with an over-size update whose refusal can't be sent, so the agent has to keep reading.A third test has two apps wait for the import of one file. The first one (asked first) stops taking messages while its connection stays open and its read-on window is set to two minutes, so only the hub's own detection can help; the second must be asked within 20 s. With the new
hub.rshalf removed it times out (the second app was never asked for the import).I ran the first two with the fix removed: both fail on
main(the other machine never gets all changes). With only thehub.rshalf removed both still fail, and with only thedocs.rshalf removed the second one fails. The failures are timeouts of the wait for the text, not sleeps that pass by chance.Latest check of this head
Checked on Windows with Postgres 17, SeaweedFS 4.48 (S3) and
CRAFT_CLOUD_REQUIRE_BACKENDS=1under a 16 GiB memory limit at18e6f36:cargo fmt --all -- --checkandcargo clippy --workspace --all-targets --locked -- -D warningspass;cargo test --workspace --locked --no-fail-fast: 957 passed, 0 failed, 2 ignored. The run ends with exit code 101 without a failed test: the agent'slive_documentstest binary aborts (memory allocation of 786432 bytes failed) on branches based onmain. It reproduces alone and is fixed by #16.craft-cloud-desktop --libalso crashes on Windows (STATUS_ACCESS_VIOLATION) on branches without the render lock of #35; pristinemaincrashes the same way.Not in this PR
crates/app): its teardown still shuts the socket down right after the close. Changes the agent never received can't be recovered by the agent alone; the app keeps the acknowledgement-and-retry job.Linux note
The first version of the first test dropped the socket with the agent's answers unread and failed on Linux CI (ubuntu-24.04), where it passed on Windows and macOS. The cause is the reset described above, not a bug the agent can repair. The tests now end like an app that reads its answers, and a fourth test quits through
craft-cloud-app, whose drop now waits for the agent's close. In WSL (Ubuntu, 24 cores): 10 of 10 runs oflive_teardown, and 60 of 60 with the test pinned to two cores and ten busy loops on the others.