Packets still lost due to buffer overflow at connection to new Region

[AndrewMeadows]

Environment

All official SL viewers, all servers.

Description

When the Viewer connects to a new Region (either by login/teleport/vew-into-neighbor-rgion) the simulator running the Region will usually send a burst of UDP messages to the Viewer -- so many that it exceed the Viewer's ability to process them and the spike of network traffic can overflow the inbound buffer. In PR #3565 there was an attempt to reduce packet loss in this case, however those measures were insufficient: we still lose packets , which continue to be a source (not the only one) of non-visible content and cloudy avatars.

We should increase the size of the buffer and try very hard to NEVER drop packets. The moment packets are queued they should be ACKed (which will reduce resends) and harvested for outbound ACKs (which will prevent unnecessary outbound resends).

Reproduction steps

Test on the 26.3 viewer and 26.4 viewer.

1. Login to a very busy region (lots of avatar) with lots of content.
2. Notice packet loss in the stats.
3. Look in logs for messages about lost packets.
4. Verify the spike of packet loss is less in the 26.4 viewer than in the 26.3 viewer.

---

This repo is using Opire - what does it mean? 👇

💵 Everyone can add rewards for this issue commenting /reward 100 (replace 100 with the amount).
🕵️‍♂️ If someone starts working on this issue to earn the rewards, they can comment /try to let everyone know!
🙌 And when they open the PR, they can comment /claim #5562 either in the PR description or in a PR's comment.

🪙 Also, everyone can tip any user commenting /tip 20 @AndrewMeadows (replace 20 with the amount, and @AndrewMeadows with the user to tip).

📖 If you want to learn more, check out our documentation.

[Dan-Linden]

Maximum bandwidth = 3000 kbps
Testing at https://maps.secondlife.com/secondlife/London%20City/79/177/23
Log in, wait 2 minutes, check Packets Lost in Help > About Second Life

26.3.0.31203661088
Windows

1. 1 lost packets
2. 1
3. 3

Mac

1. 4
2. 3
3. 6

26.4.0.35914774337
Windows

1. 0
2. 15
3. 1

Mac

1. 6
2. 0
3. 1

Testing is inconclusive.

[Dan-Linden]

The test results are inconclusive.
Is there some other method to test?

[AndrewMeadows]

Here is the recipe:

(1) Login to a good vantage point in London City where you can see lots of objects and avatars.
(2) and open the stats window: CTRL+ALT+1
(3) Expand the "UDP Data Received" and "Packet Loss" stats. They won't remain expanded when you log back in with the stats window automatically open, but those are the two stats you want to pay attention to.
(5) Open Debug Settings
(6) Search for "PrecacheDelay" and set it to 0. This won't eliminate the loading progress bar, but it will minimize the time that the loading window curtains off the scene of the world.
(4) With the stats window open... logout. We want the stats window to show up immediately upon subsequent login.
(5) Log back in to the same location
(6) As soon as the stats are visible expand "UDP Data Received" and "Packet Loss" stats.
(7) Pay attention to the general qualities of those plots, in particular notice the peak average Packet Loss.
(8) Compare the quality of those stats upon arrival between 26.3 and 26.4. Do you notice a difference between the peaks? Yes, you will lose packets in both cases, but the loss for 26.4 should be noticeably lower than for 26.3 (e.g. maybe only 8% instead of 30%).

Here is a snapshot of my arrival using 26.3:

And here is the same location using 26.4: