[hgichon]

Problem

The verbs provider hard-codes retry_count = 15 (clamped to 7 by rdma_cm) for connected QPs in vrb_msg_ep_connect() and the XRC path.

When a peer process restarts, for example after a node power cycle, a verbs;ofi_rxm connection that pointed at its previous incarnation still looks connected until it is used. The first send on it is never acknowledged. It fails with transport retry counter exceeded only after (retry_count + 1) x the local ACK timeout. Then rxm shuts the connection down and reconnects. With mlx5 that is about 8 x 0.54 s ≈ 4.3 s per stale connection.

A long-lived client that talks to several restarted endpoints pays this once per endpoint. In a DAOS cluster with a long-lived dfuse client, the first operation after a server node came back took 4.3 s, 8.6 s or 17 s, always a multiple of about 4.3 s.

Raising the ACK timeout makes this longer. Lowering it (RDMA_OPTION_ID_ACK_TIMEOUT below 14) had no effect here, which suggests ~0.54 s is the HCA's minimum. So the retry count is the only knob that shortens the delay.

Change

• Add FI_VERBS_RETRY_COUNT (0 - 7). The default is 7, the value rdma_cm already used, so nothing changes unless it is set.
• Use it in vrb_msg_ep_connect() and vrb_process_ini_conn() (XRC).
• Document it in fi_verbs.7.md.

The value goes into the CM REQ, so the side that opens the connection sets it for both QPs.

Testing

Setup: 2 nodes with mlx5 VFs, verbs;ofi_rxm with FI_OFI_RXM_USE_SRX=1, a DAOS 2.8 cluster on libfabric 1.22.0 plus this change (backported; the code is the same on main). Power off one server node for 60 s, boot it, wait until it rejoins, then time the first operation on a long-lived dfuse mount on the other node.

FI_VERBS_RETRY_COUNT	First operation	Per stale connection
unset (7)	17.3 / 8.6 / 17.2 / 8.6 / 17.1 / 4.3+4.3 s	~4.3 s
1	3.08 / 0.05 / 1.03+2.16 / 1.98 / 3.03 / 0.02 / 3.12 s	~1.05 s

• With 1, a 3-minute load (4 parallel writers and readers, about 8 GB) showed no retry-exceeded errors.
• I have not tested a lossy or flapping link. A lower value fails a connection after fewer consecutive losses, which is why the default stays unchanged.
• Builds on main (80c2fd7) with -Wall -Werror (verbs and rxm enabled). An out-of-range value (e.g. 9) is rejected with Invalid value of retry_count and the provider is not loaded.

Related: #13120 fixes a different problem in the same scenario (the rank pair that never reconnects). This one recovers by itself but costs ~4.3 s per stale connection.

[j-xiong]

There is no need to have a default value -1. Just set the default value to 7 and get rid of the vrb_conn_retry_count() function.

[hgichon]

Agreed. rdma_cm already clamps the old 15 to 7, so a default of 7 keeps the current behaviour. Done in b8f2806: the default is now 7, the valid range is 0 - 7, vrb_conn_retry_count() is gone, and both connect paths use vrb_gl_data.retry_count directly. Rebuilt with -Wall -Werror (verbs and rxm); fi_info -e shows (default: 7), and 8 or -1 are still rejected with Invalid value of retry_count. I can squash it into the first commit if you prefer.

[j-xiong]

Please squash these commits and do a force push. There is no auto squash at merge here, so a commit fixing another commit in the same PR should be squashed.