[Bug]: CN pipeline backend churn exhausts ephemeral ports under sustained distributed queries

[XuPeng-SH]

Is there an existing issue for the same bug?

• I have checked the existing issues.

This is a current reproducible recurrence of the connection-exhaustion class previously tracked by #9545. That issue was closed in 2023 and there is no matching open issue.

Branch Name

main

Commit ID

3728f38

Other Environment Information

• Deployment: make dev-up, two CNs in separate Docker containers/processes, one TN, one log service, and one proxy
• Client shape: 16 persistent MySQL sessions; the failure is not caused by repeatedly opening frontend connections
• CN ephemeral port range: 32768-60999
• Pipeline client default MaxSenderNumber per remote host: 100000
• OS: Linux x86_64

Actual Behavior

A sustained distributed shuffle query causes CN-to-CN morpc connections to grow and churn until new connections fail with EADDRNOTAVAIL:

ERROR 20505 (HY000): cannot connect to backend: dial tcp4 172.23.0.5:18100: connect: cannot assign requested address

Downstream remote pipelines can then fail with:

ERROR 20101 (HY000): internal error: remote receiver not ready within 30s, remote CN may have failed

Both CN processes remain alive.

Using 16 persistent MySQL sessions, each scheduled to execute the query 25 times, 320 statements completed with the correct result before multiple sessions aborted around statements 19-24 due to CN-to-CN connection failures. This excludes frontend short-connection churn as the cause.

A socket snapshot taken after the failures showed approximately 5.8K established connections per CN plus 4K-5K TIME_WAIT sockets. At peak, new connections to the same remote CN could no longer obtain a local ephemeral port.

The application-level limit does not protect the real resource: PipelineConfig defaults MaxSenderNumber to 100000 backends per remote host, while the observed Linux ephemeral range contains only about 28K ports.

Expected Behavior

Sustained distributed queries should not exhaust CN ephemeral ports. CN-to-CN pipeline concurrency should be bounded by a realistic socket budget and apply context-aware backpressure before connection creation fails.

The solution must preserve cancellation and timeout behavior and must not introduce a deadlock when pipelines have cyclic dependencies.

Steps to Reproduce

1. Build and start a two-CN development cluster:

   make dev-build
   make dev-up
   

2. Create the distributed test data:

   create database d1;
   use d1;
   create table t1(c1 int not null, c2 int not null, c3 int not null) cluster by c1;
   create table t2(c1 int not null, c2 int not null, c3 int not null) cluster by c1;
   create table t3(c1 int not null, c2 int not null) cluster by c1;
   insert into t1 select *,*,* from generate_series(5000000) g;
   insert into t2 select *,*,* from generate_series(4000000) g;
   insert into t3 select *,* from generate_series(1,1000000) g;
   

3. Open 16 persistent MySQL sessions through the proxy. In every session, execute the following statement 25 times:

   use d1;
   select count(*)
   from t1, t2, t3
   where t1.c1 = t2.c1
     and t1.c2 = t3.c2
     and t2.c2 < 900000
     and t3.c1 < 500000;
   

   The expected result of every statement is 499999.

4. Monitor CN socket state and logs:

   docker exec mo-cn1 cat /proc/sys/net/ipv4/ip_local_port_range
   docker exec mo-cn1 ss -s
   docker exec mo-cn2 ss -s
   

5. Observe repeated CN-to-CN dial failures with cannot assign requested address, followed by query failures or remote receiver timeouts.

Additional information

The current lifecycle is:

1. PipelineClient.NewStream calls morpc NewStream with lock=true, so a pipeline stream exclusively locks one backend/TCP connection.
2. messageSenderOnClient.close calls stream.Close(true), which closes the entire backend connection when the stream ends.
3. Closed backends are removed from the morpc pool by inactive GC, but the kernel keeps the closed TCP tuples in TIME_WAIT.
4. New distributed pipelines continue creating backends up to the configured 100000-per-host application limit, which is higher than the available ephemeral-port budget.

This is not a classic permanent connection leak: destruction is reached. It is a lifecycle and capacity-control bug in which the connection creation/close rate exceeds kernel reclamation and there is no admission control at the true resource boundary.

Historical issue #9545 documents the same failure signature and notes that blindly reusing a single TCP connection is unsafe because cyclic pipeline dependencies can fill TCP buffers and deadlock. Therefore simply changing Close(true) to Close(false) is not a safe fix. The repair needs bounded, context-aware admission/backpressure and a deliberate reuse strategy that preserves pipeline progress.

This issue is independent of #25687/#25688. The empty-dispatch change restores protocol correctness; it does not create the underlying backend limit mismatch or TCP lifecycle behavior.