feat: spread shared supervisors evenly across gateway replicas #5434

User Story

I run OpenShell on Kubernetes with several gateway replicas and a shared supervisor pool. I need the supervisors spread evenly across the gateways so that no single gateway carries most of the supervisors and their sandbox sessions.

Problem Statement

A shared supervisor attaches to whichever gateway replica the Kubernetes Service picks, and since #5051 its sandboxes' sessions follow it there. Nothing balances supervisors across gateways: two supervisors can land on the same gateway while another serves none, and a rolling restart can concentrate them further. Operators also have no metric for supervisors per gateway or sandboxes per supervisor.

Impact / Why This Matters

One gateway can end up serving most of the pool's supervisors and sandbox sessions while others sit idle. That gateway becomes the bottleneck and the blast radius when it restarts. Today the only fix is to restart supervisors until they happen to land evenly.

Proposed Design

• A supervisor that connects through the Service (new, restarted, or after losing its gateway) is placed on the existing gateway ring by its member ID and goes to the first gateway clockwise that serves fewer than an even share, ceil(live supervisors / live gateways). If every gateway is full, the ring owner is used. The cap is soft.
• The gateway redirects the supervisor to that replica's peer endpoint. The supervisor dials it at once and asks to stay. It also asks to stay when redialing the gateway that last accepted it (token refresh, short blips) and on the Service fallback after a failed redirect.
• A stopping gateway hands each attached supervisor off the same way, using the ring without itself.
• Supervisors never move on token refresh, short interruptions, or when a gateway joins. There is no background rebalancer.
• Two gauges: supervisors served per gateway, and the most sandboxes on one supervisor that gateway serves.

Acceptance Criteria

• With two gateways and two supervisors, restarting the supervisors leaves one supervisor on each gateway.
• After a gateway rolling restart, the supervisors are again one per gateway.
• A supervisor reconnecting after a token refresh stays on its gateway.
• Gateways export supervisors-per-gateway and busiest-supervisor sandbox gauges.

Alternatives Considered

• Least-loaded placement with the ring as a tie-breaker: more moving parts for the same result.
• A background rebalancer: moves supervisors, and with them their sandbox sessions, while nothing is wrong.
• Envoy or Service-level balancing: balances connections, not supervisors, and does not survive reconnects.