feat(gateway): explain compute driver discovery decisions

[elezar]

User Story

As an OpenShell operator, I want the gateway to explain how it selected or rejected local compute drivers, so that I can diagnose startup failures without reproducing them under a debugger.

Problem Statement

Gateway auto-detection probes local driver candidates, such as Docker Unix sockets, but silently treats probe failures as unavailable. When a candidate socket exists but Snap confinement denies access, the gateway eventually reports only that no suitable driver was found and systemd may restart it. Operators cannot tell which candidates were examined, whether a driver was selected, or whether a candidate failed because of a missing socket, an access denial, a timeout, or an unexpected API response.

Impact / Why This Matters

Package and confinement failures are difficult to distinguish from an absent runtime. The current workaround is to manually inspect socket paths, Snap interface connections, and journal output, then infer the cause. That is insufficient for CI and user installations because the decisive probe result is not recorded.

Proposed Design

When gateway driver auto-detection runs, expose diagnostics at debug level that identify each candidate driver and socket probe outcome without logging credentials or other sensitive data. Startup errors should provide a concise summary of the attempted drivers and why no usable driver was selected. A successful selection should state the selected driver and endpoint category in diagnostics.

Acceptance Criteria

• With debug logging enabled, each local compute-driver candidate records whether it was selected, skipped, or rejected.
• Docker socket diagnostics distinguish a missing/non-socket path, connection denial, timeout, and an incompatible or unsuccessful API response.
• Normal startup errors summarize the failed discovery decision without exposing secrets or request contents.
• A successful auto-detection records the selected driver and does not expose sensitive endpoint data.
• Coverage verifies representative successful, missing, and permission-denied probe outcomes.

Alternatives Considered

• Relying only on systemd or Snap logs: these do not report the gateway discovery decision or every candidate it attempted.
• Treating socket existence as availability: this would hide confinement failures and select unusable drivers.
• Requiring an explicitly configured driver: useful as a workaround, but does not make default installations diagnosable.

Agent Investigation

Docker discovery currently probes each candidate with a Unix-socket HTTP ping and returns only a boolean result. Failed metadata, connect, write, read, timeout, and response-validation paths are intentionally collapsed into unavailable. This behavior is shared by local API socket discovery and leaves no per-candidate diagnostic trail.

Related: #2869 (comment)

[elezar]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: Medium
Confidence: High — the behavior is localized to gateway auto-detection, local socket probing, startup diagnostics, and operator docs.
Minimal scope: Add gateway diagnostics without changing driver selection order, config schema, runtime construction, or remote-driver behavior.
LSM compatibility: Low impact. SELinux/AppArmor/Snap denials should surface as permission-denied probe outcomes; tests should avoid /proc or fork/exec assumptions.

Summary

Gateway auto-detection should explain why each compiled local compute driver was selected, skipped, or rejected. Replace the ComputeDriverRegistration.detect: fn() -> bool hook with a structured detection result, let Docker keep detailed local socket probe outcomes internally, and emit redaction-safe gateway service logs plus concise startup failure summaries for operators.

The related PR #2869 review comment reinforces the Snap/Docker interface case: a socket can exist while the gateway is denied access, so permission-denied must be distinguishable from Docker absent.

Scope

• crates/openshell-core/src/local_api_socket.rs: add a diagnostic probe API with explicit outcomes for missing/non-socket path, permission denied, connect/write/read failure, timeout, empty response, and rejected response. Keep the current first_responsive_socket() wrapper for existing callers.
• crates/openshell-driver-docker/src/lib.rs: add Docker socket discovery diagnostics and bounded endpoint categories for DOCKER_HOST, /var/run/docker.sock, Docker Desktop, and XDG_RUNTIME_DIR. Expose redacted detail summaries for gateway logs and keep raw socket paths out of log-oriented APIs.
• crates/openshell-server/src/lib.rs: change the registered detector hook to return ComputeDriverDetectionResult instead of bool. Retain the registry's ordered decision trail so gateway startup can report selected, skipped, and rejected drivers.
• crates/openshell-server/src/cli.rs: emit stored gateway auto-detection debug logs after tracing is installed. Failed detection still returns a normal startup error with the redacted gateway auto-detection summary.
• crates/openshell-gateway/src/lib.rs: wire Kubernetes, Podman, and Docker detector functions directly through the structured result API. Docker contributes richer socket details; Kubernetes and Podman provide driver-level summaries.
• docs/reference/sandbox-compute-drivers.mdx and architecture/compute-runtimes.md: document the gateway/operator diagnostics behavior and clarify that this is gateway service logging, not sandbox telemetry.

Implementation Steps

1. Add diagnostic structs/enums in local_api_socket and route the existing boolean helper through them.
2. Implement Docker-specific candidate categorization and response classification, including unsuccessful status vs incompatible API response.
3. Replace the compute-driver detection hook's boolean return with a structured detection result.
4. Store and emit redaction-safe gateway auto-detection decisions after tracing is installed.
5. Format the no-suitable-driver startup error with driver names and reason buckets, not secrets, endpoint values, or probe request contents.
6. Update focused docs for operators using OPENSHELL_LOG_LEVEL=debug.

Test Plan

• Unit tests: Add openshell-core tests for successful socket probe, missing/non-socket path, timeout/read failure, permission-denied outcome mapping, and rejected response classification. Add openshell-driver-docker tests for Docker response acceptance, Podman/incompatible response rejection, unsuccessful status rejection, and candidate category ordering/redaction. Add openshell-server tests for selected, skipped, rejected, and no-driver summary behavior.
• Integration tests: N/A; no gRPC protocol or runtime lifecycle behavior changes.
• E2E tests: N/A for implementation. Final verification should run mise run pre-commit and targeted package-level Rust tests for openshell-core, openshell-driver-docker, and openshell-server; run broader suites if the local toolchain permits.

Risks & Open Questions

• Debug logs for failed selection happen before the current full tracing setup unless startup preparation is split. The minimal plan uses the normal startup error for failed discovery and post-setup debug logs for successful detection.
• Permission-denied socket tests can be platform-sensitive or bypassed by privileged test users. Keep them Unix-only and assert the error classification separately.
• Do not broaden Docker probing to TCP DOCKER_HOST; unsupported/non-Unix values should be reported as skipped without logging the raw value.

Documentation Impact

• Update docs/reference/sandbox-compute-drivers.mdx.
• Update architecture/compute-runtimes.md because the registered driver detection contract changes from boolean availability to structured gateway diagnostics.
• No docs/reference/gateway-config.mdx update expected because no gateway TOML fields, driver config keys, defaults, or Helm rendering change.

---

Revision 2 — align implementation with gateway/operator diagnostics and structured detector results
Revision 1 — initial plan