Add supervised remote Mongo recovery

[witbrock]

What changed

• Add a general, secret-free Mongo SSH-tunnel fallback contract that derives loopback Mongo URIs in memory, validates replica-set identity, and selects only the writable forwarded member.
• Make fallback choice failure-sensitive: direct-host Atlas first for pure DNS/SRV failures; supervised SSH first for TLS/TCP and typed health/retry transport failures; stale local Mongo remains disabled by default and explicit-only.
• Add bounded sticky recovery, primary-route reprobes, replica-election reselection, typed fallback/transport diagnostics, and Atlas-aware cost-guardrail classification.
• Add a hardened macOS LaunchAgent installer with strict host-key checking, isolated SSH config, transactional rollback, passive PID/listener evidence, and conservative install/uninstall target validation.
• Document setup, status predicates, functional read-back, and safe uninstall ordering.

Why

Atlas connectivity from a workstation can fail because of the local ISP or route even while an independently connected DGX relay remains healthy. The previous local fallback could silently select an out-of-date database. Von now retains Atlas as primary, uses simpler direct-host recovery for DNS-only failures, and can automatically use an authenticated remote SSH path for broader transport failures without putting Mongo credentials in launchd state.

Validation

• 265 targeted backend/launcher tests passed.
• Focused Ruff, Python compile, shell syntax, and git diff --check passed.
• Live LaunchAgent plist validated; one managed SSH PID owned all three loopback listeners.
• All three forwarded Atlas members returned the expected replica set with exactly one writable primary.
• Forced direct-route failure and forced active-client ping failure both selected ssh_tunnel, completed a Mongo ping, and completed an exact canonical concept read.
• Terminating the managed SSH process caused launchd to replace it, restore all three listeners, and complete a post-restart Mongo hello through the tunnel.

uv.lock was unrelated and is not included.

[Copilot]

Pull request overview

This PR adds a supervised, secret-free MongoDB recovery path that can fall back to a remote SSH tunnel (loopback forwards) when direct Atlas connectivity fails, while keeping local Mongo fallback disabled by default to avoid silent downgrades to stale workstation data.

Changes:

• Introduces an SSH-tunnel fallback contract in the Mongo client (loopback-only endpoints, writable-primary selection, optional replica set pinning, bounded “sticky” recovery, and route-degradation rotation).
• Hardens observability/redaction so SSH-tunnel usage is classified as upstream Atlas (sanitised URIs only) across health, diagnostics, settings, and internal MCP guardrails.
• Adds a macOS LaunchAgent installer/status/uninstall tool for a launchd-supervised SSH tunnel, plus tests and documentation.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utilities/get_safe_env.py	Adds allowlisted keys for new Mongo fallback env vars and uses shared Mongo URI sanitisation for display-safe output.
tests/test_run_sh_start_behaviour.py	Adds launcher-path tests for SSH-tunnel status logging and output labelling.
tests/test_install_macos_mongo_ssh_tunnel.py	Adds comprehensive tests for the macOS launchd tunnel installer/status/uninstall behaviour and safety checks.
tests/test_get_safe_env.py	Updates expectations for Mongo URI sanitisation and adds coverage for standard mongodb:// URI redaction.
tests/backend/test_mongo_uri_redaction.py	Extends backend redaction/location classification tests for SSH-tunnel “upstream Atlas” semantics and fallback_kind propagation.
tests/backend/test_mongo_ssh_tunnel_fallback.py	Adds unit tests for SSH-tunnel fallback selection, loopback endpoint parsing, replica set validation, and election reselection behaviour.
tests/backend/test_mongo_recovery_behaviour.py	Adds tests ensuring transport-degraded routes rotate to SSH tunnel and that transient retry marks routes degraded appropriately.
tests/backend/test_mongo_observability_service.py	Adds MCP guardrails test ensuring SSH-tunnel Atlas usage is classified as remote/Atlas with secrets removed.
tests/backend/test_mongo_dns_fallback_recovery.py	Ensures DNS fallback recovery tests disable machine-specific SSH tunnel fallback configuration.
src/backend/services/mongo_startup_config.py	Changes strict-startup validation default so local Mongo fallback is treated as disabled unless explicitly enabled.
src/backend/services/model_registry_service.py	Includes fallback-kind context (sanitised) in the model registry authority fingerprint via safe connection-location building.
src/backend/server/utils_flask.py	Enhances health/diagnostics/db_status payloads with fallback_kind and upstream-safe location classification (no secret leakage).
src/backend/server/routes/settings_routes.py	Snapshots effective route identity after ping/recovery and returns connection_location + fallback_kind; adjusts guardrails to use logical sanitised URI.
src/backend/integrations/internal_mcp/catalogue.py	Updates MCP mongo_cost_guardrails_report to use safe connection-location building (supports SSH-tunnel upstream Atlas classification).
src/backend/db/transient_errors.py	Replaces inline transient classification with shared helpers and passes route-degradation signals into reconnect attempts.
src/backend/db/mongo_uri_redaction.py	Extends safe connection-location payload to distinguish endpoint vs upstream classification and expose logical sanitised URI + transport.
src/backend/db/mongo_error_classification.py	Adds shared helpers for transient vs transport Mongo error classification used by retry and reconnection logic.
src/backend/db/mongo_client.py	Implements SSH-tunnel fallback endpoints parsing, derived loopback Mongo URI creation, fallback target selection/ordering, sticky recovery, and route-degraded rotation.
src/backend/db/connection_manager.py	Threads transport-degradation signals through health summaries and reconnection attempts, including route rotation hints.
setup_py.sh	Updates generated .env default to disable local Mongo fallback (MONGO_ALLOW_LOCAL_FALLBACK=0).
setup_py.ps1	Updates generated .env default to disable local Mongo fallback (MONGO_ALLOW_LOCAL_FALLBACK=0).
scripts/install_macos_mongo_ssh_tunnel.py	Adds a macOS launchd LaunchAgent installer/status/uninstall tool for supervised SSH forwarding (secret-free).
run.sh	Logs SSH tunnel listener presence and refines Mongo status labelling to distinguish ssh_tunnel/dns/local fallback kinds.
README.md	Updates setup guidance to reflect default local-fallback disabled and documents the macOS SSH tunnel workflow link.
docs/engineering/environment_minimums.md	Documents SSH tunnel fallback env vars, macOS installer usage, status semantics, and safe uninstall ordering.
.env.template	Documents new fallback env vars, sets local fallback disabled by default, and adds sticky window configuration variable.