diff --git a/.agents/skills/sync-agent-infra/SKILL.md b/.agents/skills/sync-agent-infra/SKILL.md index 184ce71d5b..d3c0190e85 100644 --- a/.agents/skills/sync-agent-infra/SKILL.md +++ b/.agents/skills/sync-agent-infra/SKILL.md @@ -45,7 +45,7 @@ Use this map when product behavior, commands, or development workflows change. I | Sandbox policy schema, presets, or enforcement behavior | `generate-sandbox-policy`, `openshell-cli` | | Supervisor middleware policy, registrations, runtime, or failure behavior | `generate-sandbox-policy`, `openshell-cli`, `debug-openshell-cluster` | | Gateway deployment, Helm, runtime drivers, or health checks | `debug-openshell-cluster`, `helm-dev-environment` | -| Inference routing, providers, or `inference.local` behavior | `debug-inference`, `openshell-cli` | +| Inference providers, native model endpoints, or migration from `inference.local` | `debug-inference`, `openshell-cli`, `generate-sandbox-policy` | | TUI architecture, navigation, data fetching, or UX | `tui-development` | | Release artifacts or post-publish smoke coverage | `test-release-canary` | | GitHub Actions workflows, required checks, or CI diagnostics | `watch-github-actions`; also `test-release-canary` for release smoke coverage | diff --git a/AGENTS.md b/AGENTS.md index 9a07ca0b7a..533510737b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,8 +39,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-conformance-cli/` | Conformance CLI | Distributable `list` and `run` entrypoint for gateway conformance | | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary | | `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing | -| `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints | -| `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing | +| `crates/openshell-policy/` | Policy engine | Filesystem, network, and process constraints | | `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage | | `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary | | `crates/openshell-ocsf/` | OCSF logging | OCSF v1.8.0 event types, builders, shorthand/JSONL formatters, tracing layers | @@ -57,7 +56,6 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-driver-db-credstore/` | Database credential driver | In-process `CredentialDriver` backend for gateway database credential storage | | `crates/openshell-driver-kubernetes/` | Kubernetes compute driver | In-process `ComputeDriver` backend for K8s sandbox pods | | `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers | -| `crates/openshell-driver-mxc/` | MXC compute driver | Windows in-process `ComputeDriver` backend for MXC sandbox execution | | `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers | | `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) | | `crates/openshell-driver-mxc/` | Microsoft MXC compute driver | In-process Windows AppContainer and isolation-session compute backend | @@ -65,7 +63,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization | | `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution | | `crates/openshell-supervisor-middleware-builtins/` | Built-in middleware | First-party in-process middleware implementations | -| `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and inference routing | +| `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and provider credential injection | | `crates/openshell-supervisor-process/` | Process supervisor | Process lifecycle, namespace, and bypass monitoring | | `crates/openshell-vfio/` | VFIO support | PCI and GPU passthrough preparation and lifecycle | | `python/openshell/` | Python SDK | Python bindings and CLI packaging | @@ -116,7 +114,7 @@ Use an OCSF builder + `ocsf_emit!()` for events that represent **observable sand - SSH authentication (accepted, denied, nonce replay) - Process lifecycle (start, exit, timeout, signal failure) - Security findings (unsafe policy, unavailable controls, replay attacks) -- Configuration changes (policy load/reload, TLS setup, inference routes, settings) +- Configuration changes (policy load/reload, TLS setup, provider attachments, settings) - Application lifecycle (supervisor start, SSH server ready) ### When to use plain tracing @@ -138,7 +136,7 @@ Use `info!()`, `debug!()`, `warn!()` for **internal operational plumbing** that | SSH sessions | `SshActivityBuilder` | Authentication, channel operations | | Process start/stop | `ProcessActivityBuilder` | Entrypoint lifecycle, signal failures | | Security alerts | `DetectionFindingBuilder` | Nonce replay, bypass detection, unsafe policy. Dual-emit with the domain event. | -| Policy/config changes | `ConfigStateChangeBuilder` | Policy load, Landlock apply, TLS setup, inference routes, settings | +| Policy/config changes | `ConfigStateChangeBuilder` | Policy load, Landlock apply, TLS setup, provider attachments, settings | | Supervisor lifecycle | `AppLifecycleBuilder` | Sandbox start, SSH server ready/failed | ### Severity guidelines diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ab1fa2e7a5..14398b2e4a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -80,7 +80,7 @@ Public skills live in `skills/` and work without an OpenShell source checkout. I | --- | --- | | `openshell-cli` | CLI usage, sandbox lifecycle, provider management, and BYOC workflows | | `debug-openshell-cluster` | Diagnose gateway deployment and health issues | -| `debug-inference` | Diagnose managed, system, local, and direct external inference issues | +| `debug-inference` | Diagnose attached-provider inference, native endpoints, and migration from `inference.local` | | `generate-sandbox-policy` | Generate YAML sandbox policies from requirements or API documentation | Public skills use `openshell --help` for installed command syntax and published OpenShell documentation for product concepts and configuration. They must not depend on repository-relative source or documentation files. diff --git a/Cargo.lock b/Cargo.lock index edc940ea68..c91de40f0d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4246,24 +4246,6 @@ dependencies = [ "url", ] -[[package]] -name = "openshell-router" -version = "0.0.0" -dependencies = [ - "bytes", - "openshell-core", - "reqwest 0.12.28", - "serde", - "serde_json", - "serde_yml", - "tempfile", - "thiserror 2.0.18", - "tokio", - "tracing", - "uuid", - "wiremock", -] - [[package]] name = "openshell-sandbox" version = "0.0.0" @@ -4367,7 +4349,6 @@ dependencies = [ "openshell-policy", "openshell-prover", "openshell-providers", - "openshell-router", "openshell-supervisor-middleware", "openshell-supervisor-middleware-builtins", "opentelemetry", @@ -4474,7 +4455,6 @@ dependencies = [ "openshell-core", "openshell-ocsf", "openshell-policy", - "openshell-router", "openshell-supervisor-middleware", "openshell-supervisor-middleware-builtins", "prost-types", diff --git a/README.md b/README.md index a7ecf4179e..e57faafd07 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ bash examples/sandbox-policy-quickstart/demo.sh OpenShell isolates each sandbox in its own container with policy-enforced egress routing. A lightweight gateway coordinates sandbox lifecycle, and every outbound connection is intercepted by the policy engine, which does one of three things: - **Allows** — the destination and binary match a policy block. -- **Routes for inference** — strips caller credentials, injects backend credentials, and forwards to the managed model. +- **Binds credentials to endpoints** — injects provider credentials only after policy admits a request to a profile-authorized endpoint. - **Denies** — blocks the request and logs it. | Component | Role | @@ -116,7 +116,7 @@ OpenShell isolates each sandbox in its own container with policy-enforced egress | **Gateway** | Control-plane API that coordinates sandbox lifecycle and acts as the auth boundary. | | **Sandbox** | Isolated runtime with container supervision and policy-enforced egress routing. | | **Policy Engine** | Enforces filesystem, network, and process constraints from application layer down to kernel. | -| **Privacy Router** | Privacy-aware LLM routing that keeps sensitive context on sandbox compute. | +| **Provider Access** | Profile-defined endpoints, binary policy, and endpoint-bound credential injection for model APIs and other services. | OpenShell runs a gateway control plane that manages sandbox lifecycle through a configured compute driver. Supported compute platforms include Docker, Podman, MicroVM, and Kubernetes. @@ -129,14 +129,16 @@ OpenShell applies defense in depth across four policy domains: | Filesystem | Prevents reads/writes outside allowed paths. | Locked at sandbox creation. | | Network | Blocks unauthorized outbound connections. | Hot-reloadable at runtime. | | Process | Blocks privilege escalation and dangerous syscalls. | Locked at sandbox creation. | -| Inference | Reroutes model API calls to controlled backends. | Hot-reloadable at runtime. | +| Providers | Grants endpoint-bound credentials and network access. | Hot-reloadable at runtime. | -Policies are declarative YAML files. Static sections (filesystem, process) are locked at creation; dynamic sections (network, inference) can be hot-reloaded on a running sandbox with `openshell policy set`. +Policies are declarative YAML files. Static sections (filesystem, process) are locked at creation; network policy and provider attachments can be updated on a running sandbox. ## Providers Agents need credentials — API keys, tokens, service accounts. OpenShell manages these as **providers**: named credential bundles that are injected into sandboxes at creation. The CLI auto-discovers credentials for recognized agents (Claude, Codex, OpenCode, Copilot) from your shell environment, or you can create providers explicitly with `openshell provider create`. Credentials never leak into the sandbox filesystem; they are injected as environment variables at runtime. +Inference access uses the same provider workflow. Attach an inference-capable provider to a sandbox, call the provider's native endpoint, and select the model in the client. Provider profiles contribute the endpoint policy and bind credential placeholders to the authorized destination. + ## GPU Support (Experimental) > **Experimental** — GPU passthrough works on supported hosts but is under active development. Expect rough edges and breaking changes. @@ -159,8 +161,8 @@ Docker-backed GPU sandboxes auto-select CDI when available and otherwise fall ba | [OpenCode](https://opencode.ai/) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Works out of the box. Provider uses `OPENAI_API_KEY` or `OPENROUTER_API_KEY`. | | [Codex](https://developers.openai.com/codex) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Works out of the box. Provider uses `OPENAI_API_KEY`. | | [GitHub Copilot CLI](https://docs.github.com/en/copilot/github-copilot-in-the-cli) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Works out of the box. Provider uses `GITHUB_TOKEN` or `COPILOT_GITHUB_TOKEN`. | -| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run OpenClaw more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | -| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run Hermes Agent more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | +| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run OpenClaw more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | +| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run Hermes Agent more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | | [Ollama](https://ollama.com/) | [Community](https://github.com/NVIDIA/OpenShell-Community) | Launch with `openshell sandbox create --from ollama`. | | [Pi](https://pi.dev/) | [Community](https://github.com/NVIDIA/OpenShell-Community) | Launch with `openshell sandbox create --from pi`. | @@ -172,9 +174,9 @@ Docker-backed GPU sandboxes auto-select CDI when available and otherwise fall ba | `openshell sandbox connect [name]` | SSH into a running sandbox. | | `openshell sandbox list` | List all sandboxes. | | `openshell provider create --type [type] --from-existing` | Create a credential provider from env vars. | +| `openshell sandbox provider attach ` | Attach a provider to a running sandbox. | | `openshell policy set --policy file.yaml` | Apply or update a policy on a running sandbox. | | `openshell policy get ` | Show the active policy. | -| `openshell inference set --provider

--model ` | Configure the `inference.local` endpoint. | | `openshell logs [name] --tail` | Stream sandbox logs. | | `openshell term` | Launch the real-time terminal UI for debugging. | diff --git a/TESTING.md b/TESTING.md index 199fa35416..538c575ff0 100644 --- a/TESTING.md +++ b/TESTING.md @@ -136,8 +136,6 @@ def test_multiply(sandbox): |---|---|---| | `sandbox_client` | session | gRPC client connected to the active gateway | | `sandbox` | function | Factory returning a `Sandbox` context manager | -| `inference_client` | session | Client for managing inference routes | -| `mock_inference_route` | session | Creates a mock OpenAI-protocol route for tests | ### Rust CLI E2E (`e2e/rust/`) diff --git a/architecture/README.md b/architecture/README.md index d48a0d53f9..706db67b1f 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -7,7 +7,7 @@ the **Supervisor**. The CLI, SDK, and TUI provide user-facing access. The gateway is the authenticated control plane: it owns API access, durable state, policy and -settings delivery, provider and inference configuration, and relay +settings delivery, provider configuration and attachments, and relay coordination. The supervisor runs inside every sandbox workload and is the local security boundary. It launches the agent as a restricted child process and enforces policy where process identity, filesystem access, network egress, and @@ -50,7 +50,6 @@ flowchart TB subgraph DP["Sandbox Data Plane"] SUP["Supervisor"] PROXY["Policy proxy"] - ROUTER["Inference router"] POLICY["OPA policy engine"] AGENT["Restricted agent process"] end @@ -81,8 +80,7 @@ flowchart TB AGENT -->|"all ordinary egress"| PROXY PROXY -->|"evaluate"| POLICY PROXY -->|"allowed traffic"| EXT["External services"] - PROXY -->|"inference.local"| ROUTER - ROUTER -->|"managed inference"| MODEL["Inference backends"] + PROXY -->|"profile-authorized traffic"| MODEL["Model providers"] ``` ## Core Boundaries @@ -90,14 +88,13 @@ flowchart TB | Component | Boundary | |---|---| | CLI, SDK, TUI | User-facing management surfaces. They talk to the gateway and do not need to know which infrastructure drivers are active. | -| Gateway | Authenticated control plane, API server, durable state, policy and settings delivery, provider and inference config, supervisor session ownership, and relay coordination. | +| Gateway | Authenticated control plane, API server, durable state, policy and settings delivery, provider config, supervisor session ownership, and relay coordination. | | Compute subsystem | Sandbox lifecycle semantics: creation, deletion, watching, reconciliation, and state transitions. Platform provisioning details belong to the compute driver. | | Credentials subsystem | Logical provider and credential resolution. Secret storage and platform-native credential access belong to credentials drivers. | | Control-plane identity | Authentication and authorization for users, operators, and API clients. External identity verification belongs to identity drivers. | | Sandbox identity | Workload identity for supervisors and sandbox-to-sandbox authorization. Identity issuance or verification belongs to sandbox identity drivers. | | Supervisor | Sandbox-local security boundary. It prepares isolation, fetches config, injects credentials, runs relay endpoints, starts the proxy, and launches restricted agent processes. | -| Policy proxy | Mandatory egress path for agent traffic. It enforces destination, binary identity, SSRF, TLS/L7, credential injection, and inference interception rules. | -| Inference router | Sandbox-local forwarding for `https://inference.local` to configured model backends. | +| Policy proxy | Mandatory egress path for agent traffic. It enforces destination, binary identity, SSRF, TLS/L7, and endpoint-bound provider credential injection. | ## Integrating with the Ecosystem @@ -109,14 +106,14 @@ platforms that already provide them. The gateway owns OpenShell control-plane semantics: sandbox state, lifecycle ordering, policy and settings resolution, credential mapping, authorization, -inference configuration, and relay coordination. Drivers translate those -semantics into platform-native operations. They should stay thin, preserve +and relay coordination. Drivers translate those semantics into platform-native +operations. They should stay thin, preserve native behavior by default, and report platform lifecycle events back through the shared contracts. The supervisor owns OpenShell sandbox semantics. Filesystem policy, process -privilege reduction, network proxying, inference interception, credential -injection, security logging, and gateway relay behavior should remain +privilege reduction, network proxying, endpoint-bound credential injection, +security logging, and gateway relay behavior should remain consistent across runtimes. This keeps OpenShell usable in local single-player setups, Kubernetes @@ -128,7 +125,7 @@ ecosystem. The gateway and sandbox split control-plane authority from runtime enforcement. The gateway owns durable platform state: sandboxes, policy revisions, runtime -settings, provider records, inference configuration, session records, and +settings, provider records and profiles, session records, and authorization decisions. A sandbox owns the local execution boundary: process identity, filesystem access, network egress, credential injection, local logs, and the agent child process. @@ -150,8 +147,8 @@ sandbox configuration API for later revisions. The Windows MXC driver reports its own readiness and does not expose interactive connect or governed egress. The gateway delivers desired state; the sandbox applies it locally. Policy, -settings, credentials, and inference routes flow from the gateway to the -supervisor. The supervisor validates and applies what can change at runtime, +settings, provider attachments, and credential bindings flow from the gateway +to the supervisor. The supervisor validates and applies what can change at runtime, keeps last-known-good config when refresh fails, and leaves static isolation controls in place until the sandbox is recreated. @@ -170,7 +167,7 @@ that crate's `README.md`. | Document | Purpose | |---|---| | [Gateway](gateway.md) | Gateway control plane, auth, APIs, persistence, settings, and relay coordination. | -| [Sandbox](sandbox.md) | Sandbox supervisor, child process isolation, proxy, credentials, inference, connect, and logs. | +| [Sandbox](sandbox.md) | Sandbox supervisor, child process isolation, proxy, provider credentials, connect, and logs. | | [Sandbox Limits](sandbox-limits.md) | Sandbox supervisor and egress safety ceilings, ownership rules, current enforcement, and known gaps. | | [Security Policy](security-policy.md) | Policy model, enforcement layers, policy updates, policy advisor, and security logging. | | [Compute Runtimes](compute-runtimes.md) | Docker, Podman, Kubernetes, VM, sandbox images, and runtime-specific responsibilities. | diff --git a/architecture/gateway.md b/architecture/gateway.md index 769f57f6a0..4b0e70c730 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -2,17 +2,16 @@ The gateway is the OpenShell control plane. It exposes the API used by the CLI, SDK, and TUI; persists platform state; manages provider credentials and -inference configuration; and asks compute runtimes to create or delete sandbox -workloads. +attachments; and asks compute runtimes to create or delete sandbox workloads. ## Responsibilities - Authenticate clients and sandbox callbacks. - Serve gRPC APIs for sandbox lifecycle, provider management, policy updates, - settings, inference configuration, logs, watch streams, and relay forwarding. + settings, logs, watch streams, and relay forwarding. - Serve HTTP endpoints for health, WebSocket tunnels, and edge-auth flows. - Persist domain objects in SQLite or Postgres. -- Resolve provider credentials and inference bundles for sandbox supervisors. +- Resolve endpoint-bound provider environments for sandbox supervisors. - Coordinate supervisor relay sessions for connect, exec, file sync, and service forwarding. - Persist the canonical main-process instance ID and normalized exit code on @@ -62,7 +61,7 @@ authorization boundary. When the primary listener does not cover the address, the gateway adds a callback-only listener. Additional callback listeners accept only gRPC methods classified as sandbox-callable by the gateway's generated authorization metadata. They reject user and administrator APIs, health, -reflection, non-callback inference APIs, and HTTP routes before normal request +reflection, and HTTP routes before normal request authentication. The operator-configured primary listener retains the full multiplexed API surface. @@ -277,7 +276,6 @@ The gateway API is organized around platform objects and operational streams: | Sandbox lifecycle | Create, list, delete, watch, exec, SSH session bootstrap, ForwardTcp service forwarding. | | Providers | Store provider records, discover credentials, resolve runtime environment. | | Policy and settings | Get effective sandbox config, update sandbox policy, manage global settings. | -| Inference | Set gateway-level model/provider config and resolve sandbox route bundles. | | Observability | Push sandbox logs, stream sandbox status and logs to clients. | Domain objects use shared metadata: stable server-generated IDs, human-readable @@ -299,7 +297,7 @@ The storage schema is intentionally narrow: | Column | Purpose | |---|---| | `id` | Stable gateway-generated object ID and primary key. | -| `object_type` | Logical resource kind, such as `sandbox`, `provider`, `ssh_session`, `inference_route`, `sandbox_policy`, or `draft_policy_chunk`. | +| `object_type` | Logical resource kind, such as `sandbox`, `provider`, `provider_profile`, `ssh_session`, `sandbox_policy`, or `draft_policy_chunk`. | | `name` | Human-readable name, unique within an object type when present. | | `scope` | Optional owner or namespace for scoped/versioned records, such as a sandbox ID for policy revisions. | | `version` | Optional monotonically increasing version for scoped records. | @@ -346,9 +344,9 @@ not readable by other local users on shared hosts. The same restriction is reapplied to the `-wal` and `-shm` sidecars (created by SQLite's default WAL journal mode), which mirror the same sensitive contents. -Persisted state includes sandboxes, providers, provider credential refresh -state, SSH sessions, policy revisions, settings, inference configuration, and -deployment records, and reusable sandbox workload templates. Provider refresh +Persisted state includes sandboxes, providers, provider profiles, provider +credential refresh state, SSH sessions, policy revisions, settings, deployment +records, and reusable sandbox workload templates. Provider refresh state is stored as a separate object scoped to the provider instance through `objects.scope`. Its non-secret configuration remains inline, while refresh tokens, client secrets, private keys, and other secret source material are @@ -464,7 +462,6 @@ coverage: | Sandbox | `MustCreate` | `update_message_cas` | `list_messages` | | Provider | `MustCreate` | `update_message_cas` | `list_messages` | | ProviderProfile | `MustCreate` | `MatchResourceVersion` | `list_messages` | -| InferenceRoute | `MustCreate` | `update_message_cas` | `list_messages` | | SandboxPolicy | scoped versioning | scoped versioning | scoped query | | Settings | `Mutex`-guarded | `Mutex`-guarded | single-row | @@ -526,69 +523,25 @@ configuration, valid endpoint-bound static credentials from other attached providers, and the dynamic credential snapshot. Provider environment revisions include profile endpoint and binding changes. -## Inference Resolution +## Provider Environment Resolution -Cluster inference routes store only `provider_name`, `model_id`, and optional -timeout. The gateway resolves endpoint URLs, protocols, credentials, auth -style, and route-shaping metadata from the provider record when supervisors call -`GetInferenceBundle`. Supported provider types for cluster inference are -`openai`, `anthropic`, `nvidia`, `deepinfra`, and `google-vertex-ai`. +The gateway resolves only the providers attached to a sandbox. It combines each +provider instance with its profile, returns non-secret configuration, and marks +credentials with the profile's host, port, and path boundaries. The supervisor +uses those bindings when it replaces credential placeholders in policy-allowed +requests. -The bundle carries enough information for sandbox-local routers to construct -upstream URLs without re-deriving provider-specific routing logic. Each resolved -route may include: - -| Field | Meaning | -|---|---| -| `model_in_path` | When true, the model identifier is part of the upstream URL path, not only the request body. | -| `request_path_override` | Path override or suffix. With `model_in_path=false`, replaces the protocol-derived path; with `model_in_path=true`, appended after the model ID. | - -For standard providers these fields stay unset and the sandbox router uses default -protocol paths. Vertex AI is model-aware: the gateway constructs the base URL -from provider config (`VERTEX_AI_PROJECT_ID`, `VERTEX_AI_REGION`, optional -`VERTEX_AI_PUBLISHER`) and emits route-shaping metadata so the sandbox router -stays provider-agnostic. - -Host selection follows the configured region: - -| Region value | Vertex host | -|---|---| -| `global` | `aiplatform.googleapis.com` | -| `us` or `eu` | `aiplatform.{region}.rep.googleapis.com` | -| Any other (e.g. `us-central1`) | `{region}-aiplatform.googleapis.com` | - -Route shaping by publisher: - -- **Anthropic (Claude)** — `model_in_path=true`, base path under - `publishers/anthropic/models`, protocol `anthropic_messages` only. The gateway - resolves `request_path_override=:rawPredict`; the sandbox router keeps - `:rawPredict` for buffered requests and upgrades to `:streamRawPredict` only - for streaming proxy calls. -- **All other models** (Gemini, third-party, unknown) — OpenAI-compatible - `.../endpoints/openapi` base with `request_path_override=/chat/completions`; - protocol `openai_chat_completions`. - -Callers may supply `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` only for -non-Anthropic routes. Anthropic base URL overrides are rejected because they -cannot safely preserve model-path shaping and `anthropic_version` body -adaptation. Overrides still pin `request_path_override=/chat/completions` and -must use `https` with an official Vertex AI hostname (`aiplatform.googleapis.com`, -`aiplatform.{us,eu}.rep.googleapis.com`, or `{region}-aiplatform.googleapis.com`). - -Header passthrough is protocol-dependent. Vertex Claude rawPredict routes strip -client `anthropic-beta` headers; `anthropic-version` is not forwarded because -the sandbox router injects `anthropic_version` into the request body for Vertex -rawPredict. Non-Anthropic Vertex routes do not inherit Anthropic passthrough -headers. +Model selection, API protocol, request and response shapes, streaming behavior, +and endpoint URL construction remain responsibilities of the workload's native +client. The gateway does not parse or transform model API requests. For `google-vertex-ai` providers created with CLI `--from-gcloud-adc`, the CLI calls gateway `ConfigureProviderRefresh` with OAuth2 refresh material from gcloud ADC, then `RotateProviderCredential` to mint the first access token before reporting success. ADC-backed providers mint into `GOOGLE_VERTEX_AI_TOKEN`. A successful create therefore yields an immediately usable provider; failures roll -back the provider record. Service-account JSON and private keys are gateway-side -refresh bootstrap material only; sandbox runtime inference receives minted -access tokens, not raw service-account material. +back the provider record. Service-account JSON and private keys remain gateway-side +refresh bootstrap material; sandboxes receive minted access tokens instead. ## Supervisor Relay diff --git a/architecture/google-vertex-ai-provider.md b/architecture/google-vertex-ai-provider.md index e161471c9f..8779bd2879 100644 --- a/architecture/google-vertex-ai-provider.md +++ b/architecture/google-vertex-ai-provider.md @@ -1,544 +1,98 @@ -# Google Vertex AI Provider — Implementation Reference +# Google Vertex AI Provider -This document covers the full implementation of the `google-vertex-ai` provider in -OpenShell. It is the canonical reference for maintainers working on anything in the -Vertex AI request path, from CLI through gateway to sandbox. +The `google-vertex-ai` provider gives selected sandboxes direct access to +Google Vertex AI endpoints without exposing long-lived Google credentials. +The provider profile owns endpoint policy and credential metadata; the +sandbox attachment owns which workload receives access. ---- +## Boundaries -## 1. Overview - -OpenShell's `google-vertex-ai` provider routes `inference.local` traffic through -Google Cloud's Vertex AI platform. It differs from a direct Anthropic or OpenAI -integration in two ways that touch nearly every layer of the stack: - -1. **Authentication is OAuth2 bearer, not a static API key.** Vertex AI accepts - short-lived GCP access tokens (`ya29.*`) as `Authorization: Bearer` headers. The - gateway mints and rotates these tokens from one of two refresh sources: a GCP - service account key (JWT-bearer grant) or gcloud Application Default Credentials - (OAuth2 refresh-token grant). - -2. **The URL and wire format depend on the model family.** Anthropic Claude models - use Vertex AI's native rawPredict surface (`/publishers/anthropic/models/{model}:rawPredict`) - with the Anthropic Messages API body shape. Gemini and all other models use Vertex - AI's OpenAI-compatible Chat Completions surface - (`/v1beta1/.../endpoints/openapi/chat/completions`). The gateway selects the right - route at `openshell inference set` time based on the model name (or the explicit - `VERTEX_AI_PUBLISHER` config key). - -### Canonical provider type - -The canonical provider type string is `google-vertex-ai`. The following aliases are -accepted everywhere and normalized to the canonical string: - -| Input | Resolves to | -|---|---| -| `google-vertex-ai` | `google-vertex-ai` | -| `vertex` | `google-vertex-ai` | -| `vertex-ai` | `google-vertex-ai` | -| `google-vertex` | `google-vertex-ai` | -| `gcp-vertex` | `google-vertex-ai` | - -Alias resolution lives in `openshell_core::inference::normalize_inference_provider_type` -and is the single source of truth shared by `openshell-server`, `openshell-providers`, -and the CLI. - ---- - -## 2. Architecture — How the Pieces Fit Together - -``` -CLI (openshell provider create / openshell inference set) - │ - ├── read_gcloud_adc() reads ~/.config/gcloud/application_default_credentials.json - ├── CreateProviderRequest persists provider object (type, credentials, config) - └── ConfigureProviderRefreshRequest registers a refresh state record - -Gateway (openshell-server) - │ - ├── provider_refresh worker background loop that rotates access tokens - │ ├── mint_oauth2_refresh_token() for gcloud ADC flow - │ └── mint_google_service_account_jwt() for service account key flow - │ - ├── SetClusterInferenceRequest - │ └── resolve_vertex_ai_route() builds RouterResolvedRoute - │ ├── infer_vertex_publisher() model → publisher - │ └── vertex_location_and_host() region → Vertex API host - │ - └── GetInferenceBundleRequest (from sandbox on connect) - └── resolve_route_by_name() re-resolves live route+credentials - -Router (openshell-router) - │ - ├── proxy_with_candidates_streaming() - │ ├── build_provider_url() appends model/:rawPredict or /chat/completions - │ ├── sanitize_request_headers() strips auth, strips anthropic-beta for rawPredict - │ └── prepare_backend_request() - │ ├── bearer_auth(access_token) Authorization: Bearer ya29.* - │ ├── remove "model" from body (rawPredict only — model is in the URL) - │ └── inject "anthropic_version" (rawPredict only — must be in body, not header) - │ - └── proxy_to_backend() / proxy_to_backend_streaming() - -Sandbox (inference.local) - └── agent connects to inference.local → gateway proxy → Vertex AI -``` - -Key crates and their roles: - -| Crate | Role | -|---|---| -| `openshell-core` (`inference.rs`) | Canonical type aliases, profile constants, URL alias resolution | -| `openshell-providers` | Environment-based credential discovery for `--from-existing` | -| `openshell-server` (`inference.rs`) | Route resolution: maps provider + model → `RouterResolvedRoute` | -| `openshell-server` (`provider_refresh.rs`) | Credential refresh worker; mints access tokens | -| `openshell-router` (`backend.rs`) | Proxy: URL construction, header sanitization, body rewriting | -| `openshell-cli` (`run.rs`) | `provider create` and `--from-gcloud-adc` CLI flow | -| `providers/google-vertex-ai.yaml` | Provider type profile: credential keys, refresh strategy, endpoints | - ---- - -## 3. Credential Model — Two Flows - -Vertex AI accepts only short-lived GCP access tokens. The gateway never sends the raw -service account JSON or gcloud ADC material to Vertex AI. Both flows converge on the -same runtime secret: a `ya29.*` token stored under one of these credential keys, -searched in priority order: - -``` -GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN -VERTEX_AI_SERVICE_ACCOUNT_TOKEN -GOOGLE_VERTEX_AI_TOKEN -VERTEX_AI_TOKEN -``` - -These names are defined in `openshell_core::inference::VERTEX_AI_CREDENTIAL_KEY_NAMES` -and are shared between the profile, the CLI, and the route resolver. - -### 3a. Service Account Key Flow (production) - -``` -User: openshell provider create --type google-vertex-ai \ - --credential GOOGLE_SERVICE_ACCOUNT_KEY="$(cat key.json)" \ - --config VERTEX_AI_PROJECT_ID=my-project \ - --config VERTEX_AI_REGION=us-central1 - -User: openshell provider refresh configure vertex-prod \ - --credential-key GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN \ - --strategy google-service-account-jwt \ - --material client_email="sa@..." \ - --material private_key="..." \ - --secret-material-key private_key - -Gateway: mint_google_service_account_jwt() - 1. build JWT claims: iss=client_email, scope=cloud-platform, aud=token_url - 2. sign with RS256 using private_key - 3. POST assertion to https://oauth2.googleapis.com/token - 4. store access_token as GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN - 5. schedule next refresh 300 s before expiry (max 3600 s lifetime) -``` - -The raw `GOOGLE_SERVICE_ACCOUNT_KEY` is stored as bootstrap material for the refresh -worker. It is never exposed to sandboxes; the sandbox only ever sees the short-lived -access token. - -### 3b. gcloud Application Default Credentials Flow (local dev) - -``` -User: gcloud auth application-default login -User: openshell provider create --type google-vertex-ai \ - --from-gcloud-adc \ - --config VERTEX_AI_PROJECT_ID=my-project - -CLI: read_gcloud_adc() - checks GOOGLE_APPLICATION_CREDENTIALS → $CLOUDSDK_CONFIG/... → ~/.config/gcloud/adc.json - validates type == "authorized_user" (rejects service_account — different flow) - extracts client_id, client_secret, refresh_token - - configure_provider_refresh(strategy=oauth2_refresh_token, ...) - rotate_provider_credential(...) ← mints first token immediately - -Gateway: mint_oauth2_refresh_token() - POST to https://oauth2.googleapis.com/token - grant_type=refresh_token, client_id, client_secret, refresh_token - stores access_token as GOOGLE_VERTEX_AI_TOKEN - if response includes a new refresh_token, rotates it in state -``` - -The `--from-gcloud-adc` flag is rejected for any provider type other than -`google-vertex-ai`. The CLI validates and reads the ADC file before creating the -provider, so a missing or malformed ADC file results in a clean error with no orphaned -gateway state. - -### 3c. Refresh Worker - -`provider_refresh.rs` runs a background tokio task (`spawn_refresh_worker`) that -sweeps all `StoredProviderCredentialRefreshState` records on a configurable interval. -For each record where `next_refresh_at_ms <= now` or `status == "rotation_requested"`, -it calls `refresh_provider_credential`, which calls `mint_credential` and then -`apply_minted_credential`. The minted access token is written back to the provider's -`credentials` map under the configured `credential_key` via a CAS update. - -Key timing constants: - -- Default `refresh_before_seconds`: 300 (refresh 5 minutes before expiry) -- Default `max_lifetime_seconds`: 3600 (token lifetime cap) -- Error retry interval: 60 seconds - ---- - -## 4. Route Resolution — From Provider to RouterResolvedRoute - -When `openshell inference set --provider --model ` is called, the server -runs `resolve_vertex_ai_route` in `openshell-server/src/inference.rs`. This function -produces a `RouterResolvedRoute` that the router uses verbatim for every proxied -request. - -### 4a. Publisher Inference - -`infer_vertex_publisher(model_id)` maps model name prefixes to Vertex AI publishers: - -| Prefix | Publisher | Routing | -|---|---|---| -| `claude-*` | `anthropic` | Anthropic Messages API (rawPredict) | -| `gemini-*`, `text-bison-*`, `chat-bison-*` | `google` | OpenAI-compat Chat Completions | -| `llama-*` | `meta` | OpenAI-compat Chat Completions | -| `mistral-*`, `codestral-*` | `mistralai` | OpenAI-compat Chat Completions | -| `jamba-*` | `ai21` | OpenAI-compat Chat Completions | -| `deepseek-*` | `deepseek` | OpenAI-compat Chat Completions | -| (unrecognized) | `None` | OpenAI-compat Chat Completions | - -Only the `anthropic` result changes routing; all non-Anthropic publishers use the same -OpenAI-compatible Vertex surface. The `VERTEX_AI_PUBLISHER` config key overrides -inference: set it to `anthropic` to force rawPredict for a non-standard model name. - -### 4b. Host Resolution - -`vertex_location_and_host(region)` maps the `VERTEX_AI_REGION` config value to a -Vertex API host: - -| Region value | Host | +| Component | Responsibility | |---|---| -| `global` | `aiplatform.googleapis.com` | -| `us` | `aiplatform.us.rep.googleapis.com` | -| `eu` | `aiplatform.eu.rep.googleapis.com` | -| `us-central1`, `europe-west4`, etc. | `-aiplatform.googleapis.com` | - -Default region when not set: `us-central1`. - -### 4c. Endpoint and Protocol Selection - -**Anthropic (Claude) models:** - -``` -endpoint = https://{host}/v1/projects/{project}/locations/{location}/publishers/anthropic/models -protocol = ["anthropic_messages"] -model_in_path = true -request_path_override = ":rawPredict" -``` - -The model ID is NOT embedded in the endpoint URL. It is stored in `route.model` and -appended by `build_provider_url` at proxy time: `{endpoint}/{model}:rawPredict` for -buffered requests, `{endpoint}/{model}:streamRawPredict` for streaming. - -**Non-Anthropic models (Gemini, Llama, Mistral, etc.):** - -``` -endpoint = https://{host}/v1beta1/projects/{project}/locations/{location}/endpoints/openapi -protocol = ["openai_chat_completions"] -model_in_path = false -request_path_override = "/chat/completions" -``` - -**Base URL override (escape hatch, non-Anthropic only):** -When `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` is set: - -- `GOOGLE_VERTEX_AI_BASE_URL` takes priority over `VERTEX_AI_BASE_URL` -- Rejected with `InvalidArgument` for Anthropic models — Anthropic routes require - model-path shaping that a bare URL override cannot preserve safely -- Must be `https://`, no IP literals, no userinfo, port 443 only if explicit, must - target an official Vertex AI hostname (pattern validated by `is_allowed_vertex_override_host`) - -### 4d. Credential Lookup - -For Vertex AI specifically, `find_provider_api_key` uses `CredentialLookup::PreferredOnly`: -it searches only the four `VERTEX_AI_CREDENTIAL_KEY_NAMES` keys and returns `None` if -none match. This prevents the raw service account JSON stored under -`GOOGLE_SERVICE_ACCOUNT_KEY` from being mistakenly used as a bearer token, which would -produce a confusing auth failure from Vertex AI. - -### 4e. Model ID Validation - -Before any URL is constructed, `validate_vertex_model_id` rejects model IDs that -contain path separators (`/`, `\`), traversal segments (`..`), URL delimiters (`?`, -`#`, `%`), control characters, or surrounding whitespace. This is defense-in-depth -against injection into the URL path that appears in Anthropic rawPredict routes. - ---- - -## 5. Request Proxying — Backend Transformations - -The router (`openshell-router/src/backend.rs`) applies four Vertex-specific -transformations on every proxied request: - -### 5a. URL Construction - -`build_provider_url` is called with `model_in_path` and `request_path_override` from -the resolved route: - -- **Anthropic buffered:** `{endpoint}/{model_id}:rawPredict` -- **Anthropic streaming:** `{endpoint}/{model_id}:streamRawPredict` - (`:rawPredict` suffix is upgraded to `:streamRawPredict` when `stream_response=true`) -- **OpenAI-compat:** `{endpoint}/chat/completions` - -### 5b. Authentication - -All Vertex AI routes use `AuthHeader::Bearer`. The router injects -`Authorization: Bearer {access_token}` where `access_token` is the `ya29.*` token -read from the provider's credentials at route resolution time. - -### 5c. Header Sanitization — Stripping `anthropic-beta` - -For rawPredict routes (`is_vertex_anthropic_rawpredict_route`), `sanitize_request_headers` -strips the `anthropic-beta` header even though it is in the route's `passthrough_headers` -list. Vertex AI's rawPredict endpoint rejects requests that include `anthropic-beta` with -HTTP 400. Beta feature enablement for Vertex AI is controlled through Google Cloud -(Model Garden access), not HTTP headers. Claude Code always sends `anthropic-beta` flags; -stripping them here prevents spurious 400 errors. - -Direct Anthropic API routes (non-Vertex) still forward `anthropic-beta` unchanged. - -### 5d. Body Rewriting — `model` and `anthropic_version` - -For rawPredict routes, `prepare_backend_request` rewrites the JSON request body: - -- **Removes `"model"` field.** Vertex AI rawPredict encodes the model in the URL path. - Sending `"model"` in the body causes HTTP 400 "Extra inputs are not permitted". Claude - Code and other Anthropic SDK clients always include `"model"` in the body; the router - strips it unconditionally for rawPredict routes. - -- **Injects `"anthropic_version": "vertex-2023-10-16"`.** The standard Anthropic API - sends this as the `anthropic-version` request header. Vertex AI's rawPredict expects - it as a JSON body field instead. The router injects it only when the client has not - already sent it (`!obj.contains_key("anthropic_version")`). The constant - `VERTEX_ANTHROPIC_VERSION = "vertex-2023-10-16"` is the Google-published value. - -For OpenAI-compatible routes, the standard model rewrite applies: `"model"` in the body -is overwritten with `route.model` (the model ID configured at `openshell inference set` -time). - ---- - -## 6. Configuration Keys - -These keys are set at `openshell provider create` time via `--config KEY=VALUE` and -stored in the provider's `config` map. They are re-read on every bundle resolution -(i.e. on every sandbox connect), so changing them with `openshell provider update` takes -effect for new sandboxes without restarting the gateway. - -| Key | Required | Default | Description | -|---|---|---|---| -| `VERTEX_AI_PROJECT_ID` | Yes (unless base URL override set) | — | GCP project ID. Must be 6–30 chars, lowercase letters/digits/hyphens, no leading/trailing hyphen. | -| `VERTEX_AI_REGION` | No | `us-central1` | GCP region or `global`/`us`/`eu`. Determines the Vertex API host. | -| `GOOGLE_VERTEX_AI_BASE_URL` | No | — | Full base URL override for non-Anthropic routes. Takes priority over `VERTEX_AI_BASE_URL`. | -| `VERTEX_AI_BASE_URL` | No | — | Backward-compatible alias for `GOOGLE_VERTEX_AI_BASE_URL`. | -| `VERTEX_AI_PUBLISHER` | No | Inferred from model name | Set to `anthropic` to force rawPredict routing. | - -Config key constants are defined in `openshell_core::inference`: -`VERTEX_AI_PROJECT_ID_KEY`, `VERTEX_AI_REGION_KEY`, `VERTEX_AI_PUBLISHER_KEY`. - -The full list of config keys scanned during `--from-existing` discovery is -`VERTEX_AI_CONFIG_KEY_NAMES` in the same module. - ---- - -## 7. Credential Keys and the Provider Profile +| CLI | Discovers ADC or accepts service-account bootstrap material and creates the provider record. | +| Gateway | Stores refresh material through the credential driver and rotates short-lived access tokens. | +| Provider profile | Declares Vertex hosts, credential aliases, refresh constraints, and permitted binaries. | +| Sandbox supervisor | Delivers opaque credential placeholders and resolves them only for profile-authorized Vertex requests. | +| Workload | Selects the native Vertex endpoint, model, request format, streaming mode, and timeout. | -`providers/google-vertex-ai.yaml` is the authoritative provider type profile. It -defines: +The provider does not select a model or transform requests. Anthropic Claude +uses Vertex's publisher-model `rawPredict` or `streamRawPredict` paths. Gemini +and other models use their documented native or OpenAI-compatible Vertex API. -- **`service_account_key` (`GOOGLE_SERVICE_ACCOUNT_KEY`):** The raw service account JSON. - This is gateway-side bootstrap material, not a sandbox credential. Not injected into - sandboxes. `required: false` because some deployments use gcloud ADC instead. +## Credential Flows -- **`service_account_token` (`GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN`, - `VERTEX_AI_SERVICE_ACCOUNT_TOKEN`):** Short-lived access token minted from the service - account key via `google_service_account_jwt` refresh strategy. `auth_style: bearer`. - Refreshed 300 s before expiry; max lifetime 3600 s; scope - `https://www.googleapis.com/auth/cloud-platform`. +Vertex accepts short-lived Google OAuth2 access tokens. Both supported setup +flows converge on a rotating access token stored in the provider record. -- **`gcloud_adc_token` (`GOOGLE_VERTEX_AI_TOKEN`, `VERTEX_AI_TOKEN`):** Short-lived - access token minted from gcloud ADC via `oauth2_refresh_token` refresh strategy. - Same `auth_style: bearer`, same timing. +### Service account -The `discovery` section lists `[service_account_token, gcloud_adc_token]` as the two -credential sources the gateway will scan during `--from-existing`. +The operator creates the provider with service-account bootstrap material and +configures `google-service-account-jwt` refresh. The private key remains in the +gateway credential store. The gateway mints +`GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` and refreshes it before expiry. -The `endpoints` section enumerates all Vertex AI API hosts that sandbox network -policies permit through provider profile composition: +### gcloud ADC -- `*-aiplatform.googleapis.com:443` (regional endpoints) -- `aiplatform.googleapis.com:443` (global endpoint) -- `aiplatform.us.rep.googleapis.com:443` (US multi-region) -- `aiplatform.eu.rep.googleapis.com:443` (EU multi-region) +For local development, `--from-gcloud-adc` reads an authorized-user ADC file, +stores the refresh grant at the gateway, and mints +`GOOGLE_VERTEX_AI_TOKEN`. The ADC file and refresh token do not enter the +sandbox. ---- +## Runtime Data Flow -## 8. `openshell-providers` — Discovery Plugin +1. The operator attaches the provider to a sandbox. +2. The effective policy includes the profile's Vertex endpoint and binary + rules. +3. A newly launched workload receives the current token as an opaque + placeholder plus non-secret project and region configuration. +4. The workload calls the native Vertex endpoint with that placeholder in the + `Authorization: Bearer` header. +5. After policy and endpoint binding pass, the proxy substitutes the current + real access token and forwards the request. +6. Token refresh updates the resolver; the workload continues using the same + placeholder. -`openshell-providers` handles credential discovery for `--from-existing`. There is no -dedicated `vertex.rs` plugin file because Vertex AI token discovery is profile-driven: -the gateway reads the `providers/google-vertex-ai.yaml` profile and scans the -credential env vars listed there. +The raw `GOOGLE_SERVICE_ACCOUNT_KEY` credential is bootstrap-only and is never +part of sandbox runtime material. -Vertex AI config keys (`VERTEX_AI_PROJECT_ID`, `VERTEX_AI_REGION`, etc.) are not listed -in the profile's `discovery.credentials` section, so they are scanned separately in -`discover_existing_provider_data` in the CLI: +## Endpoint Boundary -```rust -if provider_type == VERTEX_AI_PROVIDER_TYPE { - for key in openshell_core::inference::VERTEX_AI_CONFIG_KEY_NAMES { - if let Ok(val) = std::env::var(key) { ... } - } -} -``` - -Provider type normalization for the `ProviderRegistry` (non-inference providers like -`claude-code`, `github`, `gitlab`) is handled by `normalize_provider_type` in -`openshell-providers/src/lib.rs`, which delegates Vertex AI aliases to -`normalize_inference_provider_type` in `openshell-core`. - ---- - -## 9. Inference Routing in the Sandbox - -When a sandbox agent connects to `https://inference.local`, the sandbox fetches the -inference bundle from the gateway (`GetInferenceBundleRequest`). The bundle contains one -or more `ResolvedRoute` proto messages built by `resolve_route_by_name`. For a Vertex AI -route the bundle contains: - -``` -ResolvedRoute { - name: "inference.local", - base_url: "https://us-central1-aiplatform.googleapis.com/v1/projects/.../publishers/anthropic/models", - model_id: "claude-sonnet-4-20250514", - api_key: "ya29.", - protocols: ["anthropic_messages"], - provider_type: "google-vertex-ai", - model_in_path: true, - request_path_override: ":rawPredict", -} -``` +The built-in profile permits official Vertex hosts: -The sandbox proxy uses this bundle to configure the local `inference.local` route. The -bundle is re-fetched on reconnect, which picks up rotated access tokens automatically -without sandbox restart. +- `-aiplatform.googleapis.com` +- `aiplatform.googleapis.com` +- `aiplatform.us.rep.googleapis.com` +- `aiplatform.eu.rep.googleapis.com` -The gateway does NOT expose GCP credentials (project ID, region, service account key) -in the bundle. Sandboxes see only the short-lived access token. +The workload constructs the documented project, location, publisher, and model +path. The proxy does not infer the publisher or rewrite the body. ---- +For Claude on Vertex, a request uses this form: -## 10. Sandbox Usage Pattern - -Agents inside sandboxes connect to Vertex AI through `inference.local`. The correct -setup differs by model family: - -**Claude (Anthropic Messages API):** - -```sh -ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare +```text +https://-aiplatform.googleapis.com/v1/projects//locations//publishers/anthropic/models/:rawPredict ``` -The `ANTHROPIC_API_KEY` value is stripped by the gateway and replaced with the real -GCP token. `--bare` skips Claude Code's OAuth flow. Do NOT set -`CLAUDE_CODE_USE_VERTEX=1` inside a sandbox — that makes Claude Code connect directly -to Vertex AI and attempt GCP ADC discovery, which fails in the sandbox environment. - -**Gemini / other models (OpenAI-compat):** -Point the SDK's base URL at `https://inference.local/v1` and use any non-empty value -as the API key. - -**Common sandbox policy denials to expect:** - -- `metadata.google.internal:80` — resolves to `169.254.169.254` (GCE metadata service). - Always blocked by the proxy unconditionally. -- `downloads.claude.ai:443` — Claude Code update checking. Block or approve per policy. -- `storage.googleapis.com:443` — GCS access. Optional; approve if the agent needs it. - ---- - -## 11. Key Files Reference - -| File | Purpose | -|---|---| -| `providers/google-vertex-ai.yaml` | Provider type profile: credential keys, refresh strategy params, allowed endpoints | -| `docs/providers/google-vertex-ai.mdx` | User-facing documentation | -| `crates/openshell-core/src/inference.rs` | Canonical provider type aliases, `InferenceProviderProfile`, `VERTEX_AI_*` constants, auth header logic | -| `crates/openshell-server/src/inference.rs` | Route resolution: `resolve_vertex_ai_route`, `infer_vertex_publisher`, `vertex_location_and_host`, model ID validation | -| `crates/openshell-server/src/provider_refresh.rs` | Refresh worker: `mint_google_service_account_jwt`, `mint_oauth2_refresh_token`, `apply_minted_credential` | -| `crates/openshell-router/src/backend.rs` | Proxy engine: `build_provider_url`, `sanitize_request_headers`, `prepare_backend_request` (model strip + anthropic_version inject) | -| `crates/openshell-router/src/config.rs` | `ResolvedRoute` struct with `model_in_path`, `request_path_override` fields | -| `crates/openshell-cli/src/run.rs` | `provider_create`, `read_gcloud_adc`, `rollback_provider_create_after_vertex_adc_failure` | -| `crates/openshell-providers/src/lib.rs` | `ProviderRegistry`, `normalize_provider_type`, `discover_existing_provider_data` | - ---- - -## 12. Maintenance Notes - -### Adding a new Vertex AI region - -No code changes are needed for standard regional endpoints (`-aiplatform.googleapis.com`). -`vertex_location_and_host` constructs the host from the region string dynamically. For -new special-case multi-region endpoints (analogous to `us` and `eu`), update the match -arm in `vertex_location_and_host` in `openshell-server/src/inference.rs`. - -Add the new host pattern to the `endpoints` list in `providers/google-vertex-ai.yaml` -so it is included in sandbox network policy injection. - -### Adding a new model family publisher - -Add a prefix match arm to `infer_vertex_publisher` in `openshell-server/src/inference.rs`. -Unless the new publisher uses a different wire format than the OpenAI-compatible -endpoint, no other changes are needed — all non-Anthropic publishers currently route -to the same `endpoints/openapi/chat/completions` surface. - -If the new publisher requires a separate rawPredict-style surface (like Anthropic), -update `resolve_vertex_ai_route` to add a new branch for that publisher, and add the -corresponding body transformation logic in `prepare_backend_request` in -`openshell-router/src/backend.rs`. - -### Updating `VERTEX_ANTHROPIC_VERSION` - -The constant `VERTEX_ANTHROPIC_VERSION = "vertex-2023-10-16"` in -`openshell-router/src/backend.rs` is the version string Google requires in the body of -rawPredict requests. Update it when Google publishes a new required version for the -Vertex AI Anthropic API. Check the -[Google Cloud Claude documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude) -for the current required value. - -### Rotating the GCP OAuth2 token endpoint URL - -`google_token_url` in `provider_refresh.rs` defaults to `https://oauth2.googleapis.com/token` -when `state.token_url` is empty. The `providers/google-vertex-ai.yaml` profile sets this -in the `refresh.token_url` field. If Google changes the token endpoint, update both: - -- `refresh.token_url` in `providers/google-vertex-ai.yaml` -- The default in `google_token_url` in `provider_refresh.rs` - -### Changing the `cloud-platform` OAuth scope +The request body includes the Vertex Anthropic API version expected by Google. +Streaming uses the corresponding native streaming endpoint. -The scope `https://www.googleapis.com/auth/cloud-platform` is set in the -`refresh.scopes` list in `providers/google-vertex-ai.yaml`. The gateway uses it when -constructing JWT claims and OAuth2 refresh requests. If a narrower scope becomes -available (e.g. a Vertex AI-specific scope), update the profile's `scopes` list. +## Invariants -### Adding a new Vertex AI body invariant +- Refresh bootstrap material remains gateway-only. +- Sandboxes receive placeholders, never real access tokens. +- A token resolves only at endpoints covered by the attached provider profile. +- Detach and expiry revoke placeholder resolution. +- Project IDs, regions, publishers, and model IDs are non-secret workload + configuration. +- Provider attachment does not grant access when a gateway global policy + override suppresses provider-derived policy. +- Provider environment keys and dynamic credential bindings remain + unambiguous across all providers attached to one sandbox. -Body transformations for rawPredict live in `prepare_backend_request` in -`openshell-router/src/backend.rs`, gated by `is_vertex_anthropic_rawpredict_route`. -When adding a new transformation: +## Operational Notes -1. Add the transformation in the `needs_vertex_anthropic_version` branch. -2. Add a wiremock-based integration test directly below the existing tests in that file. -3. Ensure the new transformation does not apply to standard Anthropic or OpenAI routes. +Attach or detach the provider with the normal sandbox provider lifecycle. A +running sandbox observes policy and resolver changes, but a process must be +launched after attachment to receive new environment variables. Direct native +requests provide the end-to-end verification path; provider creation does not +probe a model endpoint or validate a model ID. diff --git a/architecture/sandbox-limits.md b/architecture/sandbox-limits.md index 9635bc1c30..9045ddc359 100644 --- a/architecture/sandbox-limits.md +++ b/architecture/sandbox-limits.md @@ -121,21 +121,15 @@ middleware. A passed binary logical message still advances the active middleware session sequence and emits coverage telemetry, so a later text RPC can contain a valid sequence gap. -## Inference and Upstream Proxying +## Network and Upstream Proxying | Path | Current bound | Terminal behavior | |---|---:|---| -| `inference.local` request parse buffer | 10 MiB | Return `413` for an oversized request. | -| Chunked inference request | 10 MiB and 4,096 chunks | Reject an invalid or over-limit request. | -| Streaming inference response | 32 MiB and 120 s chunk idle | Truncate the stream and attempt a safe SSE error. | | Corporate proxy CONNECT response headers | 8 KiB | Fail the tunnel. | | Corporate proxy CONNECT handshake | 30 s total | Fail the tunnel; validated-address attempts share the aggregate budget. | | Token-grant HTTP request | 30 s request and connect | Fail credential resolution. | | Response-derived token cache TTL | 5 min default; 1 h response cap; 30 s expiry margin | A positive profile `cache_ttl_seconds` override replaces the response-derived calculation. | -Streaming response byte limits are integrity-relevant. Protocols whose clients -require one complete buffered object do not use the truncating SSE path. - ## Sandbox-Local Surfaces | Surface | Current bound | Scope and behavior | diff --git a/architecture/sandbox.md b/architecture/sandbox.md index 055ef7e4a3..0bb7c42b93 100644 --- a/architecture/sandbox.md +++ b/architecture/sandbox.md @@ -28,7 +28,7 @@ only when the set is already empty; any other outcome fails the spawn. 2. The supervisor loads policy and runtime settings from local files or the gateway, depending on mode. 3. It prepares filesystem access, process restrictions, network namespace - routing, trust stores, provider credential resolution, and inference routes. + routing, trust stores, and provider credential resolution. 4. It launches the persisted canonical main-process argv and retains its PTY or pipes in the main-session multiplexer. 5. It starts the policy proxy and local SSH server. @@ -45,12 +45,12 @@ OpenShell uses overlapping controls rather than a single sandbox primitive: | Process policy | The child process runs as a non-root user with reduced privileges. | | Seccomp | Blocks dangerous syscalls, including raw socket paths that bypass the proxy. | | Network namespace | Forces ordinary agent egress through the local CONNECT proxy. | -| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. | +| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and endpoint-bound credential injection. | The supervisor may enrich baseline filesystem allowances for runtime-required paths, such as proxy support files or GPU device paths when a GPU is present. -## Network and Inference +## Network and Provider Access See [Sandbox Limits](sandbox-limits.md) for the current numeric safety ceilings, their ownership, terminal behavior, and known gaps. @@ -214,17 +214,11 @@ security logs. See [Supervisor Middleware](../docs/extensibility/supervisor-middleware.mdx) for configuration and protocol details. -`https://inference.local` is special. It bypasses OPA network policy and is -handled by the inference interception path: - -1. The proxy terminates the local TLS connection with the sandbox CA. -2. It detects known OpenAI, Anthropic, and compatible inference request shapes. -3. It strips caller-supplied credentials and disallowed headers. -4. It forwards through `openshell-router` using the route bundle fetched from - the gateway. - -External inference endpoints that do not use `inference.local` are treated like -ordinary network traffic and must be allowed by policy. +Inference providers use the same egress path as other external services. An +attached provider profile contributes endpoint and binary policy. The proxy +then resolves the provider's credential placeholder only when both policy and +the profile's endpoint binding authorize the native request. Model selection, +request shape, headers, streaming, and timeouts remain client concerns. In proxy-required networks, the supervisor chains upstream TLS tunnels through a corporate forward proxy with HTTP CONNECT instead of connecting directly, diff --git a/architecture/security-policy.md b/architecture/security-policy.md index 9203ba3178..e4b30c9292 100644 --- a/architecture/security-policy.md +++ b/architecture/security-policy.md @@ -15,7 +15,7 @@ For the field-by-field YAML reference, use | Filesystem | Landlock restricts read-only and read-write paths. | | Process | The supervisor launches the agent as an unprivileged user with reduced capabilities. | | Network | The proxy evaluates destination, port, calling binary, and optional L7 rules. | -| Inference | `inference.local` is configured through gateway inference settings, not OPA network policy. | +| Provider access | Attached provider profiles contribute endpoint and binary rules; credentials remain bound to profile-authorized endpoints. | | Runtime settings | Typed settings are delivered with policy and can be global or sandbox scoped. | Filesystem and process policy are startup-time controls. Network policy is diff --git a/crates/openshell-cli/src/main.rs b/crates/openshell-cli/src/main.rs index befac54759..6195e84d93 100644 --- a/crates/openshell-cli/src/main.rs +++ b/crates/openshell-cli/src/main.rs @@ -264,7 +264,6 @@ const HELP_TEMPLATE: &str = "\ gateway: Manage gateways status: Show gateway status and information whoami: Show the authenticated user identity - inference: Manage inference configuration doctor: Diagnose gateway issues \x1b[1mADDITIONAL COMMANDS\x1b[0m @@ -401,12 +400,6 @@ const GATEWAY_EXAMPLES: &str = "\x1b[1mALIAS\x1b[0m $ openshell gateway remove my-gateway "; -const INFERENCE_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m - $ openshell inference set --provider openai --model gpt-4 - $ openshell inference get - $ openshell inference update --model gpt-4-turbo -"; - const DOCTOR_HELP: &str = "\x1b[1mALIAS\x1b[0m dr @@ -608,13 +601,6 @@ enum Commands { output: OutputFormat, }, - /// Manage inference configuration. - #[command(after_help = INFERENCE_EXAMPLES, help_template = SUBCOMMAND_HELP_TEMPLATE)] - Inference { - #[command(subcommand)] - command: Option, - }, - // =================================================================== // DIAGNOSTIC COMMANDS // =================================================================== @@ -1246,80 +1232,6 @@ enum GatewayCommands { }, } -// ----------------------------------------------------------------------- -// Inference commands -// ----------------------------------------------------------------------- - -#[derive(Subcommand, Debug)] -enum InferenceCommands { - /// Set workspace-level inference provider and model. - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Set { - /// Provider name. - #[arg(long, add = ArgValueCompleter::new(completers::complete_provider_names))] - provider: String, - - /// Model identifier to force for generation calls. - #[arg(long)] - model: String, - - /// Configure the system inference route instead of the user-facing - /// route. System inference is used by platform functions (e.g. the - /// agent harness) and is not accessible to user code. - #[arg(long)] - system: bool, - - /// Skip endpoint verification before saving the route. - #[arg(long)] - no_verify: bool, - - /// Request timeout in seconds for inference calls (0 = default 60s). - #[arg(long, default_value_t = 0)] - timeout: u64, - }, - - /// Update workspace-level inference configuration (partial update). - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Update { - /// Provider name (unchanged if omitted). - #[arg(long, add = ArgValueCompleter::new(completers::complete_provider_names))] - provider: Option, - - /// Model identifier (unchanged if omitted). - #[arg(long)] - model: Option, - - /// Target the system inference route. - #[arg(long)] - system: bool, - - /// Skip endpoint verification before saving the route. - #[arg(long)] - no_verify: bool, - - /// Request timeout in seconds for inference calls (0 = default 60s, unchanged if omitted). - #[arg(long)] - timeout: Option, - }, - - /// Get workspace-level inference provider and model. - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Get { - /// Show the system inference route instead of the user-facing route. - /// When omitted, both routes are displayed. - #[arg(long)] - system: bool, - }, - - /// Delete a workspace-level inference route. - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Delete { - /// Delete the system inference route instead of the user-facing route. - #[arg(long)] - system: bool, - }, -} - // ----------------------------------------------------------------------- // Doctor (diagnostic) commands // ----------------------------------------------------------------------- @@ -3115,69 +3027,6 @@ async fn run_async() -> Result<()> { } } - // ----------------------------------------------------------- - // Inference commands - // ----------------------------------------------------------- - Some(Commands::Inference { - command: Some(command), - }) => { - let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; - let endpoint = &ctx.endpoint; - let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); - match command { - InferenceCommands::Set { - provider, - model, - system, - no_verify, - timeout, - } => { - let route_name = if system { "sandbox-system" } else { "" }; - run::gateway_inference_set( - endpoint, - &provider, - &model, - route_name, - no_verify, - timeout, - &cli.workspace, - &tls, - ) - .await?; - } - InferenceCommands::Update { - provider, - model, - system, - no_verify, - timeout, - } => { - let route_name = if system { "sandbox-system" } else { "" }; - run::gateway_inference_update( - endpoint, - provider.as_deref(), - model.as_deref(), - route_name, - no_verify, - timeout, - &cli.workspace, - &tls, - ) - .await?; - } - InferenceCommands::Get { system } => { - let route_name = if system { Some("sandbox-system") } else { None }; - run::gateway_inference_get(endpoint, route_name, &cli.workspace, &tls).await?; - } - InferenceCommands::Delete { system } => { - let route_name = if system { "sandbox-system" } else { "" }; - run::gateway_inference_delete(endpoint, route_name, &cli.workspace, &tls) - .await?; - } - } - } - // ----------------------------------------------------------- // Sandbox commands // ----------------------------------------------------------- @@ -4031,13 +3880,6 @@ async fn run_async() -> Result<()> { .print_help() .expect("Failed to print help"); } - Some(Commands::Inference { command: None }) => { - Cli::command() - .find_subcommand_mut("inference") - .expect("inference subcommand exists") - .print_help() - .expect("Failed to print help"); - } Some(Commands::Rule { command: None }) => { Cli::command() .find_subcommand_mut("rule") @@ -4387,6 +4229,11 @@ mod tests { assert!(Cli::try_parse_from(["openshell", "info"]).is_err()); } + #[test] + fn managed_inference_is_not_a_command() { + assert!(Cli::try_parse_from(["openshell", "inference", "get"]).is_err()); + } + #[test] fn hidden_aliases_still_parse() { let cli = Cli::try_parse_from(["openshell", "lg", "sandbox-1"]) @@ -4398,54 +4245,6 @@ mod tests { )); } - #[test] - fn inference_set_accepts_no_verify_flag() { - let cli = Cli::try_parse_from([ - "openshell", - "inference", - "set", - "--provider", - "openai-dev", - "--model", - "gpt-4.1", - "--no-verify", - ]) - .expect("inference set should parse --no-verify"); - - assert!(matches!( - cli.command, - Some(Commands::Inference { - command: Some(InferenceCommands::Set { - no_verify: true, - .. - }) - }) - )); - } - - #[test] - fn inference_update_accepts_no_verify_flag() { - let cli = Cli::try_parse_from([ - "openshell", - "inference", - "update", - "--provider", - "openai-dev", - "--no-verify", - ]) - .expect("inference update should parse --no-verify"); - - assert!(matches!( - cli.command, - Some(Commands::Inference { - command: Some(InferenceCommands::Update { - no_verify: true, - .. - }) - }) - )); - } - #[test] fn completion_script_uses_openshell_command_name() { let script = normalize_completion_script( diff --git a/crates/openshell-cli/src/run.rs b/crates/openshell-cli/src/run.rs index 78a794aa30..4001159e3c 100644 --- a/crates/openshell-cli/src/run.rs +++ b/crates/openshell-cli/src/run.rs @@ -35,9 +35,8 @@ pub use crate::commands::provider::{ use crate::color::Colorize; use crate::policy_update::build_policy_update_plan; -use crate::tls::{TlsOptions, grpc_client, grpc_inference_client}; +use crate::tls::{TlsOptions, grpc_client}; use futures::StreamExt; -use indicatif::{ProgressBar, ProgressStyle}; use miette::{IntoDiagnostic, Result, WrapErr, miette}; use openshell_bootstrap::{ GatewayMetadata, clear_last_sandbox_if_matches, get_gateway_metadata, save_last_sandbox, @@ -46,20 +45,18 @@ use openshell_core::net::set_tcp_nodelay_best_effort; use openshell_core::proto::{ ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, ClearDraftChunksRequest, CreateSandboxRequest, CreateSandboxTemplateRequest, CreateSshSessionRequest, - DeleteInferenceRouteRequest, DeleteSandboxRequest, DeleteSandboxTemplateRequest, - DeleteServiceRequest, ExecSandboxRequest, ExposeServiceRequest, GetCurrentUserRequest, - GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest, - GetInferenceRouteRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, + DeleteSandboxRequest, DeleteSandboxTemplateRequest, DeleteServiceRequest, ExecSandboxRequest, + ExposeServiceRequest, GetCurrentUserRequest, GetDraftHistoryRequest, GetDraftPolicyRequest, + GetGatewayConfigRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxPolicyStatusRequest, GetSandboxRequest, GetSandboxTemplateRequest, GetServiceRequest, GpuResourceRequirements, ListSandboxPoliciesRequest, ListSandboxTemplatesRequest, ListSandboxesRequest, ListServicesRequest, PolicySource, PolicyStatus, RejectDraftChunkRequest, ResourceRequirements, RevokeSshSessionRequest, Sandbox, SandboxPhase, SandboxPolicy, SandboxResources, SandboxServiceLevel, SandboxSpec, SandboxStartup, SandboxTemplate, SandboxWorkloadConfig, - SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec, ServiceEndpointResponse, - SetInferenceRouteRequest, SettingScope, StartSandboxRequest, StopSandboxRequest, - TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, WatchSandboxRequest, - exec_sandbox_event, tcp_forward_init, + SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec, ServiceEndpointResponse, SettingScope, + StartSandboxRequest, StopSandboxRequest, TcpForwardFrame, TcpForwardInit, TcpRelayTarget, + UpdateConfigRequest, WatchSandboxRequest, exec_sandbox_event, tcp_forward_init, }; use openshell_core::settings; use openshell_core::{ObjectId, ObjectName, ObjectWorkspace}; @@ -3825,283 +3822,6 @@ fn workspace_to_json(workspace: &openshell_core::proto::Workspace) -> serde_json serde_json::Value::Object(obj) } -#[allow(clippy::too_many_arguments)] -pub async fn gateway_inference_set( - server: &str, - provider_name: &str, - model_id: &str, - route_name: &str, - no_verify: bool, - timeout_secs: u64, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let progress = if std::io::stdout().is_terminal() { - let spinner = ProgressBar::new_spinner(); - spinner.set_style( - ProgressStyle::with_template("{spinner:.cyan} {msg} ({elapsed})") - .unwrap_or_else(|_| ProgressStyle::default_spinner()), - ); - spinner.set_message("Configuring inference..."); - spinner.enable_steady_tick(Duration::from_millis(120)); - Some(spinner) - } else { - None - }; - - let mut client = grpc_inference_client(server, tls).await?; - let response = client - .set_inference_route(SetInferenceRouteRequest { - provider_name: provider_name.to_string(), - model_id: model_id.to_string(), - route_name: route_name.to_string(), - verify: false, - no_verify, - timeout_secs, - workspace: workspace.to_string(), - }) - .await; - - if let Some(progress) = &progress { - progress.finish_and_clear(); - } - - let response = response.map_err(format_inference_status)?; - - let configured = response.into_inner(); - let label = if configured.route_name == "sandbox-system" { - "System inference configured:" - } else { - "Inference configured:" - }; - println!("{}", label.cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Route:".dimmed(), configured.route_name); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - if configured.validation_performed { - println!(" {}", "Validated Endpoints:".dimmed()); - for endpoint in configured.validated_endpoints { - println!(" - {} ({})", endpoint.url, endpoint.protocol); - } - } - Ok(()) -} - -#[allow(clippy::too_many_arguments)] -pub async fn gateway_inference_update( - server: &str, - provider_name: Option<&str>, - model_id: Option<&str>, - route_name: &str, - no_verify: bool, - timeout_secs: Option, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - if provider_name.is_none() && model_id.is_none() && timeout_secs.is_none() { - return Err(miette::miette!( - "at least one of --provider, --model, or --timeout must be specified" - )); - } - - let mut client = grpc_inference_client(server, tls).await?; - - // Fetch current config to use as base for the partial update. - let current = client - .get_inference_route(GetInferenceRouteRequest { - route_name: route_name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - - let provider = provider_name.unwrap_or(¤t.provider_name); - let model = model_id.unwrap_or(¤t.model_id); - let timeout = timeout_secs.unwrap_or(current.timeout_secs); - - let progress = if std::io::stdout().is_terminal() { - let spinner = ProgressBar::new_spinner(); - spinner.set_style( - ProgressStyle::with_template("{spinner:.cyan} {msg} ({elapsed})") - .unwrap_or_else(|_| ProgressStyle::default_spinner()), - ); - spinner.set_message("Configuring inference..."); - spinner.enable_steady_tick(Duration::from_millis(120)); - Some(spinner) - } else { - None - }; - - let response = client - .set_inference_route(SetInferenceRouteRequest { - provider_name: provider.to_string(), - model_id: model.to_string(), - route_name: route_name.to_string(), - verify: false, - no_verify, - timeout_secs: timeout, - workspace: workspace.to_string(), - }) - .await; - - if let Some(progress) = &progress { - progress.finish_and_clear(); - } - - let response = response.map_err(format_inference_status)?; - - let configured = response.into_inner(); - let label = if configured.route_name == "sandbox-system" { - "System inference updated:" - } else { - "Inference updated:" - }; - println!("{}", label.cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Route:".dimmed(), configured.route_name); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - if configured.validation_performed { - println!(" {}", "Validated Endpoints:".dimmed()); - for endpoint in configured.validated_endpoints { - println!(" - {} ({})", endpoint.url, endpoint.protocol); - } - } - Ok(()) -} - -pub async fn gateway_inference_get( - server: &str, - route_name: Option<&str>, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_inference_client(server, tls).await?; - - if let Some(name) = route_name { - // Show a single route (--system was specified). - let response = client - .get_inference_route(GetInferenceRouteRequest { - route_name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let configured = response.into_inner(); - let label = if name == "sandbox-system" { - "System inference:" - } else { - "Inference:" - }; - println!("{}", label.cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - } else { - // Show both routes by default. - print_inference_route(&mut client, "Inference", "", workspace).await; - println!(); - print_inference_route(&mut client, "System inference", "sandbox-system", workspace).await; - } - Ok(()) -} - -pub async fn gateway_inference_delete( - server: &str, - route_name: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_inference_client(server, tls).await?; - - let response = client - .delete_inference_route(DeleteInferenceRouteRequest { - route_name: route_name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let label = if route_name == "sandbox-system" { - "System inference route" - } else { - "Inference route" - }; - - if response.into_inner().deleted { - println!("{label} deleted."); - } else { - println!("{label} not found (already deleted)."); - } - Ok(()) -} - -async fn print_inference_route( - client: &mut crate::tls::GrpcInferenceClient, - label: &str, - route_name: &str, - workspace: &str, -) { - match client - .get_inference_route(GetInferenceRouteRequest { - route_name: route_name.to_string(), - workspace: workspace.to_string(), - }) - .await - { - Ok(response) => { - let configured = response.into_inner(); - println!("{}", format!("{label}:").cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - } - Err(e) if e.code() == Code::NotFound => { - println!("{}", format!("{label}:").cyan().bold()); - println!(); - println!(" {}", "Not configured".dimmed()); - } - Err(e) => { - println!("{}", format!("{label}:").cyan().bold()); - println!(); - println!(" {} {}", "Error:".red(), e.message()); - } - } -} - -fn print_timeout(timeout_secs: u64) { - if timeout_secs == 0 { - println!(" {} {}s (default)", "Timeout:".dimmed(), 60); - } else { - println!(" {} {}s", "Timeout:".dimmed(), timeout_secs); - } -} - -fn format_inference_status(status: Status) -> miette::Report { - let message = status.message().trim(); - - if message.is_empty() { - return miette::miette!("inference configuration failed ({})", status.code()); - } - - miette::miette!("{message}") -} - pub fn git_repo_root(local_path: &Path) -> Result { let git_dir = if local_path.is_dir() { local_path diff --git a/crates/openshell-cli/src/tls.rs b/crates/openshell-cli/src/tls.rs index c24b84c7dc..dd1e3d559c 100644 --- a/crates/openshell-cli/src/tls.rs +++ b/crates/openshell-cli/src/tls.rs @@ -4,7 +4,6 @@ use miette::{IntoDiagnostic, Result, WrapErr}; use openshell_core::auth::EdgeAuthInterceptor; use openshell_core::net::set_tcp_nodelay_best_effort; -use openshell_core::proto::inference_client::InferenceClient; use openshell_core::proto::open_shell_client::OpenShellClient; use rustls::{ RootCertStore, @@ -26,8 +25,6 @@ use url::{Host, Url}; /// Concrete gRPC client type used by all commands. pub type GrpcClient = OpenShellClient>; -/// Concrete inference client type. -pub type GrpcInferenceClient = InferenceClient>; #[derive(Clone, Debug, Default)] pub struct TlsOptions { @@ -466,12 +463,6 @@ fn interceptor_from_tls(tls: &TlsOptions) -> Result { EdgeAuthInterceptor::new(tls.oidc_token.as_deref(), tls.edge_token.as_deref()) } -pub async fn grpc_inference_client(server: &str, tls: &TlsOptions) -> Result { - let channel = build_channel(server, tls).await?; - let interceptor = interceptor_from_tls(tls)?; - Ok(InferenceClient::with_interceptor(channel, interceptor)) -} - #[cfg(test)] mod tests { use super::tls_server_name; diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index 54f0db6902..315ff0b72f 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -1,8 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! gRPC client for fetching sandbox policy, provider environment, and inference -//! route bundles from `OpenShell` server. +//! gRPC client for fetching sandbox policy and provider environment from the +//! `OpenShell` server. //! //! Every request carries a sandbox bearer credential in the `Authorization` //! header. The token is resolved at startup from one of three sources: @@ -24,11 +24,10 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH}; use crate::proto::{ DenialSummary, ExchangeProviderSubjectTokenRequest, GetDraftPolicyRequest, - GetInferenceBundleRequest, GetInferenceBundleResponse, GetSandboxConfigRequest, - GetSandboxProviderEnvironmentRequest, IssueSandboxTokenRequest, NetworkActivitySummary, - PolicyChunk, PolicySource, PolicyStatus, RefreshSandboxTokenRequest, ReportPolicyStatusRequest, - SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse, - UpdateConfigRequest, inference_client::InferenceClient, open_shell_client::OpenShellClient, + GetSandboxConfigRequest, GetSandboxProviderEnvironmentRequest, IssueSandboxTokenRequest, + NetworkActivitySummary, PolicyChunk, PolicySource, PolicyStatus, RefreshSandboxTokenRequest, + ReportPolicyStatusRequest, SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest, + SubmitPolicyAnalysisResponse, UpdateConfigRequest, open_shell_client::OpenShellClient, }; use crate::sandbox_env; use miette::{IntoDiagnostic, Result, WrapErr}; @@ -681,12 +680,6 @@ async fn connect(endpoint: &str) -> Result> { Ok(OpenShellClient::new(channel)) } -/// Connect to the inference service. -async fn connect_inference(endpoint: &str) -> Result> { - let channel = connect_channel(endpoint).await?; - Ok(InferenceClient::new(channel)) -} - /// Fetch sandbox policy from `OpenShell` server via gRPC. /// /// Returns `Ok(Some(policy))` when the server has a policy configured, @@ -1222,17 +1215,3 @@ impl CachedOpenShellClient { Ok(()) } } - -/// Fetch the resolved inference route bundle from the server. -pub async fn fetch_inference_bundle(endpoint: &str) -> Result { - debug!(endpoint = %endpoint, "Fetching inference route bundle"); - - let mut client = connect_inference(endpoint).await?; - - let response = client - .get_inference_bundle(GetInferenceBundleRequest {}) - .await - .into_diagnostic()?; - - Ok(response.into_inner()) -} diff --git a/crates/openshell-core/src/inference.rs b/crates/openshell-core/src/inference.rs deleted file mode 100644 index 2be79d45ee..0000000000 --- a/crates/openshell-core/src/inference.rs +++ /dev/null @@ -1,509 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use std::collections::HashSet; - -// --------------------------------------------------------------------------- -// Auth header abstraction -// --------------------------------------------------------------------------- - -/// How to inject an API key on outgoing inference requests. -/// -/// Defined in `openshell-core` so both `openshell-router` (which applies it) -/// and `openshell-server` / `openshell-sandbox` (which resolve it from -/// provider metadata) can share the same type. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum AuthHeader { - /// `Authorization: Bearer ` - Bearer, - /// Custom header name (e.g. `x-api-key` for Anthropic). - Custom(&'static str), - /// Do not inject any auth header on outgoing requests. The upstream - /// is expected to authenticate itself — used when the configured - /// `default_base_url` (or operator-supplied base-URL override) points - /// at a translating bridge / proxy that holds operator-side - /// credentials in its own pod and ignores caller-supplied auth. - /// Currently used by the `aws-bedrock` profile, where `SigV4` signing - /// is deferred to a follow-up PR; today the only supported shape is - /// a bridge-fronted upstream. - None, -} - -// --------------------------------------------------------------------------- -// Inference provider profiles -// --------------------------------------------------------------------------- - -/// Static metadata describing how to talk to a specific inference provider's API. -/// -/// This is the single source of truth for provider-specific inference knowledge: -/// default endpoint, supported protocols, credential key lookup order, auth -/// header style, default headers, and allowed client-supplied passthrough -/// headers. -/// -/// This is separate from [`openshell_providers::ProviderPlugin`] which handles -/// credential *discovery* (scanning env vars). `InferenceProviderProfile` handles -/// how to *use* discovered credentials to make inference API calls. -pub struct InferenceProviderProfile { - pub provider_type: &'static str, - pub default_base_url: &'static str, - pub protocols: &'static [&'static str], - /// Credential map key names to search for the API key, in priority order. - pub credential_key_names: &'static [&'static str], - /// Config map key names to search for a base URL override, in priority order. - pub base_url_config_keys: &'static [&'static str], - /// Auth header style for outgoing requests. - pub auth: AuthHeader, - /// Default headers injected on every outgoing request. - pub default_headers: &'static [(&'static str, &'static str)], - /// Client-supplied headers that may be forwarded to the upstream backend. - /// - /// Header names must be lowercase and must not include auth headers. - pub passthrough_headers: &'static [&'static str], -} - -const OPENAI_PROTOCOLS: &[&str] = &[ - "openai_chat_completions", - "openai_completions", - "openai_responses", - "openai_embeddings", - "model_discovery", -]; - -const ANTHROPIC_PROTOCOLS: &[&str] = &["anthropic_messages", "model_discovery"]; - -/// Default protocol set for the Vertex AI profile. These are overridden at route -/// resolution time in `resolve_vertex_ai_route`: Anthropic models use -/// `anthropic_messages`, while Gemini and other models use the OpenAI-compatible -/// endpoint with `openai_chat_completions`. This default applies only to the -/// base-URL-override escape hatch path. -const VERTEX_AI_PROTOCOLS: &[&str] = &["anthropic_messages", "model_discovery"]; - -// `aws_bedrock_invoke_stream` (`/model/{id}/invoke-with-response-stream`) is -// deferred to a follow-up alongside protocol-aware AWS event-stream error -// handling: the shared streaming relay's truncation/timeout path injects -// SSE-formatted error frames, which would corrupt downstream Bedrock -// event-stream parsers. Until that lands, this profile advertises only -// the buffered `InvokeModel` shape. -const AWS_BEDROCK_PROTOCOLS: &[&str] = &["aws_bedrock_invoke"]; - -static OPENAI_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "openai", - default_base_url: "https://api.openai.com/v1", - protocols: OPENAI_PROTOCOLS, - credential_key_names: &["OPENAI_API_KEY"], - base_url_config_keys: &["OPENAI_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &["openai-organization", "x-model-id"], -}; - -static ANTHROPIC_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "anthropic", - default_base_url: "https://api.anthropic.com/v1", - protocols: ANTHROPIC_PROTOCOLS, - credential_key_names: &["ANTHROPIC_API_KEY"], - base_url_config_keys: &["ANTHROPIC_BASE_URL"], - auth: AuthHeader::Custom("x-api-key"), - default_headers: &[("anthropic-version", "2023-06-01")], - passthrough_headers: &["anthropic-version", "anthropic-beta"], -}; - -/// Credential environment variable names for the Vertex AI provider, in priority order. -/// -/// These are referenced by both the provider discovery logic in `openshell-providers` -/// and the inference profile here so both crates agree on which env vars hold credentials. -pub const VERTEX_AI_CREDENTIAL_KEY_NAMES: &[&str] = &[ - "GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN", - "VERTEX_AI_SERVICE_ACCOUNT_TOKEN", - "GOOGLE_VERTEX_AI_TOKEN", - "VERTEX_AI_TOKEN", -]; - -/// GCP project ID config key for Vertex AI providers. -pub const VERTEX_AI_PROJECT_ID_KEY: &str = "VERTEX_AI_PROJECT_ID"; - -/// GCP region/location config key for Vertex AI providers. -pub const VERTEX_AI_REGION_KEY: &str = "VERTEX_AI_REGION"; - -/// Publisher override config key for Vertex AI providers. -/// -/// Set to `"anthropic"` to force Anthropic Messages API routing regardless of model name, -/// or any other value to force OpenAI-compatible routing. -pub const VERTEX_AI_PUBLISHER_KEY: &str = "VERTEX_AI_PUBLISHER"; - -/// Config key names scanned during provider discovery, in addition to credential keys. -/// -/// These are referenced by the provider discovery plugin in `openshell-providers` to -/// collect Vertex AI config from the environment during `--from-existing` flows. -pub const VERTEX_AI_CONFIG_KEY_NAMES: &[&str] = &[ - VERTEX_AI_PROJECT_ID_KEY, - VERTEX_AI_REGION_KEY, - "GOOGLE_VERTEX_AI_BASE_URL", - "VERTEX_AI_BASE_URL", - VERTEX_AI_PUBLISHER_KEY, -]; - -static VERTEX_AI_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "google-vertex-ai", - // Base URL is project/region specific and built at route resolution time. - default_base_url: "", - protocols: VERTEX_AI_PROTOCOLS, - credential_key_names: VERTEX_AI_CREDENTIAL_KEY_NAMES, - base_url_config_keys: &["GOOGLE_VERTEX_AI_BASE_URL", "VERTEX_AI_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &[], -}; - -static NVIDIA_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "nvidia", - default_base_url: "https://integrate.api.nvidia.com/v1", - protocols: OPENAI_PROTOCOLS, - credential_key_names: &["NVIDIA_API_KEY"], - base_url_config_keys: &["NVIDIA_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &["x-model-id"], -}; - -static DEEPINFRA_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "deepinfra", - default_base_url: "https://api.deepinfra.com/v1/openai", - protocols: OPENAI_PROTOCOLS, - credential_key_names: &["DEEPINFRA_API_KEY"], - base_url_config_keys: &["DEEPINFRA_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &["x-model-id"], -}; - -// AWS Bedrock — registered as bridge-fronted (no router-side auth -// injection). Real AWS Bedrock requires `SigV4` signing of every request, -// which is deferred to a follow-up PR (see #1704 thread). Until then, -// operators point `BEDROCK_BASE_URL` at a translating bridge or -// Bedrock-compatible proxy that handles auth in its own pod. The router -// passes Bedrock `InvokeModel` requests through opaquely; the L7 pattern -// `/model/{modelId}/invoke` is wired up in -// `crates/openshell-sandbox/src/l7/inference.rs`. `InvokeModelWithResponseStream` -// is deferred to the same follow-up that adds protocol-aware error framing. -// -// Note: `default_base_url` is intentionally an empty string. Without -// `BEDROCK_BASE_URL` config, route resolution rejects the provider -// rather than silently forwarding prompts to real AWS Bedrock with -// `auth: None` (which would fail upstream and risks operator -// surprise). Once the `SigV4` follow-up lands, the default can revert -// to `https://bedrock-runtime.us-east-1.amazonaws.com`. -static AWS_BEDROCK_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "aws-bedrock", - default_base_url: "", - protocols: AWS_BEDROCK_PROTOCOLS, - // No single API key for Bedrock — `SigV4` takes four credentials - // (access key id, secret, session token, region) and signs requests - // rather than injecting a header. Until the `SigV4` follow-up lands - // the router-side auth shape is `None` and no credential lookup is - // required at route time. - credential_key_names: &[], - base_url_config_keys: &["BEDROCK_BASE_URL"], - auth: AuthHeader::None, - default_headers: &[], - passthrough_headers: &[], -}; - -/// Canonicalize an inference provider type string to a well-known identifier. -/// -/// Returns `Some(canonical_name)` for recognized inference providers, -/// `None` for unrecognized inputs. This is the single source of truth for -/// Vertex AI (and other inference provider) alias resolution so that both -/// [`profile_for`] and `openshell-providers` normalization agree. -#[must_use] -pub fn normalize_inference_provider_type(input: &str) -> Option<&'static str> { - match input.trim().to_ascii_lowercase().as_str() { - "openai" => Some("openai"), - "anthropic" => Some("anthropic"), - "nvidia" => Some("nvidia"), - "deepinfra" => Some("deepinfra"), - "aws-bedrock" => Some("aws-bedrock"), - "google-vertex-ai" | "vertex" | "vertex-ai" | "google-vertex" | "gcp-vertex" => { - Some("google-vertex-ai") - } - _ => None, - } -} - -/// Look up the inference provider profile for a given provider type. -/// -/// Returns `None` for provider types that don't support inference routing -/// (e.g. `github`, `gitlab`, `outlook`). -pub fn profile_for(provider_type: &str) -> Option<&'static InferenceProviderProfile> { - match normalize_inference_provider_type(provider_type)? { - "openai" => Some(&OPENAI_PROFILE), - "anthropic" => Some(&ANTHROPIC_PROFILE), - "nvidia" => Some(&NVIDIA_PROFILE), - "deepinfra" => Some(&DEEPINFRA_PROFILE), - "google-vertex-ai" => Some(&VERTEX_AI_PROFILE), - "aws-bedrock" => Some(&AWS_BEDROCK_PROFILE), - _ => None, - } -} - -/// Derive the [`AuthHeader`] and default headers for a provider type string. -/// -/// This is a convenience wrapper around [`profile_for`] for callers that only -/// need the auth/header information (e.g. the sandbox bundle-to-route -/// conversion). -pub fn auth_for_provider_type(provider_type: &str) -> (AuthHeader, Vec<(String, String)>) { - let (auth, headers, _) = route_headers_for_provider_type(provider_type); - (auth, headers) -} - -/// Derive routing header policy for a provider type string. -/// -/// Returns the auth injection mode, route-level default headers, and the -/// allowed client-supplied passthrough headers for `inference.local`. -pub fn route_headers_for_provider_type( - provider_type: &str, -) -> (AuthHeader, Vec<(String, String)>, Vec) { - profile_for(provider_type).map_or_else( - || (AuthHeader::Bearer, Vec::new(), Vec::new()), - |profile| { - let headers = profile - .default_headers - .iter() - .map(|(k, v)| ((*k).to_string(), (*v).to_string())) - .collect(); - let passthrough_headers = profile - .passthrough_headers - .iter() - .map(|name| (*name).to_string()) - .collect(); - (profile.auth.clone(), headers, passthrough_headers) - }, - ) -} - -/// Derive routing header policy for a specific resolved route. -/// -/// Most providers only need their provider type. Vertex AI is special because -/// Claude routes should forward `anthropic-beta`, while Gemini/OpenAI-compatible -/// routes should not inherit Anthropic passthrough headers. -pub fn route_headers_for_route( - provider_type: &str, - protocols: &[String], -) -> (AuthHeader, Vec<(String, String)>, Vec) { - let (auth, headers, mut passthrough_headers) = route_headers_for_provider_type(provider_type); - if profile_for(provider_type).is_some_and(|profile| profile.provider_type == "google-vertex-ai") - { - let is_vertex_anthropic = protocols - .iter() - .any(|protocol| protocol == "anthropic_messages"); - if is_vertex_anthropic && !passthrough_headers.iter().any(|h| h == "anthropic-beta") { - passthrough_headers.push("anthropic-beta".to_string()); - } - } - (auth, headers, passthrough_headers) -} - -// --------------------------------------------------------------------------- -// Protocol normalization -// --------------------------------------------------------------------------- - -/// Normalize a list of protocol strings: trim, lowercase, deduplicate, skip empty. -pub fn normalize_protocols(protocols: &[String]) -> Vec { - let mut normalized = Vec::new(); - let mut seen = HashSet::new(); - - for protocol in protocols { - let candidate = protocol.trim().to_ascii_lowercase(); - if candidate.is_empty() { - continue; - } - if seen.insert(candidate.clone()) { - normalized.push(candidate); - } - } - - normalized -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn normalizes_and_deduplicates() { - let input = vec![ - "OpenAI_Chat_Completions".to_string(), - " openai_chat_completions ".to_string(), - "anthropic_messages".to_string(), - ]; - let result = normalize_protocols(&input); - assert_eq!( - result, - vec!["openai_chat_completions", "anthropic_messages"] - ); - } - - #[test] - fn skips_empty_and_whitespace() { - let input = vec![String::new(), " ".to_string(), "valid".to_string()]; - let result = normalize_protocols(&input); - assert_eq!(result, vec!["valid"]); - } - - #[test] - fn empty_input() { - let result = normalize_protocols(&[]); - assert!(result.is_empty()); - } - - #[test] - fn profile_for_known_types() { - assert!(profile_for("openai").is_some()); - assert!(profile_for("anthropic").is_some()); - assert!(profile_for("nvidia").is_some()); - assert!(profile_for("deepinfra").is_some()); - assert!(profile_for("aws-bedrock").is_some()); - assert!(profile_for("OpenAI").is_some()); // case insensitive - assert!(profile_for("AWS-Bedrock").is_some()); // case insensitive - } - - #[test] - fn aws_bedrock_uses_no_auth_header() { - let (auth, headers) = auth_for_provider_type("aws-bedrock"); - assert_eq!(auth, AuthHeader::None); - assert!(headers.is_empty()); - } - - #[test] - fn aws_bedrock_profile_has_no_credential_keys() { - let profile = profile_for("aws-bedrock").expect("profile registered"); - // No router-side credential lookup until the `SigV4` follow-up. - assert!(profile.credential_key_names.is_empty()); - assert_eq!(profile.base_url_config_keys, &["BEDROCK_BASE_URL"]); - } - - #[test] - fn aws_bedrock_protocols_are_bedrock_specific() { - let profile = profile_for("aws-bedrock").expect("profile registered"); - assert!(profile.protocols.contains(&"aws_bedrock_invoke")); - // `aws_bedrock_invoke_stream` is deferred to the follow-up that adds - // protocol-aware AWS event-stream error framing; until then the - // profile advertises only the buffered `InvokeModel` shape. - assert!(!profile.protocols.contains(&"aws_bedrock_invoke_stream")); - } - - #[test] - fn profile_for_deepinfra() { - let profile = profile_for("deepinfra").expect("deepinfra profile should exist"); - assert_eq!(profile.provider_type, "deepinfra"); - assert_eq!( - profile.default_base_url, - "https://api.deepinfra.com/v1/openai" - ); - assert_eq!(profile.auth, AuthHeader::Bearer); - } - - #[test] - fn openai_compatible_profiles_include_embeddings() { - for provider_type in ["openai", "nvidia", "deepinfra"] { - let profile = profile_for(provider_type).expect("provider profile should exist"); - assert!( - profile.protocols.contains(&"openai_embeddings"), - "{provider_type} should route OpenAI-compatible embeddings" - ); - } - } - - #[test] - fn profile_for_unknown_types() { - assert!(profile_for("github").is_none()); - assert!(profile_for("gitlab").is_none()); - assert!(profile_for("unknown").is_none()); - } - - #[test] - fn auth_for_anthropic_uses_custom_header() { - let (auth, headers) = auth_for_provider_type("anthropic"); - assert_eq!(auth, AuthHeader::Custom("x-api-key")); - assert!(headers.iter().any(|(k, _)| k == "anthropic-version")); - } - - #[test] - fn route_headers_for_openai_include_passthrough_headers() { - let (_, _, passthrough_headers) = route_headers_for_provider_type("openai"); - assert!( - passthrough_headers - .iter() - .any(|name| name == "openai-organization") - ); - assert!(passthrough_headers.iter().any(|name| name == "x-model-id")); - } - - #[test] - fn route_headers_for_anthropic_include_passthrough_headers() { - let (_, _, passthrough_headers) = route_headers_for_provider_type("anthropic"); - assert!( - passthrough_headers - .iter() - .any(|name| name == "anthropic-version") - ); - assert!( - passthrough_headers - .iter() - .any(|name| name == "anthropic-beta") - ); - } - - #[test] - fn auth_for_openai_uses_bearer() { - let (auth, headers) = auth_for_provider_type("openai"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - } - - #[test] - fn profile_for_vertex_types() { - for key in &["google-vertex-ai", "vertex", "vertex-ai"] { - let profile = profile_for(key).expect("vertex profile should be Some"); - assert_eq!(profile.provider_type, "google-vertex-ai"); - } - } - - #[test] - fn auth_for_vertex_uses_bearer() { - let (auth, headers) = auth_for_provider_type("google-vertex-ai"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - } - - #[test] - fn route_headers_for_vertex_anthropic_route_forward_beta_only() { - let (_, headers, passthrough_headers) = - route_headers_for_route("google-vertex-ai", &["anthropic_messages".to_string()]); - assert!(headers.is_empty()); - assert_eq!(passthrough_headers, vec!["anthropic-beta".to_string()]); - } - - #[test] - fn route_headers_for_vertex_openai_route_do_not_forward_anthropic_headers() { - let (_, headers, passthrough_headers) = - route_headers_for_route("google-vertex-ai", &["openai_chat_completions".to_string()]); - assert!(headers.is_empty()); - assert!(passthrough_headers.is_empty()); - } - - #[test] - fn auth_for_unknown_defaults_to_bearer() { - let (auth, headers) = auth_for_provider_type("unknown"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - } - - #[test] - fn route_headers_for_unknown_are_empty() { - let (auth, headers, passthrough_headers) = route_headers_for_provider_type("unknown"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - assert!(passthrough_headers.is_empty()); - } -} diff --git a/crates/openshell-core/src/lib.rs b/crates/openshell-core/src/lib.rs index 7acb72dd6f..3ebc601637 100644 --- a/crates/openshell-core/src/lib.rs +++ b/crates/openshell-core/src/lib.rs @@ -27,7 +27,6 @@ pub mod gpu; pub mod grpc_client; pub mod host_pattern; pub mod image; -pub mod inference; pub mod jwt; pub mod local_api_socket; pub mod metadata; diff --git a/crates/openshell-core/src/metadata.rs b/crates/openshell-core/src/metadata.rs index f885812c4e..b72bded843 100644 --- a/crates/openshell-core/src/metadata.rs +++ b/crates/openshell-core/src/metadata.rs @@ -6,9 +6,9 @@ //! These traits provide uniform access to `ObjectMeta` fields across all resource types. use crate::proto::{ - InferenceRoute, ObjectForTest, Provider, Sandbox, SandboxStatus, SandboxWorkloadTemplate, - ServiceEndpoint, SshSession, StoredProviderCredentialRefreshState, StoredProviderProfile, - Workspace, WorkspaceMember, + ObjectForTest, Provider, Sandbox, SandboxStatus, SandboxWorkloadTemplate, ServiceEndpoint, + SshSession, StoredProviderCredentialRefreshState, StoredProviderProfile, Workspace, + WorkspaceMember, }; use std::collections::HashMap; @@ -401,48 +401,6 @@ impl ObjectWorkspace for ServiceEndpoint { } } -// Implementations for InferenceRoute -impl ObjectId for InferenceRoute { - fn object_id(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.id.as_str()) - } -} - -impl ObjectName for InferenceRoute { - fn object_name(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.name.as_str()) - } -} - -impl ObjectLabels for InferenceRoute { - fn object_labels(&self) -> Option> { - self.metadata.as_ref().map(|m| m.labels.clone()) - } -} - -impl SetResourceVersion for InferenceRoute { - fn set_resource_version(&mut self, version: u64) { - if let Some(meta) = self.metadata.as_mut() { - meta.resource_version = version; - } - } -} - -impl GetResourceVersion for InferenceRoute { - fn get_resource_version(&self) -> u64 { - self.metadata.as_ref().map_or(0, |m| m.resource_version) - } -} - -impl ObjectWorkspace for InferenceRoute { - fn object_workspace(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) - } - fn requires_workspace() -> bool { - true - } -} - // Implementations for WorkspaceMember impl ObjectId for WorkspaceMember { fn object_id(&self) -> &str { diff --git a/crates/openshell-core/src/policy.rs b/crates/openshell-core/src/policy.rs index 1645b9da44..076639e25c 100644 --- a/crates/openshell-core/src/policy.rs +++ b/crates/openshell-core/src/policy.rs @@ -101,7 +101,7 @@ impl TryFrom for SandboxPolicy { fn try_from(proto: ProtoSandboxPolicy) -> Result { // In cluster mode we always run with proxy networking so all egress - // can be evaluated by OPA and `inference.local` is always addressable. + // can be evaluated by OPA. let network = NetworkPolicy { mode: NetworkMode::Proxy, proxy: Some(ProxyPolicy { http_addr: None }), diff --git a/crates/openshell-core/src/proto/mod.rs b/crates/openshell-core/src/proto/mod.rs index d3b3405813..cd4b24afe9 100644 --- a/crates/openshell-core/src/proto/mod.rs +++ b/crates/openshell-core/src/proto/mod.rs @@ -59,10 +59,6 @@ pub mod test { pub use super::generated::openshell::test::v1::*; } -pub mod inference { - pub use super::generated::openshell::inference::v1; -} - pub mod middleware { pub use super::generated::openshell::middleware::v1; } @@ -73,7 +69,6 @@ pub mod gateway_interceptor { pub use datamodel::v1::*; pub use gateway_interceptor::v1::*; -pub use inference::v1::*; pub use middleware::v1::*; pub use openshell::*; pub use sandbox::v1::*; diff --git a/crates/openshell-gateway-interceptors/src/proto_json.rs b/crates/openshell-gateway-interceptors/src/proto_json.rs index d7ea5a241c..3abee67535 100644 --- a/crates/openshell-gateway-interceptors/src/proto_json.rs +++ b/crates/openshell-gateway-interceptors/src/proto_json.rs @@ -369,7 +369,6 @@ mod tests { let codec = ProtoJsonCodec::openshell().unwrap(); for (message_name, field_name) in [ ("openshell.datamodel.v1.Provider", "credentials"), - ("openshell.inference.v1.ResolvedRoute", "api_key"), ("openshell.compute.v1.DriverSandboxSpec", "sandbox_token"), ("openshell.v1.IssueSandboxTokenResponse", "token"), ("openshell.v1.RefreshSandboxTokenResponse", "token"), diff --git a/crates/openshell-ocsf/src/builders/api_activity.rs b/crates/openshell-ocsf/src/builders/api_activity.rs index 03c857f082..f3cddbad8e 100644 --- a/crates/openshell-ocsf/src/builders/api_activity.rs +++ b/crates/openshell-ocsf/src/builders/api_activity.rs @@ -159,9 +159,9 @@ mod tests { .ai_model(AiModel::new("claude-3-haiku", "anthropic")) .http_request(HttpRequest::new( "POST", - Url::new("https", "inference.local", "/v1/messages", 443), + Url::new("https", "api.anthropic.com", "/v1/messages", 443), )) - .dst_endpoint(Endpoint::from_domain("inference.local", 443)) + .dst_endpoint(Endpoint::from_domain("api.anthropic.com", 443)) .unmapped("latency_ms", 701_u64) .unmapped("input_tokens", 12_u64) .message("Model call: claude-3-haiku via anthropic") diff --git a/crates/openshell-ocsf/src/events/api_activity.rs b/crates/openshell-ocsf/src/events/api_activity.rs index 55c2f70d6d..456c0c768b 100644 --- a/crates/openshell-ocsf/src/events/api_activity.rs +++ b/crates/openshell-ocsf/src/events/api_activity.rs @@ -110,7 +110,7 @@ mod tests { actor: Actor { process: Process::new("supervisor", 1), }, - src_endpoint: Endpoint::from_domain("inference.local", 443), + src_endpoint: Endpoint::from_domain("api.anthropic.com", 443), http_request: None, http_response: None, dst_endpoint: None, @@ -128,7 +128,7 @@ mod tests { assert_eq!(json["activity_id"], 99); assert_eq!(json["api"]["operation"], "POST /v1/messages"); assert_eq!(json["actor"]["process"]["name"], "supervisor"); - assert_eq!(json["src_endpoint"]["domain"], "inference.local"); + assert_eq!(json["src_endpoint"]["domain"], "api.anthropic.com"); assert_eq!(json["ai_model"]["name"], "claude-3-haiku"); } diff --git a/crates/openshell-ocsf/src/events/config_state_change.rs b/crates/openshell-ocsf/src/events/config_state_change.rs index 69333554a1..d2c8b900c1 100644 --- a/crates/openshell-ocsf/src/events/config_state_change.rs +++ b/crates/openshell-ocsf/src/events/config_state_change.rs @@ -10,7 +10,7 @@ use crate::events::base_event::BaseEventData; /// OCSF Device Config State Change Event [5019]. /// -/// Policy engine and inference routing configuration changes. +/// Policy engine and runtime configuration changes. #[derive(Debug, Clone, PartialEq, Eq, Deserialize)] pub struct DeviceConfigStateChangeEvent { /// Common base event fields. diff --git a/crates/openshell-ocsf/tests/roundtrip.rs b/crates/openshell-ocsf/tests/roundtrip.rs index e7d37c9036..d179510b1a 100644 --- a/crates/openshell-ocsf/tests/roundtrip.rs +++ b/crates/openshell-ocsf/tests/roundtrip.rs @@ -197,12 +197,11 @@ fn api_activity_round_trips() { .status(StatusId::Success) .http_request(HttpRequest { http_method: HttpMethod::Post, - url: Some(Url::new("https", "inference.local", "/v1/chat", 443)), + url: Some(Url::new("https", "api.example.com", "/v1/chat", 443)), }) - .dst_endpoint(Endpoint::from_domain("inference.local", 443)) + .dst_endpoint(Endpoint::from_domain("api.example.com", 443)) .ai_model(AiModel::new("llama-3.1-8b", "nvidia")) - .message("inference request routed") - .unmapped("route", "system") + .message("inference request completed") .build(); assert_round_trips("api_activity", &event); diff --git a/crates/openshell-policy/src/lib.rs b/crates/openshell-policy/src/lib.rs index 5be8173e3e..f43215952e 100644 --- a/crates/openshell-policy/src/lib.rs +++ b/crates/openshell-policy/src/lib.rs @@ -1106,7 +1106,7 @@ pub const LEGACY_CONTAINER_POLICY_PATH: &str = "/etc/navigator/policy.yaml"; /// This policy grants filesystem access to standard system paths, leaves /// process identity selection to the compute runtime, enables Landlock in /// best-effort mode, and **blocks all network access** (no network policies, -/// no inference routing). +/// and no provider-derived endpoint access). pub fn restrictive_default_policy() -> SandboxPolicy { SandboxPolicy { version: 1, diff --git a/crates/openshell-prover/testdata/accepted-risks.yaml b/crates/openshell-prover/testdata/accepted-risks.yaml deleted file mode 100644 index 6af5c7146c..0000000000 --- a/crates/openshell-prover/testdata/accepted-risks.yaml +++ /dev/null @@ -1,7 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -accepted_risks: - - query: inference_relay - reason: "Demo environment — inference.local not configured." - accepted_by: demo diff --git a/crates/openshell-providers/src/discovery.rs b/crates/openshell-providers/src/discovery.rs index 16cf497e92..00e8da1890 100644 --- a/crates/openshell-providers/src/discovery.rs +++ b/crates/openshell-providers/src/discovery.rs @@ -67,7 +67,7 @@ pub fn discover_from_profile( } if profile.id == "google-vertex-ai" { - for key in openshell_core::inference::VERTEX_AI_CONFIG_KEY_NAMES { + for key in crate::VERTEX_AI_CONFIG_KEY_NAMES { if let Some(value) = context.env_var(key) && !value.trim().is_empty() { diff --git a/crates/openshell-providers/src/lib.rs b/crates/openshell-providers/src/lib.rs index c5112ee3f0..8b9f0d3132 100644 --- a/crates/openshell-providers/src/lib.rs +++ b/crates/openshell-providers/src/lib.rs @@ -24,6 +24,16 @@ pub use profiles::{ strategy_output_env_key, strategy_output_spec, strategy_primary_env_key, validate_profile_set, }; +pub const VERTEX_AI_PROJECT_ID_KEY: &str = "VERTEX_AI_PROJECT_ID"; +pub const VERTEX_AI_REGION_KEY: &str = "VERTEX_AI_REGION"; +pub const VERTEX_AI_CONFIG_KEY_NAMES: &[&str] = &[ + VERTEX_AI_PROJECT_ID_KEY, + VERTEX_AI_REGION_KEY, + "GOOGLE_VERTEX_AI_BASE_URL", + "VERTEX_AI_BASE_URL", + "VERTEX_AI_PUBLISHER", +]; + #[derive(Debug, thiserror::Error)] pub enum ProviderError { #[error("unsupported provider type: {0}")] @@ -128,13 +138,16 @@ impl ProviderRegistry { #[must_use] pub fn normalize_provider_type(input: &str) -> Option<&'static str> { - // Inference provider aliases are canonicalized in openshell-core so that - // openshell-server and openshell-providers agree on the same mapping. - if let Some(canonical) = openshell_core::inference::normalize_inference_provider_type(input) { - return Some(canonical); - } let normalized = input.trim().to_ascii_lowercase(); match normalized.as_str() { + "openai" => Some("openai"), + "anthropic" => Some("anthropic"), + "nvidia" => Some("nvidia"), + "deepinfra" => Some("deepinfra"), + "aws-bedrock" => Some("aws-bedrock"), + "google-vertex-ai" | "vertex" | "vertex-ai" | "google-vertex" | "gcp-vertex" => { + Some("google-vertex-ai") + } "claude" | "claude-code" | "claude_code" => Some("claude-code"), "codex" => Some("codex"), "copilot" => Some("copilot"), @@ -169,16 +182,18 @@ mod tests { assert_eq!(normalize_provider_type("openai"), Some("openai")); assert_eq!(normalize_provider_type("anthropic"), Some("anthropic")); assert_eq!(normalize_provider_type("nvidia"), Some("nvidia")); + assert_eq!(normalize_provider_type("deepinfra"), Some("deepinfra")); + assert_eq!(normalize_provider_type("aws-bedrock"), Some("aws-bedrock")); assert_eq!(normalize_provider_type("copilot"), Some("copilot")); - assert_eq!( - normalize_provider_type("google-vertex-ai"), - Some("google-vertex-ai") - ); - assert_eq!(normalize_provider_type("vertex"), Some("google-vertex-ai")); - assert_eq!( - normalize_provider_type("vertex-ai"), - Some("google-vertex-ai") - ); + for alias in [ + "google-vertex-ai", + "vertex", + "vertex-ai", + "google-vertex", + "gcp-vertex", + ] { + assert_eq!(normalize_provider_type(alias), Some("google-vertex-ai")); + } assert_eq!(normalize_provider_type("unknown"), None); } diff --git a/crates/openshell-providers/src/providers/vertex.rs b/crates/openshell-providers/src/providers/vertex.rs index ba3aacd007..e8e1857a91 100644 --- a/crates/openshell-providers/src/providers/vertex.rs +++ b/crates/openshell-providers/src/providers/vertex.rs @@ -4,9 +4,8 @@ use std::collections::HashMap; use openshell_core::google_cloud; -use openshell_core::inference; -use crate::{Provider, ProviderPlugin}; +use crate::{Provider, ProviderPlugin, VERTEX_AI_PROJECT_ID_KEY, VERTEX_AI_REGION_KEY}; pub struct VertexProvider; @@ -18,7 +17,7 @@ impl ProviderPlugin for VertexProvider { fn inject_env(&self, provider: &Provider, env: &mut HashMap) { if let Some(project) = provider .config - .get(inference::VERTEX_AI_PROJECT_ID_KEY) + .get(VERTEX_AI_PROJECT_ID_KEY) .filter(|v| !v.trim().is_empty()) { let trimmed = project.trim().to_string(); @@ -32,7 +31,7 @@ impl ProviderPlugin for VertexProvider { if let Some(region) = provider .config - .get(inference::VERTEX_AI_REGION_KEY) + .get(VERTEX_AI_REGION_KEY) .filter(|v| !v.trim().is_empty()) { let trimmed = region.trim().to_string(); diff --git a/crates/openshell-router/Cargo.toml b/crates/openshell-router/Cargo.toml deleted file mode 100644 index 97bbf4dc76..0000000000 --- a/crates/openshell-router/Cargo.toml +++ /dev/null @@ -1,31 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -[package] -name = "openshell-router" -description = "Inference routing library for OpenShell" -version.workspace = true -edition.workspace = true -rust-version.workspace = true -license.workspace = true -repository.workspace = true - -[dependencies] -openshell-core = { path = "../openshell-core", default-features = false } -bytes = { workspace = true } -reqwest = { workspace = true } -serde = { workspace = true } -serde_json = { workspace = true } -thiserror = { workspace = true } -tracing = { workspace = true } -tokio = { workspace = true } -serde_yml = { workspace = true } -uuid = { workspace = true } - -[dev-dependencies] -wiremock = "0.6" -tokio = { workspace = true, features = ["test-util"] } -tempfile = "3" - -[lints] -workspace = true diff --git a/crates/openshell-router/README.md b/crates/openshell-router/README.md deleted file mode 100644 index 8c9527da32..0000000000 --- a/crates/openshell-router/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# openshell-router - -`openshell-router` is the inference routing and upstream execution engine used -by the sandbox proxy and gateway inference validation paths. - -## Responsibilities - -- Select an upstream route from a candidate set (based on protocol compatibility). -- Forward raw HTTP requests to the selected upstream backend. -- Normalize upstream failures into router-level errors (`unauthorized`, `unavailable`, protocol/internal errors). -- Keep routing decision logic in one place so strategies can evolve (fallbacks, scoring, health-based routing). - -## Non-responsibilities - -- Authentication and sandbox identity. -- Authorization and policy enforcement. -- Persistence of routes/entities. -- Loading sandbox or policy objects. - -These are owned by `openshell-server` and `openshell-sandbox`. - -## Integration Contract - -Current split: - -- `openshell-server`: - - authenticates user-facing inference configuration changes - - resolves managed route candidates from provider records - - validates backend endpoints -- `openshell-sandbox`: - - intercepts `https://inference.local` - - detects the source inference protocol - - passes sanitized requests and resolved route candidates to the router -- `openshell-router`: - - picks a route from candidates (`proxy_with_candidates`) - - forwards the HTTP request upstream and returns the raw response - -## Public APIs - -- `Router::proxy_with_candidates(source_protocol, method, path, headers, body, &[ResolvedRoute])` - - Filters candidates by protocol compatibility, then forwards the request to the first match. - - Preferred path for entity-driven server routing. - -## Notes - -- Route selection matches candidates by `protocol` field (e.g. `openai_chat_completions`). -- Route selection is intentionally simple and will evolve. diff --git a/crates/openshell-router/src/backend.rs b/crates/openshell-router/src/backend.rs deleted file mode 100644 index 84a5c6acf5..0000000000 --- a/crates/openshell-router/src/backend.rs +++ /dev/null @@ -1,2729 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use crate::RouterError; -use crate::config::{AuthHeader, ResolvedRoute}; -use crate::mock; -use std::collections::HashSet; - -/// Maximum buffered inference response body, in bytes. The buffered path -/// reads the whole response into memory; the route timeout bounds time, not -/// memory, so without this cap an oversized upstream could force unbounded -/// allocation. Mirrors the sandbox streaming byte cap. Over-cap responses fail -/// as an upstream protocol error. -const MAX_BUFFERED_RESPONSE_BODY: usize = 32 * 1024 * 1024; - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct ValidatedEndpoint { - pub url: String, - pub protocol: String, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ValidationFailureKind { - RequestShape, - Credentials, - RateLimited, - Connectivity, - UpstreamHealth, - Unexpected, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct ValidationFailure { - pub kind: ValidationFailureKind, - pub details: String, -} - -struct ValidationProbe { - path: &'static str, - protocol: &'static str, - body: bytes::Bytes, - /// Alternate body to try when the primary probe is rejected specifically - /// for `max_completion_tokens`. Used for `OpenAI` chat completions where - /// newer models require `max_completion_tokens` while legacy/self-hosted - /// backends only accept `max_tokens`. The retry is gated on the error - /// body naming that parameter, so an unrelated request-shape rejection - /// (wrong protocol for the model) falls through instead. - fallback_body: Option, -} - -/// Response from a proxied HTTP request to a backend (fully buffered). -#[derive(Debug)] -pub struct ProxyResponse { - pub status: u16, - pub headers: Vec<(String, String)>, - pub body: bytes::Bytes, - /// Model identity of the route that produced this response. - pub route_model: Option, - /// Endpoint of the route that produced this response. - pub route_endpoint: Option, -} - -/// Response from a proxied HTTP request where the body can be streamed -/// incrementally via [`StreamingProxyResponse::next_chunk`]. -pub struct StreamingProxyResponse { - pub status: u16, - pub headers: Vec<(String, String)>, - /// Model identity of the route that produced this response. - pub route_model: Option, - /// Endpoint of the route that produced this response. - pub route_endpoint: Option, - /// Either a live response to stream from, or a pre-buffered body (for mock routes). - body: StreamingBody, -} - -enum StreamingBody { - /// Live upstream response — call `chunk().await` to read incrementally. - Live(reqwest::Response), - /// Pre-buffered body (e.g. from mock routes). Drained on first `next_chunk()`. - Buffered(Option), -} - -/// The `anthropic_version` value required by Vertex AI's rawPredict endpoint for -/// Anthropic Claude models. Google publishes this version string; update here if -/// the Vertex AI Anthropic API version changes. -/// -/// See: -const VERTEX_ANTHROPIC_VERSION: &str = "vertex-2023-10-16"; - -const COMMON_INFERENCE_REQUEST_HEADERS: [&str; 4] = - ["content-type", "accept", "accept-encoding", "user-agent"]; - -/// Anthropic-SDK-only body fields that Vertex AI rawPredict does not accept. -/// Vertex AI's rawPredict endpoint uses strict pydantic validation and rejects -/// any extra inputs not in its schema. The Anthropic SDKs' native Vertex -/// transport (`AnthropicVertex`) strips these automatically; we must do the same -/// when proxying through `inference.local`. -/// -/// `model` is handled separately (rewritten or stripped depending on route type) -/// and is intentionally omitted here. -const VERTEX_UNSUPPORTED_BODY_FIELDS: &[&str] = &["context_management"]; - -impl StreamingProxyResponse { - /// Create from a fully-buffered [`ProxyResponse`] (for mock routes). - pub fn from_buffered(resp: ProxyResponse) -> Self { - Self { - status: resp.status, - headers: resp.headers, - route_model: resp.route_model, - route_endpoint: resp.route_endpoint, - body: StreamingBody::Buffered(Some(resp.body)), - } - } - - /// Read the next body chunk. Returns `None` when the body is exhausted. - pub async fn next_chunk(&mut self) -> Result, RouterError> { - match &mut self.body { - StreamingBody::Live(response) => response.chunk().await.map_err(|e| { - RouterError::UpstreamProtocol(format!("failed to read response chunk: {e}")) - }), - StreamingBody::Buffered(buf) => Ok(buf.take()), - } - } -} - -fn sanitize_request_headers( - route: &ResolvedRoute, - headers: &[(String, String)], -) -> Vec<(String, String)> { - let mut allowed = HashSet::new(); - allowed.extend( - COMMON_INFERENCE_REQUEST_HEADERS - .iter() - .map(|name| (*name).to_string()), - ); - allowed.extend( - route - .passthrough_headers - .iter() - .map(|name| name.to_ascii_lowercase()), - ); - allowed.extend( - route - .default_headers - .iter() - .map(|(name, _)| name.to_ascii_lowercase()), - ); - - // Vertex AI Anthropic rawPredict endpoints do not accept the - // `anthropic-beta` header. Beta feature enablement for Vertex AI is - // controlled through Google Cloud, not HTTP headers. Strip it here so - // clients (e.g. Claude Code) that always send beta flags don't cause - // HTTP 400 errors from the Vertex AI backend. - let strip_anthropic_beta = is_vertex_anthropic_rawpredict_route(route); - - headers - .iter() - .filter_map(|(name, value)| { - let name_lc = name.to_ascii_lowercase(); - if should_strip_request_header(&name_lc) || !allowed.contains(&name_lc) { - return None; - } - if strip_anthropic_beta && name_lc == "anthropic-beta" { - return None; - } - Some((name.clone(), value.clone())) - }) - .collect() -} - -fn should_strip_request_header(name: &str) -> bool { - matches!( - name, - "authorization" | "x-api-key" | "host" | "content-length" - ) || is_hop_by_hop_header(name) -} - -fn is_hop_by_hop_header(name: &str) -> bool { - matches!( - name, - "connection" - | "keep-alive" - | "proxy-authenticate" - | "proxy-authorization" - | "proxy-connection" - | "te" - | "trailer" - | "transfer-encoding" - | "upgrade" - ) -} - -/// Build and send an HTTP request to the backend configured in `route`. -/// -/// Returns the prepared [`reqwest::RequestBuilder`] with auth, headers, model -/// rewrite, and body applied. The caller decides whether to apply a total -/// request timeout before sending. -/// -/// `stream_response` controls whether Vertex AI Anthropic routes upgrade the -/// stored `:rawPredict` suffix to `:streamRawPredict` in the upstream URL. -/// It must match the transport the caller intends to use: -/// -/// | Caller | `stream_response` | Vertex suffix used | -/// |-----------------------------|-------------------|------------------------| -/// | `send_backend_request` | `false` | `:rawPredict` (unary) | -/// | `send_backend_request_streaming` | `true` | `:streamRawPredict` | -/// -/// `verify_backend_endpoint` explicitly passes `false` to probe the unary -/// `:rawPredict` endpoint during validation. The `inference.local` intercept -/// path always calls `send_backend_request_streaming` (and therefore always -/// passes `true`), but `:streamRawPredict` accepts both streaming and -/// non-streaming request bodies, so the behaviour is correct in all cases. -fn prepare_backend_request( - client: &reqwest::Client, - route: &ResolvedRoute, - method: &str, - path: &str, - headers: &[(String, String)], - body: bytes::Bytes, - stream_response: bool, -) -> Result<(reqwest::RequestBuilder, String), RouterError> { - // For AWS Bedrock routes the model id is encoded in the URL path - // (`/model/{modelId}/invoke[-with-response-stream]`), not in the - // JSON body. The caller's path can carry any model id; rewrite it - // to the operator-configured `route.model` so a sandbox cannot - // pick a different upstream model than what `inference set` - // configured. If the path is not a recognized Bedrock shape on a - // Bedrock route, reject the request rather than forwarding - // verbatim. - let rewritten_path: String; - let path = if route_is_bedrock(route) { - match rewrite_bedrock_path(route, path) { - Some(p) => { - rewritten_path = p; - rewritten_path.as_str() - } - None => { - return Err(RouterError::Internal(format!( - "AWS Bedrock route received unprocessable path '{path}' or invalid \ - route.model; expected /model//invoke and a model id with no \ - path separators, URL delimiters, percent escapes, traversal \ - segments, whitespace, or control characters" - ))); - } - } - } else { - path - }; - let url = build_provider_url(route, &route.model, path, stream_response); - let headers = sanitize_request_headers(route, headers); - - let reqwest_method: reqwest::Method = method - .parse() - .map_err(|_| RouterError::Internal(format!("invalid HTTP method: {method}")))?; - - let mut builder = client.request(reqwest_method, &url); - - // Inject API key using the route's configured auth mechanism. - match &route.auth { - AuthHeader::Bearer => { - builder = builder.bearer_auth(&route.api_key); - } - AuthHeader::Custom(header_name) => { - builder = builder.header(*header_name, &route.api_key); - } - AuthHeader::None => { - // Bridge-fronted upstream: no router-side auth injection. - // The configured `endpoint` is expected to be a translating - // bridge / proxy whose own pod holds operator-side - // credentials. Used today by the `aws-bedrock` profile - // (SigV4 signing is a separate follow-up). - } - } - for (name, value) in &headers { - builder = builder.header(name.as_str(), value.as_str()); - } - - // Apply route-level default headers (e.g. anthropic-version) unless - // the client already sent them. - for (name, value) in &route.default_headers { - let already_sent = headers.iter().any(|(h, _)| h.eq_ignore_ascii_case(name)); - if !already_sent { - builder = builder.header(name.as_str(), value.as_str()); - } - } - - // Rewrite the JSON body for backend compatibility: - // - Standard routes: set "model" to the route's configured model so the - // backend receives the correct model ID regardless of what the client sent. - // - Vertex AI rawPredict routes: remove "model" (it is encoded in the URL - // path) and inject "anthropic_version" (required in the body, not a header). - // Non-JSON bodies pass through unchanged; model rewrite and version injection - // are silently skipped. Such bodies would be rejected by the upstream anyway. - let body = match serde_json::from_slice::(&body) { - Ok(mut json) => { - if let Some(obj) = json.as_object_mut() { - // Vertex AI Anthropic endpoints require anthropic_version in the body. - // Standard Anthropic SDK sends it as a header; Vertex AI needs it as a body field. - // We inject it only for the Vertex rawPredict-style route contract used for - // Anthropic publisher endpoints, not for arbitrary model-in-path routes. - let needs_vertex_anthropic_version = is_vertex_anthropic_rawpredict_route(route); - if needs_vertex_anthropic_version { - // Vertex AI rawPredict encodes the model in the URL path, not - // the request body. Strip "model" and any Anthropic-SDK-only - // beta fields that Vertex's strict pydantic validation rejects. - obj.remove("model"); - for field in VERTEX_UNSUPPORTED_BODY_FIELDS { - obj.remove(*field); - } - } else if route_is_bedrock(route) { - // AWS Bedrock InvokeModel encodes the model in the URL - // path; the request body is the raw provider-specific - // payload (e.g. an Anthropic Messages body for Claude - // models, a Mistral payload for Mistral models). The - // body must not be mutated — injecting a "model" field - // here would either be silently ignored or rejected as - // an unexpected key by the upstream / bridge. - } else { - obj.insert( - "model".to_string(), - serde_json::Value::String(route.model.clone()), - ); - } - if needs_vertex_anthropic_version && !obj.contains_key("anthropic_version") { - obj.insert( - "anthropic_version".to_string(), - serde_json::Value::String(VERTEX_ANTHROPIC_VERSION.to_string()), - ); - } - } - - bytes::Bytes::from(serde_json::to_vec(&json).map_err(|err| { - RouterError::Internal(format!( - "failed to serialize rewritten inference request body: {err}" - )) - })?) - } - Err(_) => body, - }; - builder = builder.body(body); - - Ok((builder, url)) -} - -/// Send an error-mapped request, shared by both buffered and streaming paths. -fn map_send_error(e: reqwest::Error, url: &str) -> RouterError { - if e.is_timeout() { - RouterError::UpstreamUnavailable(format!("request to {url} timed out")) - } else if e.is_connect() { - RouterError::UpstreamUnavailable(format!("failed to connect to {url}: {e}")) - } else { - RouterError::Internal(format!("HTTP request failed: {e}")) - } -} - -/// Build and send an HTTP request to the backend with a total request timeout. -/// -/// The timeout covers the entire request lifecycle (connect + headers + body). -/// Suitable for non-streaming responses where the body is buffered completely. -async fn send_backend_request( - client: &reqwest::Client, - route: &ResolvedRoute, - method: &str, - path: &str, - headers: &[(String, String)], - body: bytes::Bytes, -) -> Result { - let (builder, url) = - prepare_backend_request(client, route, method, path, headers, body, false)?; - builder - .timeout(route.timeout) - .send() - .await - .map_err(|e| map_send_error(e, &url)) -} - -/// Build and send an HTTP request without a total request timeout. -/// -/// For streaming responses, the total duration is unbounded — liveness is -/// enforced by the caller's per-chunk idle timeout instead. Connection -/// establishment is still bounded by the client-level `connect_timeout`. -async fn send_backend_request_streaming( - client: &reqwest::Client, - route: &ResolvedRoute, - method: &str, - path: &str, - headers: &[(String, String)], - body: bytes::Bytes, -) -> Result { - let (builder, url) = prepare_backend_request(client, route, method, path, headers, body, true)?; - builder.send().await.map_err(|e| map_send_error(e, &url)) -} - -/// Validation probes for a route, in preference order. -/// -/// A managed route advertises every protocol in its provider profile, so an -/// embeddings model resolves to a route that also lists chat/completions. The -/// caller tries these in order and falls through to the next on a request-shape -/// rejection, so such a model validates against `/v1/embeddings` even though -/// the chat probe rejects it. Embeddings is ordered last so a genuinely -/// chat-capable route still validates against chat. Empty when the route -/// exposes no writable protocol. -fn validation_probes(route: &ResolvedRoute) -> Vec { - let has = |protocol: &str| route.protocols.iter().any(|p| p == protocol); - let mut probes = Vec::new(); - - if has("openai_chat_completions") { - // Use max_completion_tokens (modern OpenAI parameter, required by GPT-5+) - // with max_tokens as fallback for legacy/self-hosted backends. - probes.push(ValidationProbe { - path: "/v1/chat/completions", - protocol: "openai_chat_completions", - body: bytes::Bytes::from_static( - br#"{"messages":[{"role":"user","content":"ping"}],"max_completion_tokens":32}"#, - ), - fallback_body: Some(bytes::Bytes::from_static( - br#"{"messages":[{"role":"user","content":"ping"}],"max_tokens":32}"#, - )), - }); - } - - if has("anthropic_messages") { - probes.push(ValidationProbe { - path: "/v1/messages", - protocol: "anthropic_messages", - body: bytes::Bytes::from_static( - br#"{"messages":[{"role":"user","content":"ping"}],"max_tokens":32}"#, - ), - fallback_body: None, - }); - } - - if has("openai_responses") { - probes.push(ValidationProbe { - path: "/v1/responses", - protocol: "openai_responses", - body: bytes::Bytes::from_static(br#"{"input":"ping","max_output_tokens":32}"#), - fallback_body: None, - }); - } - - if has("openai_completions") { - probes.push(ValidationProbe { - path: "/v1/completions", - protocol: "openai_completions", - body: bytes::Bytes::from_static(br#"{"prompt":"ping","max_tokens":32}"#), - fallback_body: None, - }); - } - - // Last so a chat-capable route prefers a chat probe, but an embeddings-only - // model still validates against its single writable endpoint. - if has("openai_embeddings") { - probes.push(ValidationProbe { - path: "/v1/embeddings", - protocol: "openai_embeddings", - body: bytes::Bytes::from_static(br#"{"input":"ping"}"#), - fallback_body: None, - }); - } - - probes -} - -/// The request-shape failure for a route that advertises no writable protocol. -/// -/// Shared by the empty-probe guard and the all-probes-failed fallback so the -/// otherwise-unreachable terminal case is a value rather than a panic. -fn no_writable_protocol_failure(route: &ResolvedRoute) -> ValidationFailure { - ValidationFailure { - kind: ValidationFailureKind::RequestShape, - details: format!( - "route '{}' does not expose a writable inference protocol for validation", - route.name - ), - } -} - -pub async fn verify_backend_endpoint( - client: &reqwest::Client, - route: &ResolvedRoute, -) -> Result { - let probes = validation_probes(route); - let Some(first) = probes.first() else { - return Err(no_writable_protocol_failure(route)); - }; - - if mock::is_mock_route(route) { - return Ok(ValidatedEndpoint { - url: build_provider_url(route, &route.model, first.path, false), - protocol: first.protocol.to_string(), - }); - } - - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - let mut last_shape_failure = None; - - for probe in &probes { - match try_validation_probe(client, route, probe, &headers).await { - Ok(endpoint) => return Ok(endpoint), - // A request-shape rejection means this protocol is wrong for the - // model (e.g. a chat probe against an embeddings model), so fall - // through to the next advertised protocol. Any other failure - // describes the backend itself (credentials, rate limit, - // connectivity, health) and is terminal across all protocols. - // - // Keep the first shape failure: it is the most-preferred protocol's - // rejection and the most actionable error to report. - Err(err) if err.kind == ValidationFailureKind::RequestShape => { - last_shape_failure.get_or_insert(err); - } - Err(err) => return Err(err), - } - } - - Err(last_shape_failure.unwrap_or_else(|| no_writable_protocol_failure(route))) -} - -/// Run one validation probe, retrying with its fallback body only when the -/// upstream specifically rejected `max_completion_tokens`. -/// -/// That retry exists for the GPT-5+ (`max_completion_tokens`) versus legacy -/// (`max_tokens`) chat split. Firing it for any request-shape rejection would -/// issue a second, pointless probe when the real signal is "wrong protocol for -/// this model", and a transient `429`/`5xx` on that retry could become a -/// terminal failure that stops the caller from reaching a protocol that would -/// have validated. -async fn try_validation_probe( - client: &reqwest::Client, - route: &ResolvedRoute, - probe: &ValidationProbe, - headers: &[(String, String)], -) -> Result { - let result = try_validation_request( - client, - route, - probe.path, - probe.protocol, - headers, - probe.body.clone(), - ) - .await; - - if let (Err(err), Some(fallback_body)) = (&result, &probe.fallback_body) - && err.kind == ValidationFailureKind::RequestShape - && err.details.contains("max_completion_tokens") - { - return try_validation_request( - client, - route, - probe.path, - probe.protocol, - headers, - fallback_body.clone(), - ) - .await; - } - - result -} - -/// Send a single validation request and classify the response. -async fn try_validation_request( - client: &reqwest::Client, - route: &ResolvedRoute, - path: &str, - protocol: &str, - headers: &[(String, String)], - body: bytes::Bytes, -) -> Result { - let response = send_backend_request(client, route, "POST", path, headers, body) - .await - .map_err(|err| match err { - RouterError::UpstreamUnavailable(details) => ValidationFailure { - kind: ValidationFailureKind::Connectivity, - details, - }, - RouterError::Internal(details) | RouterError::UpstreamProtocol(details) => { - ValidationFailure { - kind: ValidationFailureKind::Unexpected, - details, - } - } - RouterError::RouteNotFound(details) - | RouterError::NoCompatibleRoute(details) - | RouterError::Unauthorized(details) => ValidationFailure { - kind: ValidationFailureKind::Unexpected, - details, - }, - })?; - let url = build_provider_url(route, &route.model, path, false); - - if response.status().is_success() { - return Ok(ValidatedEndpoint { - url, - protocol: protocol.to_string(), - }); - } - - let status = response.status(); - let body = response.text().await.map_err(|e| ValidationFailure { - kind: ValidationFailureKind::Unexpected, - details: format!("failed to read validation response body: {e}"), - })?; - let body = body.trim(); - let body_suffix = if body.is_empty() { - String::new() - } else { - format!( - " Response body: {}", - body.chars().take(200).collect::() - ) - }; - - // Some OpenAI-compatible providers report an auth failure as 400/404/422 - // with an auth-shaped error body rather than 401/403. Classify those as a - // terminal credential failure so a bad key is not mistaken for a - // wrong-protocol probe and masked by a later probe that accepts it. - let kind = match status.as_u16() { - 401 | 403 => ValidationFailureKind::Credentials, - 400 | 404 | 422 if body_looks_like_auth_error(body) => ValidationFailureKind::Credentials, - 400 | 404 | 405 | 422 => ValidationFailureKind::RequestShape, - 429 => ValidationFailureKind::RateLimited, - 500..=599 => ValidationFailureKind::UpstreamHealth, - _ => ValidationFailureKind::Unexpected, - }; - - let summary = match kind { - ValidationFailureKind::Credentials => "upstream rejected credentials", - ValidationFailureKind::RateLimited => "upstream rate-limited the validation request", - ValidationFailureKind::UpstreamHealth => "upstream returned a server error", - ValidationFailureKind::RequestShape => "upstream rejected the validation request", - _ => "upstream returned an unexpected response", - }; - - Err(ValidationFailure { - kind, - details: format!("{summary} with HTTP {status}.{body_suffix}"), - }) -} - -/// Whether an upstream error body reads as an authentication or authorization -/// failure. Some OpenAI-compatible providers return these as HTTP 400/404/422 -/// rather than 401/403, so validation inspects the body to avoid classifying a -/// bad key as a wrong-protocol probe. Matching is conservative: only strong, -/// auth-specific phrases, lowercased, to avoid catching generic "invalid model" -/// request-shape errors. -fn body_looks_like_auth_error(body: &str) -> bool { - let body = body.to_ascii_lowercase(); - [ - "invalid_api_key", - "invalid api key", - "incorrect api key", - "invalid_authentication", - "authentication_error", - "authentication failed", - "unauthorized", - "permission_denied", - "permission denied", - "missing api key", - ] - .iter() - .any(|needle| body.contains(needle)) -} - -/// Extract status and headers from a [`reqwest::Response`]. -fn extract_response_metadata(response: &reqwest::Response) -> (u16, Vec<(String, String)>) { - let status = response.status().as_u16(); - let headers: Vec<(String, String)> = response - .headers() - .iter() - .map(|(k, v)| (k.to_string(), v.to_str().unwrap_or("").to_string())) - .collect(); - (status, headers) -} - -/// Forward a raw HTTP request to the backend configured in `route`. -/// -/// Buffers the entire response body before returning. Suitable for -/// non-streaming responses or mock routes. -pub async fn proxy_to_backend( - client: &reqwest::Client, - route: &ResolvedRoute, - _source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, -) -> Result { - let response = send_backend_request(client, route, method, path, &headers, body).await?; - let (status, resp_headers) = extract_response_metadata(&response); - let body = read_capped_response_body(response, MAX_BUFFERED_RESPONSE_BODY).await?; - - Ok(ProxyResponse { - status, - headers: resp_headers, - body, - route_model: Some(route.model.clone()), - route_endpoint: Some(route.endpoint.clone()), - }) -} - -/// Read a response body fully into memory, rejecting anything over `max` bytes. -/// -/// Used by the buffered proxy path so a misbehaving upstream cannot force -/// unbounded allocation. The `Content-Length` check is a fast early-out; the -/// chunk loop is the real guard and bounds an absent, chunked, or -/// under-reported length. The cap counts the bytes reqwest yields: with no -/// decompression features enabled (see `Cargo.toml`) those are wire bytes, so -/// enabling a compression feature later would change what the cap measures. -/// Over-cap responses fail as `UpstreamProtocol` and are never partially -/// returned. -async fn read_capped_response_body( - mut response: reqwest::Response, - max: usize, -) -> Result { - if let Some(len) = response.content_length() - && len > max as u64 - { - return Err(RouterError::UpstreamProtocol(format!( - "inference response body of {len} bytes exceeds the {max} byte cap" - ))); - } - - // Preallocate to the advertised length when it is within the cap; the loop - // still enforces the bound for an absent or under-reported length. - let mut body: Vec = match response.content_length() { - Some(len) if len <= max as u64 => Vec::with_capacity(usize::try_from(len).unwrap_or(max)), - _ => Vec::new(), - }; - while let Some(chunk) = response - .chunk() - .await - .map_err(|e| RouterError::UpstreamProtocol(format!("failed to read response body: {e}")))? - { - if body.len() + chunk.len() > max { - return Err(RouterError::UpstreamProtocol(format!( - "inference response body exceeds the {max} byte cap" - ))); - } - body.extend_from_slice(&chunk); - } - Ok(bytes::Bytes::from(body)) -} - -/// Forward a raw HTTP request to the backend, returning response headers -/// immediately without buffering the body. -/// -/// The caller streams the body incrementally via -/// [`StreamingProxyResponse::response`] using `chunk().await`. -pub async fn proxy_to_backend_streaming( - client: &reqwest::Client, - route: &ResolvedRoute, - _source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, -) -> Result { - let response = - send_backend_request_streaming(client, route, method, path, &headers, body).await?; - let (status, resp_headers) = extract_response_metadata(&response); - - Ok(StreamingProxyResponse { - status, - headers: resp_headers, - route_model: Some(route.model.clone()), - route_endpoint: Some(route.endpoint.clone()), - body: StreamingBody::Live(response), - }) -} - -/// Build the upstream URL for a provider route. -/// -/// `stream_response` selects between the unary and streaming Vertex AI -/// Anthropic endpoint suffixes. Pass the same value used for the enclosing -/// [`prepare_backend_request`] call. See that function's documentation for the -/// full caller table. -/// -/// Behavior matrix (`request_path_override`, `model_in_path`): -/// - `(Some(suffix), true)`: `{endpoint}/{model_id}{suffix}` -/// Used by Vertex AI Anthropic: `stream_response=false` keeps `:rawPredict` -/// (unary); `stream_response=true` upgrades to `:streamRawPredict`. -/// - `(Some(override_path), false)`: `{endpoint}{override_path}` -/// Used when a fixed path replaces the protocol-derived path. -/// - `(None, true)`: `{endpoint}/{model_id}/{protocol_path}` -/// Model embedded before protocol path. -/// - `(None, false)`: delegates to `build_backend_url` (default, with /v1 dedup). -fn build_provider_url( - route: &ResolvedRoute, - model_id: &str, - protocol_path: &str, - stream_response: bool, -) -> String { - let base = route.endpoint.trim_end_matches('/'); - match (&route.request_path_override, route.model_in_path) { - // Vertex AI publisher endpoint: model in URL path with suffix - // e.g. .../publishers/anthropic/models/claude-3-5-sonnet@20241022:rawPredict - (Some(suffix), true) => { - // suffix is appended directly after model_id (e.g. ":rawPredict"). - // It must not start with '/' — use the (Some, false) arm for path overrides. - debug_assert!( - !suffix.starts_with('/'), - "suffix in model_in_path branch must not start with '/'; got: {suffix:?}" - ); - let suffix = if stream_response - && suffix == ":rawPredict" - && is_vertex_anthropic_rawpredict_route(route) - { - ":streamRawPredict" - } else { - suffix.as_str() - }; - format!("{base}/{model_id}{suffix}") - } - // Explicit path override, model NOT in URL. - // Normalize: ensure override_path begins with '/' so the concatenation - // never produces a broken URL like `https://host.compath`. - (Some(override_path), false) => { - if override_path.starts_with('/') || override_path.is_empty() { - format!("{base}{override_path}") - } else { - format!("{base}/{override_path}") - } - } - // Model in path, no override — append model then protocol-derived path - (None, true) => { - let path = protocol_path.trim_start_matches('/'); - format!("{base}/{model_id}/{path}") - } - // Default: existing behavior (includes /v1 deduplication) - (None, false) => build_backend_url(&route.endpoint, protocol_path), - } -} - -fn build_backend_url(endpoint: &str, path: &str) -> String { - let base = endpoint.trim_end_matches('/'); - // Strip the /v1 prefix from the request path when the base URL's path - // component has /v1 as its first segment (e.g. openai/nvidia: "/v1", - // deepinfra: "/v1/openai") or its final segment (e.g. groq: - // "/openai/v1"). This covers all known provider shapes while preserving - // the full path for proxy endpoints where /v1 is buried in the middle - // (e.g. "https://proxy.example/api/v1/openai" → path "/api/v1/openai", - // neither first nor last segment). - let base_path_has_v1_edge_segment = base - .find("://") - .and_then(|i| base[i + 3..].find('/').map(|j| i + 3 + j)) - .is_some_and(|path_start| { - let base_path = &base[path_start..]; - base_path.starts_with("/v1/") || base_path.ends_with("/v1") - }); - if base_path_has_v1_edge_segment - && let Some(rest) = path - .strip_prefix("/v1") - .filter(|rest| rest.is_empty() || rest.starts_with('/')) - { - return format!("{base}{rest}"); - } - - format!("{base}{path}") -} - -/// Check whether a route targets an AWS Bedrock `InvokeModel` endpoint. -/// -/// Returns true when any of the route's protocols is one of the Bedrock -/// invocation protocols. Used to gate Bedrock-specific request shaping -/// (path-segment rewriting, skipped body-model injection) in -/// [`prepare_backend_request`]. -/// -/// `aws_bedrock_invoke_stream` is recognized for forward-compatibility -/// with the streaming follow-up but is not currently advertised by the -/// L7 pattern set. -fn route_is_bedrock(route: &ResolvedRoute) -> bool { - route - .protocols - .iter() - .any(|p| p == "aws_bedrock_invoke" || p == "aws_bedrock_invoke_stream") -} - -/// Parse a Bedrock invocation path into its `(model_id, action_suffix, query_tail)` -/// components. -/// -/// Recognized shape (caller's path on the way into the router): -/// - `/model//invoke[?]` → action `/invoke` -/// -/// `` must be non-empty and contain no `/`. The query tail -/// (including the leading `?`) is preserved so [`rewrite_bedrock_path`] -/// can restore it; the L7 matcher accepts queries, so silently dropping -/// them here would mutate the request shape between the matcher and -/// the upstream. Returns `None` when the path does not match — the -/// caller treats that as a malformed request and rejects rather than -/// forwarding verbatim. -/// -/// `InvokeModelWithResponseStream` (`/invoke-with-response-stream`) is -/// deferred until the streaming relay grows protocol-aware AWS -/// event-stream error termination; the L7 pattern set does not -/// advertise it today, so it cannot reach this parser. -fn parse_bedrock_invocation_path(path: &str) -> Option<(&str, &'static str, &str)> { - // Slice up to but not including `?`, then keep the `?`-prefixed - // tail so callers can re-attach it without reconstructing the - // delimiter. - let (path_only, query_tail) = path - .find('?') - .map_or((path, ""), |idx| (&path[..idx], &path[idx..])); - let rest = path_only.strip_prefix("/model/")?; - let slash_at = rest.find('/')?; - if slash_at == 0 { - return None; - } - let model_id = &rest[..slash_at]; - let suffix = &rest[slash_at..]; - let action: &'static str = match suffix { - "/invoke" => "/invoke", - _ => return None, - }; - Some((model_id, action, query_tail)) -} - -/// Rewrite a Bedrock invocation path so the model segment is the -/// operator-configured `route.model` rather than whatever the caller -/// supplied. Returns the rewritten path on success, or `None` when the -/// inbound path is not a recognized Bedrock invocation shape or when -/// `route.model` is not a valid Bedrock model id. -/// -/// Why rewrite rather than reject: the inbound L7 pattern detector -/// already accepts only `/model/{x}/invoke` shapes for Bedrock routes, -/// so a caller-supplied model segment that differs from the -/// operator-configured one is the only case this function changes — -/// and changing it (vs. rejecting) lets sandbox code that hardcodes a -/// different model continue to work, while still guaranteeing the -/// operator's chosen model is what reaches the upstream. -/// -/// Defense-in-depth model-ID validation: the server-side resolver -/// (`openshell-server::inference::resolve_provider_route`) already -/// rejects malformed Bedrock model ids at route-save time, but the -/// router enforces the same contract before interpolating -/// `route.model` into a URL path segment. Values containing `/`, `\`, -/// `?`, `#`, `%`, traversal segments, whitespace, or control chars -/// are rejected so a stale or hand-edited route store cannot produce -/// ambiguous or malformed upstream paths. -fn rewrite_bedrock_path(route: &ResolvedRoute, path: &str) -> Option { - if !is_valid_bedrock_model_id(&route.model) { - return None; - } - let (_caller_model, action, query_tail) = parse_bedrock_invocation_path(path)?; - Some(format!("/model/{}{}{}", route.model, action, query_tail)) -} - -/// Defense-in-depth predicate matching the server-side -/// `validate_aws_bedrock_model_id` contract — see that function for the -/// authoritative reasoning. Returns `true` when `value` is safe to -/// interpolate into a Bedrock URL path segment. The router uses this -/// before constructing an upstream path so a stale or out-of-band route -/// store cannot bypass the resolver's validation. -fn is_valid_bedrock_model_id(value: &str) -> bool { - if value.is_empty() || value != value.trim() { - return false; - } - if value.contains('/') || value.contains('\\') { - return false; - } - if value.chars().any(|c| matches!(c, '?' | '#' | '%')) { - return false; - } - if value.contains("..") { - return false; - } - if value.chars().any(|c| c.is_control() || c.is_whitespace()) { - return false; - } - true -} - -/// Check whether a route targets a Vertex AI Anthropic rawPredict endpoint. -/// -/// The predicate is purely structural — it tests `model_in_path`, -/// `anthropic_messages` protocol, and `:rawPredict` suffix — so any future -/// provider with the same route shape automatically inherits the same -/// transforms without code changes. -/// -/// The router stores the neutral `:rawPredict` suffix on resolved routes. -/// [`build_provider_url`] upgrades it to `:streamRawPredict` when -/// `stream_response=true` (see [`prepare_backend_request`] for the caller -/// table). [`verify_backend_endpoint`] deliberately passes `stream_response=false` -/// to probe the unary endpoint during validation. -fn is_vertex_anthropic_rawpredict_route(route: &ResolvedRoute) -> bool { - route.model_in_path - && route.protocols.iter().any(|p| p == "anthropic_messages") - && route - .request_path_override - .as_deref() - .is_some_and(|suffix| suffix == ":rawPredict") -} - -#[cfg(test)] -mod tests { - use super::{ - ValidationFailure, ValidationFailureKind, build_backend_url, build_provider_url, - parse_bedrock_invocation_path, prepare_backend_request, proxy_to_backend, - rewrite_bedrock_path, route_is_bedrock, verify_backend_endpoint, - }; - use crate::RouterError; - use crate::config::{DEFAULT_ROUTE_TIMEOUT, ResolvedRoute}; - use openshell_core::inference::AuthHeader; - use std::time::Duration; - use wiremock::matchers::{body_partial_json, header, method, path}; - use wiremock::{Mock, MockServer, ResponseTemplate}; - - #[test] - fn build_backend_url_dedupes_v1_prefix() { - assert_eq!( - build_backend_url("https://api.openai.com/v1", "/v1/chat/completions"), - "https://api.openai.com/v1/chat/completions" - ); - } - - #[test] - fn build_backend_url_preserves_non_versioned_base() { - assert_eq!( - build_backend_url("https://api.anthropic.com", "/v1/messages"), - "https://api.anthropic.com/v1/messages" - ); - } - - #[test] - fn build_backend_url_handles_exact_v1_path() { - assert_eq!( - build_backend_url("https://api.openai.com/v1", "/v1"), - "https://api.openai.com/v1" - ); - } - - #[test] - fn build_backend_url_dedupes_v1_for_base_with_v1_subpath() { - // DeepInfra base URL contains /v1/ internally — /v1 in the request - // path must still be stripped so chat/completions is not doubled. - assert_eq!( - build_backend_url( - "https://api.deepinfra.com/v1/openai", - "/v1/chat/completions" - ), - "https://api.deepinfra.com/v1/openai/chat/completions" - ); - } - - #[test] - fn build_backend_url_dedupes_v1_for_base_ending_with_v1() { - // Providers like Groq use a base URL where /v1 is the final segment - // below a non-root prefix (e.g. /openai/v1). The /v1 in the request - // path must still be stripped so it is not doubled. - assert_eq!( - build_backend_url("https://api.groq.com/openai/v1", "/v1/chat/completions"), - "https://api.groq.com/openai/v1/chat/completions" - ); - } - - #[test] - fn build_backend_url_preserves_v1_for_nested_proxy_path() { - // A proxy whose base path has /v1 buried in the middle (neither first - // nor last segment) must NOT have /v1 stripped — the full request - // path must be appended so the upstream receives the correct prefix. - assert_eq!( - build_backend_url( - "https://proxy.example/api/v1/openai", - "/v1/chat/completions" - ), - "https://proxy.example/api/v1/openai/v1/chat/completions" - ); - } - - fn test_route(endpoint: &str, protocols: &[&str], auth: AuthHeader) -> ResolvedRoute { - ResolvedRoute { - name: "inference.local".to_string(), - endpoint: endpoint.to_string(), - model: "test-model".to_string(), - api_key: "sk-test".to_string(), - protocols: protocols.iter().map(|p| (*p).to_string()).collect(), - auth, - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - /// The buffered path must reject an over-cap upstream response rather than - /// buffer it. Guards the DoS/OOM exposure of reading the body unbounded. - #[tokio::test] - async fn proxy_to_backend_rejects_over_cap_response_body() { - use super::{MAX_BUFFERED_RESPONSE_BODY, proxy_to_backend}; - - let mock_server = MockServer::start().await; - // One byte over the cap. wiremock sets an accurate Content-Length, so - // the size check rejects before the body is buffered. - let oversized = vec![b'a'; MAX_BUFFERED_RESPONSE_BODY + 1]; - Mock::given(method("GET")) - .and(path("/v1/models")) - .respond_with(ResponseTemplate::new(200).set_body_bytes(oversized)) - .mount(&mock_server) - .await; - - let route = test_route(&mock_server.uri(), &["model_discovery"], AuthHeader::Bearer); - let client = reqwest::Client::new(); - let result = proxy_to_backend( - &client, - &route, - "model_discovery", - "GET", - "/v1/models", - vec![], - bytes::Bytes::new(), - ) - .await; - - assert!( - matches!(result, Err(RouterError::UpstreamProtocol(_))), - "over-cap response must fail as UpstreamProtocol, got: {result:?}" - ); - } - - /// Spawn a one-shot HTTP/1.1 upstream that replies with a chunked body and - /// no `Content-Length`, so the buffered read cannot pre-check a length and - /// must enforce the cap inside the chunk loop. - async fn spawn_chunked_upstream(chunks: &'static [&'static str]) -> std::net::SocketAddr { - use std::fmt::Write as _; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr = listener.local_addr().unwrap(); - tokio::spawn(async move { - let (mut sock, _) = listener.accept().await.unwrap(); - let mut buf = [0u8; 1024]; - let _ = sock.read(&mut buf).await; - let mut resp = String::from( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\n\r\n", - ); - for c in chunks { - let _ = write!(resp, "{:x}\r\n{c}\r\n", c.len()); - } - resp.push_str("0\r\n\r\n"); - sock.write_all(resp.as_bytes()).await.unwrap(); - }); - addr - } - - /// The chunk-accumulation guard (not the `Content-Length` pre-check) must - /// reject an over-cap body when the response advertises no length. - #[tokio::test] - async fn read_capped_response_body_rejects_over_cap_chunked() { - let addr = spawn_chunked_upstream(&["aaaa", "bbbb", "cccc"]).await; - let response = reqwest::Client::new() - .get(format!("http://{addr}/")) - .send() - .await - .unwrap(); - assert!( - response.content_length().is_none(), - "chunked response should advertise no Content-Length" - ); - let result = super::read_capped_response_body(response, 8).await; - assert!( - matches!(result, Err(RouterError::UpstreamProtocol(_))), - "over-cap chunked body must be rejected by the loop, got: {result:?}" - ); - } - - /// A body exactly at the cap is accepted (inclusive bound) and returned - /// intact through the chunk loop. - #[tokio::test] - async fn read_capped_response_body_accepts_body_at_cap() { - let addr = spawn_chunked_upstream(&["aaaa", "bbbb"]).await; - let response = reqwest::Client::new() - .get(format!("http://{addr}/")) - .send() - .await - .unwrap(); - let body = super::read_capped_response_body(response, 8).await.unwrap(); - assert_eq!(&body[..], b"aaaabbbb"); - } - - #[test] - fn sanitize_request_headers_drops_unknown_sensitive_headers() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.example.com/v1".to_string(), - model: "test-model".to_string(), - api_key: "sk-test".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["openai-organization".to_string()], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let kept = super::sanitize_request_headers( - &route, - &[ - ("content-type".to_string(), "application/json".to_string()), - ("authorization".to_string(), "Bearer client".to_string()), - ("cookie".to_string(), "session=1".to_string()), - ("x-amz-security-token".to_string(), "token".to_string()), - ("openai-organization".to_string(), "org_123".to_string()), - ], - ); - - assert!( - kept.iter() - .any(|(name, _)| name.eq_ignore_ascii_case("content-type")) - ); - assert!( - kept.iter() - .any(|(name, _)| name.eq_ignore_ascii_case("openai-organization")) - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("authorization")) - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("cookie")) - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("x-amz-security-token")) - ); - } - - #[test] - fn sanitize_request_headers_preserves_allowed_provider_headers() { - let route = test_route( - "https://api.anthropic.com/v1", - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - let kept = super::sanitize_request_headers( - &route, - &[ - ("anthropic-version".to_string(), "2024-10-22".to_string()), - ( - "anthropic-beta".to_string(), - "tool-use-2024-10-22".to_string(), - ), - ("x-api-key".to_string(), "client-key".to_string()), - ], - ); - - assert!(kept.iter().any( - |(name, value)| name.eq_ignore_ascii_case("anthropic-version") && value == "2024-10-22" - )); - assert!( - kept.iter() - .any(|(name, value)| name.eq_ignore_ascii_case("anthropic-beta") - && value == "tool-use-2024-10-22") - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("x-api-key")) - ); - } - - #[test] - fn vertex_anthropic_rawpredict_strips_anthropic_beta() { - // Vertex AI rawPredict endpoints reject the anthropic-beta header. - // The router must strip it before forwarding to avoid HTTP 400 errors - // from the Vertex AI backend when clients (e.g. Claude Code) always - // send beta feature flags. - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://us-central1-aiplatform.googleapis.com/v1/projects/proj/locations/us-central1/publishers/anthropic/models".to_string(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - let headers = vec![ - ("content-type".to_string(), "application/json".to_string()), - ( - "anthropic-beta".to_string(), - "prompt-caching-scope-2026-01-05,redact-thinking-2026-02-12".to_string(), - ), - ]; - - let kept = super::sanitize_request_headers(&route, &headers); - - assert!( - kept.iter() - .any(|(name, _)| name.eq_ignore_ascii_case("content-type")), - "content-type should be preserved" - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("anthropic-beta")), - "anthropic-beta must be stripped for Vertex AI rawPredict routes" - ); - } - - #[test] - fn direct_anthropic_preserves_anthropic_beta() { - // The anthropic-beta header must still pass through for direct - // Anthropic API routes -- only Vertex AI rawPredict strips it. - let route = test_route( - "https://api.anthropic.com/v1", - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - let headers = vec![ - ("content-type".to_string(), "application/json".to_string()), - ( - "anthropic-beta".to_string(), - "prompt-caching-2024-07-31".to_string(), - ), - ]; - - let kept = super::sanitize_request_headers(&route, &headers); - - assert!( - kept.iter() - .any(|(name, value)| name.eq_ignore_ascii_case("anthropic-beta") - && value == "prompt-caching-2024-07-31"), - "anthropic-beta must be preserved for direct Anthropic API routes" - ); - } - - #[tokio::test] - async fn verify_backend_endpoint_uses_route_auth_and_shape() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "sk-test")) - .and(header("content-type", "application/json")) - .and(header("anthropic-version", "2023-06-01")) - .and(body_partial_json(serde_json::json!({ - "model": "test-model", - "max_tokens": 32, - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "anthropic_messages"); - assert_eq!(validated.url, format!("{}/v1/messages", mock_server.uri())); - } - - #[tokio::test] - async fn verify_backend_endpoint_accepts_mock_routes() { - let route = test_route( - "mock://test-backend", - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "openai_chat_completions"); - assert_eq!(validated.url, "mock://test-backend/v1/chat/completions"); - } - - /// GPT-5+ models reject `max_tokens` — the primary probe uses - /// `max_completion_tokens` so validation should succeed directly. - #[tokio::test] - async fn verify_openai_chat_uses_max_completion_tokens() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "max_completion_tokens": 32, - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "chatcmpl-1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "openai_chat_completions"); - } - - /// Legacy/self-hosted backends that reject `max_completion_tokens` - /// should succeed on the fallback probe using `max_tokens`. - #[tokio::test] - async fn verify_openai_chat_falls_back_to_max_tokens() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - - // Reject the primary probe (max_completion_tokens) with 400. - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "max_completion_tokens": 32, - }))) - .respond_with(ResponseTemplate::new(400).set_body_string( - r#"{"error":{"message":"Unsupported parameter: 'max_completion_tokens'"}}"#, - )) - .expect(1) - .mount(&mock_server) - .await; - - // Accept the fallback probe (max_tokens). - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "max_tokens": 32, - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "chatcmpl-2"})), - ) - .expect(1) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "openai_chat_completions"); - } - - /// A managed route for an embeddings model advertises the full provider - /// protocol set. The chat probe (tried first) rejects the embeddings model - /// as wrong-shape, so validation must fall through to the embeddings probe - /// rather than fail the route. - #[tokio::test] - async fn verify_embeddings_model_falls_through_chat_probe() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &[ - "openai_chat_completions", - "openai_completions", - "openai_responses", - "openai_embeddings", - "model_discovery", - ], - AuthHeader::Bearer, - ); - - // Chat, completions, and responses probes reject the embedding model. - for chat_path in ["/v1/chat/completions", "/v1/completions", "/v1/responses"] { - Mock::given(method("POST")) - .and(path(chat_path)) - .respond_with( - ResponseTemplate::new(400) - .set_body_string(r#"{"error":{"message":"not a chat model"}}"#), - ) - .mount(&mock_server) - .await; - } - // The embeddings probe accepts it. - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::new(); - let validated = verify_backend_endpoint(&client, &route) - .await - .expect("embeddings model should validate via the embeddings probe"); - assert_eq!(validated.protocol, "openai_embeddings"); - } - - /// A non-request-shape failure (credentials) is terminal: validation must - /// stop at the first probe and not fall through to a protocol that would - /// succeed, so a bad key is reported as such rather than masked. - #[tokio::test] - async fn verify_stops_on_credentials_failure() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(401).set_body_string(r#"{"error":"bad key"}"#)) - .mount(&mock_server) - .await; - // Would succeed, but credentials failure on the first probe is terminal - // and this must never be reached. - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::new(); - let err = verify_backend_endpoint(&client, &route) - .await - .expect_err("a 401 must fail validation"); - assert_eq!(err.kind, ValidationFailureKind::Credentials); - } - - /// A 429 on the first probe is terminal (`RateLimited`) and must not fall - /// through to a later probe that would succeed. - #[tokio::test] - async fn verify_stops_on_rate_limit() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(429).set_body_string(r#"{"error":"slow down"}"#)) - .mount(&mock_server) - .await; - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let err = reqwest_verify(&route).await; - assert_eq!(err.kind, ValidationFailureKind::RateLimited); - } - - /// An auth failure reported as HTTP 400 with an auth-shaped body is terminal - /// (`Credentials`), not a request-shape fall-through, so a bad key cannot be - /// masked by a later probe that accepts it. - #[tokio::test] - async fn verify_auth_error_as_400_is_terminal() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(400).set_body_string( - r#"{"error":{"code":"invalid_api_key","message":"Incorrect API key provided"}}"#, - )) - .mount(&mock_server) - .await; - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let err = reqwest_verify(&route).await; - assert_eq!(err.kind, ValidationFailureKind::Credentials); - } - - /// When every probe is rejected as request-shape, validation returns the - /// first (most-preferred protocol's) failure, not the last. - #[tokio::test] - async fn verify_all_probes_request_shape_returns_first() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with( - ResponseTemplate::new(404).set_body_string(r#"{"error":"model not found: chat"}"#), - ) - .mount(&mock_server) - .await; - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(400) - .set_body_string(r#"{"error":"not an embeddings model"}"#), - ) - .mount(&mock_server) - .await; - - let err = reqwest_verify(&route).await; - assert_eq!(err.kind, ValidationFailureKind::RequestShape); - assert!( - err.details.contains("model not found: chat"), - "should report the first (chat) failure, got: {}", - err.details - ); - } - - /// Helper: run `verify_backend_endpoint` and return the expected failure. - async fn reqwest_verify(route: &ResolvedRoute) -> ValidationFailure { - verify_backend_endpoint(&reqwest::Client::new(), route) - .await - .expect_err("validation should fail") - } - - /// Non-chat-completions probes (e.g. `anthropic_messages`) should not - /// have a fallback — a 400 remains a hard failure. - #[tokio::test] - async fn verify_non_chat_completions_no_fallback() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .respond_with(ResponseTemplate::new(400).set_body_string("bad request")) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let result = verify_backend_endpoint(&client, &route).await; - - assert!(result.is_err()); - assert_eq!( - result.unwrap_err().kind, - ValidationFailureKind::RequestShape - ); - } - - #[tokio::test] - async fn verify_vertex_anthropic_route_uses_buffered_rawpredict_probe() { - let mock_server = MockServer::start().await; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!( - "{}/v1/projects/my-project/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path( - "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet@20241022:rawPredict", - )) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_vertex_verify" - }))) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - assert!( - validated.url.ends_with(":rawPredict"), - "buffered verification should probe the unary Vertex endpoint, got: {}", - validated.url - ); - } - - /// Vertex AI pattern: `model_in_path=true`, `request_path_override=Some(":rawPredict")` - /// means buffered requests POST to `base_url/model_id:rawPredict`. - #[test] - fn build_provider_url_model_in_path_with_suffix() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: - "https://us-east5-aiplatform.googleapis.com/v1/projects/my-project/locations/us-east5/publishers/anthropic/models" - .to_string(), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - let url = build_provider_url(&route, "claude-3-5-sonnet@20241022", "/v1/messages", false); - assert!( - url.ends_with("/claude-3-5-sonnet@20241022:rawPredict"), - "expected URL to end with model id and suffix, got: {url}" - ); - assert!( - !url.contains("/v1/messages"), - "expected no protocol path appended, got: {url}" - ); - } - - #[test] - fn build_provider_url_vertex_anthropic_streaming_upgrades_to_stream_rawpredict() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: - "https://us-east5-aiplatform.googleapis.com/v1/projects/my-project/locations/us-east5/publishers/anthropic/models" - .to_string(), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - let url = build_provider_url(&route, "claude-3-5-sonnet@20241022", "/v1/messages", true); - assert!( - url.ends_with("/claude-3-5-sonnet@20241022:streamRawPredict"), - "expected streaming URL to upgrade the suffix, got: {url}" - ); - } - - /// Vertex AI pattern: `model_in_path=true`, `request_path_override=Some("")` (empty suffix) - /// means POST directly to `base_url/model_id` with no additional path segment. - #[test] - fn build_provider_url_model_in_path_empty_suffix() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://example.com/models".to_string(), - model: "my-model".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(String::new()), - }; - - let url = build_provider_url(&route, "my-model", "/v1/messages", false); - assert_eq!(url, "https://example.com/models/my-model"); - } - - /// Explicit path override: `request_path_override=Some("/v1/chat/completions")` - /// appends the override path to `base_url`, ignoring `model_in_path`. - #[test] - fn build_provider_url_with_path_override() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.example.com".to_string(), - model: "some-model".to_string(), - api_key: "key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("/v1/chat/completions".to_string()), - }; - - let url = build_provider_url(&route, "some-model", "/v1/chat/completions", false); - assert!( - url.ends_with("/v1/chat/completions"), - "expected URL to end with path override, got: {url}" - ); - } - - /// Default behavior: `model_in_path=false`, `request_path_override=None` uses - /// the existing `build_backend_url` logic (protocol-derived path only). - #[test] - fn build_provider_url_default_behavior() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.openai.com/v1".to_string(), - model: "gpt-4o".to_string(), - api_key: "key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let url = build_provider_url(&route, "gpt-4o", "/v1/chat/completions", false); - assert_eq!( - url, "https://api.openai.com/v1/chat/completions", - "default behavior should dedupe v1 prefix and use protocol path" - ); - } - - #[test] - fn build_provider_url_override_path_normalizes_missing_leading_slash() { - // An override_path without a leading '/' must not produce a broken URL. - let route = ResolvedRoute { - name: "test".to_string(), - endpoint: "https://example.com/v1/projects/proj/locations/us/endpoints/openapi" - .to_string(), - model: "gemini-pro".to_string(), - api_key: "key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("chat/completions".to_string()), // no leading slash - }; - let url = build_provider_url(&route, &route.model, "/v1/chat/completions", false); - // Must not produce https://...openaichat/completions - assert!( - url.contains("/chat/completions"), - "URL must contain /chat/completions, got: {url}" - ); - assert!( - !url.contains("openaichat"), - "URL must not smash endpoint and path, got: {url}" - ); - assert_eq!( - url, - "https://example.com/v1/projects/proj/locations/us/endpoints/openapi/chat/completions" - ); - } - - /// Vertex AI Anthropic routes require `anthropic_version` in the request body. - /// Verify it is injected on the buffered `:rawPredict` path when the client - /// did not already include it. - #[tokio::test] - async fn vertex_ai_body_injects_anthropic_version() { - let mock_server = MockServer::start().await; - - // Build a Vertex-AI-style route: model in path, suffix :rawPredict - let base_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!("{}{base_path}", mock_server.uri()), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path(format!( - "{base_path}/claude-3-5-sonnet@20241022:rawPredict" - ))) - .and(body_partial_json(serde_json::json!({ - "anthropic_version": "vertex-2023-10-16", - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_vertex_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let response = builder.send().await.unwrap(); - assert_eq!( - response.status().as_u16(), - 200, - "mock should match body with anthropic_version injected" - ); - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body.as_object().unwrap().contains_key("model"), - "Vertex Anthropic route must not inject model into the body, got: {received_body}" - ); - } - - /// Claude Code and other Anthropic SDK clients always send "model" in the - /// request body. For Vertex AI rawPredict routes the model is in the URL - /// path; the body field must be stripped to avoid HTTP 400 - /// "Extra inputs are not permitted" from the Vertex AI backend. - #[tokio::test] - async fn vertex_ai_body_strips_client_model_field() { - let mock_server = MockServer::start().await; - - let base_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!("{}{base_path}", mock_server.uri()), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path(format!( - "{base_path}/claude-3-5-sonnet@20241022:rawPredict" - ))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - // Simulate a client (e.g. Claude Code) that always sends "model" in the body. - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "model": "claude-3-5-sonnet-20241022", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let response = builder.send().await.unwrap(); - assert_eq!(response.status().as_u16(), 200); - let received = mock_server.received_requests().await.unwrap(); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body.as_object().unwrap().contains_key("model"), - "model field must be stripped from Vertex AI rawPredict body, got: {received_body}" - ); - } - - #[tokio::test] - async fn vertex_ai_body_strips_unsupported_beta_fields() { - let mock_server = MockServer::start().await; - - let base_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!("{}{base_path}", mock_server.uri()), - model: "claude-sonnet-4-6@20250514".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path(format!( - "{base_path}/claude-sonnet-4-6@20250514:rawPredict" - ))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-6-20250514", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "context_management": {"enabled": true}, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let response = builder.send().await.unwrap(); - assert_eq!(response.status().as_u16(), 200); - let received = mock_server.received_requests().await.unwrap(); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - let obj = received_body.as_object().unwrap(); - assert!( - !obj.contains_key("context_management"), - "context_management must be stripped for Vertex AI rawPredict, got: {received_body}" - ); - assert!( - !obj.contains_key("model"), - "model must also be stripped for Vertex AI rawPredict, got: {received_body}" - ); - assert!( - obj.contains_key("messages"), - "standard fields must be preserved, got: {received_body}" - ); - } - - #[tokio::test] - async fn direct_anthropic_preserves_beta_fields() { - let route = ResolvedRoute { - name: "direct-anthropic".to_string(), - endpoint: "https://api.anthropic.com/v1".to_string(), - model: "claude-sonnet-4-6-20250514".to_string(), - api_key: "sk-test".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-6-20250514", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "context_management": {"enabled": true}, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let request = builder.build().unwrap(); - let sent_body: serde_json::Value = - serde_json::from_slice(request.body().unwrap().as_bytes().unwrap()).unwrap(); - assert!( - sent_body - .as_object() - .unwrap() - .contains_key("context_management"), - "context_management must be preserved for direct Anthropic API routes" - ); - } - - #[tokio::test] - async fn vertex_ai_body_preserves_client_anthropic_version() { - // When the client already sends anthropic_version, the router must NOT overwrite it. - let mock_server = MockServer::start().await; - - // Expect the body to contain the client's version, NOT "vertex-2023-10-16" - Mock::given(method("POST")) - .and(body_partial_json(serde_json::json!({ - "anthropic_version": "custom-client-version", - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_1", - "type": "message", - "role": "assistant", - "model": "claude-3-5-sonnet@20241022", - "content": [{"type": "text", "text": "ok"}] - }))) - .mount(&mock_server) - .await; - - let router = crate::Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "vertex-test".to_string(), - endpoint: format!( - "{}/v1/projects/proj/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.test".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "anthropic_version": "custom-client-version", - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!( - response.status, 200, - "proxy should succeed when client sends anthropic_version" - ); - } - - /// Standard Anthropic route (`model_in_path=false`) must NOT inject `anthropic_version`. - /// Vertex body injection must not affect non-Vertex Anthropic providers. - #[tokio::test] - async fn standard_anthropic_body_does_not_inject_vertex_anthropic_version() { - let mock_server = MockServer::start().await; - - let route = ResolvedRoute { - name: "anthropic-direct".to_string(), - endpoint: mock_server.uri(), - model: "claude-3-5-sonnet-20241022".to_string(), - api_key: "sk-ant-test".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - builder.send().await.unwrap(); - - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body - .as_object() - .unwrap() - .contains_key("anthropic_version"), - "standard Anthropic route must not inject anthropic_version, got: {received_body}" - ); - } - - /// Model-in-path alone is not enough; only Vertex rawPredict-style routes should inject. - #[tokio::test] - async fn anthropic_model_in_path_without_rawpredict_suffix_does_not_inject_version() { - let mock_server = MockServer::start().await; - - let route = ResolvedRoute { - name: "non-vertex-model-path".to_string(), - endpoint: format!("{}/publisher/models", mock_server.uri()), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(String::new()), - }; - - Mock::given(method("POST")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"id": "msg_model_path"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - builder.send().await.unwrap(); - - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body - .as_object() - .unwrap() - .contains_key("anthropic_version"), - "non-rawPredict model-in-path routes must not inject anthropic_version, got: {received_body}" - ); - } - - /// Vertex AI Gemini route (`model_in_path=false`, `openai_chat_completions`) must NOT inject. - #[tokio::test] - async fn vertex_gemini_body_does_not_inject_vertex_anthropic_version() { - let mock_server = MockServer::start().await; - - let route = ResolvedRoute { - name: "vertex-gemini".to_string(), - endpoint: format!( - "{}/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi", - mock_server.uri() - ), - model: "gemini-pro".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - Mock::given(method("POST")) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_gemini"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/chat/completions", - &headers, - body, - false, - ) - .unwrap(); - - builder.send().await.unwrap(); - - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body - .as_object() - .unwrap() - .contains_key("anthropic_version"), - "Vertex Gemini route must not inject anthropic_version, got: {received_body}" - ); - assert_eq!( - received_body - .as_object() - .unwrap() - .get("model") - .and_then(serde_json::Value::as_str), - Some("gemini-pro"), - "Vertex Gemini route must still rewrite the model field, got: {received_body}" - ); - } - - // ============================================================ - // AWS Bedrock route shaping (path rewriting + body preservation) - // ============================================================ - - /// `parse_bedrock_invocation_path` rejects malformed paths. - #[test] - fn parse_bedrock_invocation_path_rejects_malformed() { - // Empty model id: `/model//invoke` - assert!(parse_bedrock_invocation_path("/model//invoke").is_none()); - // Multi-segment model id: `/model/a/b/invoke` - assert!(parse_bedrock_invocation_path("/model/a/b/invoke").is_none()); - // Unknown action: `/model/foo/converse` - assert!(parse_bedrock_invocation_path("/model/foo/converse").is_none()); - // Streaming variant is deferred until protocol-aware error - // framing exists; the parser must reject it the same way it - // rejects any other unknown action. - assert!(parse_bedrock_invocation_path("/model/foo/invoke-with-response-stream").is_none()); - // Wrong prefix: `/v1/messages` - assert!(parse_bedrock_invocation_path("/v1/messages").is_none()); - // Missing slash before action - assert!(parse_bedrock_invocation_path("/model/foo").is_none()); - } - - #[test] - fn parse_bedrock_invocation_path_accepts_invoke() { - let parsed = parse_bedrock_invocation_path( - "/model/anthropic.claude-3-5-sonnet-20241022-v2:0/invoke", - ); - assert_eq!( - parsed, - Some(("anthropic.claude-3-5-sonnet-20241022-v2:0", "/invoke", "")) - ); - } - - /// Query strings on Bedrock invoke paths are preserved through the - /// rewrite so the matcher (which accepts queries) and the upstream - /// see the same shape. - #[test] - fn parse_bedrock_invocation_path_preserves_query_string() { - let parsed = - parse_bedrock_invocation_path("/model/anthropic.claude-opus-4-7/invoke?trace=1"); - assert_eq!( - parsed, - Some(("anthropic.claude-opus-4-7", "/invoke", "?trace=1")) - ); - } - - /// `route_is_bedrock` matches the Bedrock invocation protocol(s). - /// `aws_bedrock_invoke_stream` is recognized for forward-compatibility - /// even though no L7 pattern advertises it today. - #[test] - fn route_is_bedrock_matches_invoke_protocols() { - let invoke_only = test_route( - "https://example.com", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - assert!(route_is_bedrock(&invoke_only)); - - let stream_forward_compat = test_route( - "https://example.com", - &["aws_bedrock_invoke_stream"], - AuthHeader::None, - ); - assert!(route_is_bedrock(&stream_forward_compat)); - - let openai = test_route( - "https://example.com", - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - assert!(!route_is_bedrock(&openai)); - } - - /// `rewrite_bedrock_path` swaps caller's model segment for the - /// route-configured model and preserves any query string. - #[test] - fn rewrite_bedrock_path_substitutes_operator_model() { - let mut route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - route.model = "anthropic.claude-opus-4-7".to_string(); - - let rewritten = rewrite_bedrock_path(&route, "/model/some-other-model/invoke"); - assert_eq!( - rewritten, - Some("/model/anthropic.claude-opus-4-7/invoke".to_string()) - ); - - let rewritten_with_query = - rewrite_bedrock_path(&route, "/model/some-other-model/invoke?trace=1"); - assert_eq!( - rewritten_with_query, - Some("/model/anthropic.claude-opus-4-7/invoke?trace=1".to_string()) - ); - } - - #[test] - fn rewrite_bedrock_path_returns_none_for_non_bedrock_path() { - let route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - assert_eq!(rewrite_bedrock_path(&route, "/v1/messages"), None); - assert_eq!(rewrite_bedrock_path(&route, "/model//invoke"), None); - assert_eq!(rewrite_bedrock_path(&route, "/model/a/b/invoke"), None); - // Streaming variant is deferred at the L7 layer; the router - // must not produce an upstream path for it either. - assert_eq!( - rewrite_bedrock_path(&route, "/model/x/invoke-with-response-stream"), - None - ); - } - - /// Defense-in-depth: `rewrite_bedrock_path` rejects route models - /// that would produce ambiguous or malformed upstream URL paths, - /// even if a malformed value somehow reached the router store. - #[test] - fn rewrite_bedrock_path_rejects_unsafe_route_model() { - let mut route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - - for unsafe_model in [ - "anthropic.claude/../../etc/passwd", - "anthropic.claude\\backslash", - "model?injected=1", - "model#fragment", - "percent%2fencoded", - "..", - " leading-space", - "trailing-space ", - "tab\there", - "newline\nhere", - "", - ] { - route.model = unsafe_model.to_string(); - assert!( - rewrite_bedrock_path(&route, "/model/foo/invoke").is_none(), - "rewrite_bedrock_path must reject unsafe route.model: {unsafe_model:?}" - ); - } - } - - /// End-to-end: an inbound Bedrock request that names a different - /// model in the path arrives at the upstream/bridge with the - /// operator's model, and the body is unchanged (no `"model"` - /// injection). - #[tokio::test] - async fn bedrock_route_rewrites_model_in_path_and_preserves_body() { - let mock_server = MockServer::start().await; - let mut route = test_route( - &mock_server.uri(), - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - route.model = "anthropic.claude-opus-4-7".to_string(); - - // The mock asserts the upstream sees the operator's model in - // the path, NOT the caller's model. - Mock::given(method("POST")) - .and(path("/model/anthropic.claude-opus-4-7/invoke")) - // Caller body has a "model" key; we expect it to pass - // through unchanged. The mock uses body_partial_json so - // additional fields are OK; the assertion below pins the - // body more tightly. - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({"ok": true}))) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder() - .timeout(Duration::from_secs(5)) - .build() - .expect("client"); - - // Caller-supplied body — we deliberately include a "model" - // field naming a DIFFERENT model than the operator's, to - // verify the router does not inject route.model on top of - // it. The body should pass through verbatim because Bedrock - // encodes the model in the path. - let caller_body = serde_json::json!({ - "model": "caller-supplied-model-name", - "messages": [{"role": "user", "content": "hi"}], - }); - - let (builder, url) = prepare_backend_request( - &client, - &route, - "POST", - "/model/some-other-model/invoke", - &[], - bytes::Bytes::from(caller_body.to_string()), - false, - ) - .expect("prepare should succeed"); - - // URL should target the operator's model, not the caller's. - assert!( - url.ends_with("/model/anthropic.claude-opus-4-7/invoke"), - "URL must use operator model, got: {url}" - ); - - let resp = builder.send().await.expect("send"); - assert_eq!(resp.status(), 200); - - // Inspect what wiremock actually received. - let received = mock_server.received_requests().await.expect("requests"); - assert_eq!(received.len(), 1); - let req = &received[0]; - let received_body: serde_json::Value = - serde_json::from_slice(&req.body).expect("json body"); - // Caller's model name should pass through (NOT replaced by - // route.model). This proves the body is untouched. - assert_eq!( - received_body.get("model").and_then(|v| v.as_str()), - Some("caller-supplied-model-name"), - "Bedrock route must NOT rewrite body model, got: {received_body}" - ); - assert!( - received_body.get("messages").is_some(), - "messages field should pass through unchanged" - ); - } - - /// Vertex AI's OpenAI-compatible endpoint requires the body `model` field to - /// carry a publisher prefix (e.g. `google/gemini-2.5-flash`). This test - /// simulates the fix: `resolve_vertex_ai_route` sets `route.model` to the - /// prefixed form, and the body rewrite here forwards that value to Vertex. - /// - /// The mock server only accepts the prefixed form — matching Vertex's - /// behaviour — and returns 400 "Malformed publisher model" for the bare name. - #[tokio::test] - async fn vertex_openai_compat_rewrites_body_model_to_publisher_prefixed_form() { - let mock_server = MockServer::start().await; - - // Simulate Vertex accepting only the publisher-prefixed model name. - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_partial_json( - serde_json::json!({"model": "google/gemini-2.5-flash"}), - )) - .respond_with( - ResponseTemplate::new(200).set_body_json( - serde_json::json!({"choices": [{"message": {"content": "hi"}}]}), - ), - ) - .expect(1) - .mount(&mock_server) - .await; - - // Simulate Vertex rejecting the bare model name — the pre-fix failure. - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_partial_json( - serde_json::json!({"model": "gemini-2.5-flash"}), - )) - .respond_with(ResponseTemplate::new(400).set_body_json( - serde_json::json!({"error": {"message": "Malformed publisher model"}}), - )) - .expect(0) // must never be reached after the fix - .mount(&mock_server) - .await; - - // Route as produced by resolve_vertex_ai_route after the fix: - // route.model carries the publisher prefix. - let route = ResolvedRoute { - name: "vertex-gemini".to_string(), - endpoint: mock_server.uri(), - model: "google/gemini-2.5-flash".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("/chat/completions".to_string()), - }; - - // The client sends the bare model name; the body rewrite must replace it - // with route.model (the publisher-prefixed form) before forwarding. - let client_body = serde_json::to_vec(&serde_json::json!({ - "model": "gemini-2.5-flash", - "messages": [{"role": "user", "content": "hello"}] - })) - .unwrap(); - - let client = reqwest::Client::new(); - let result = proxy_to_backend( - &client, - &route, - "openai_chat_completions", - "POST", - "/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(client_body), - ) - .await - .expect("proxy should succeed"); - - assert_eq!( - result.status, 200, - "Vertex mock must accept the publisher-prefixed model; \ - got {}: body rewrite did not apply the prefix", - result.status - ); - } - - /// Defense-in-depth: a Bedrock route receiving a non-Bedrock path - /// is rejected rather than forwarded. The L7 pattern detector - /// upstream of the router should never produce this combination, - /// but if it ever did, we must not silently forward. - #[test] - fn bedrock_route_rejects_non_bedrock_path() { - let client = reqwest::Client::new(); - let route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - let result = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &[], - bytes::Bytes::from(r"{}"), - false, - ); - match result { - Err(RouterError::Internal(msg)) => { - assert!( - msg.contains("Bedrock") && msg.contains("/v1/messages"), - "error must name the offending path, got: {msg}" - ); - } - other => panic!("expected RouterError::Internal, got {other:?}"), - } - } -} diff --git a/crates/openshell-router/src/config.rs b/crates/openshell-router/src/config.rs deleted file mode 100644 index 81fac60487..0000000000 --- a/crates/openshell-router/src/config.rs +++ /dev/null @@ -1,348 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use serde::Deserialize; -use std::path::Path; -use std::time::Duration; - -pub use openshell_core::inference::AuthHeader; - -use crate::RouterError; - -pub const DEFAULT_ROUTE_TIMEOUT: Duration = Duration::from_secs(60); - -#[derive(Debug, Clone, Deserialize)] -pub struct RouterConfig { - pub routes: Vec, -} - -#[derive(Debug, Clone, Deserialize)] -pub struct RouteConfig { - pub name: String, - pub endpoint: String, - pub model: String, - #[serde(default)] - pub provider_type: Option, - #[serde(default)] - pub protocols: Vec, - #[serde(default)] - pub api_key: Option, - #[serde(default)] - pub api_key_env: Option, -} - -/// A fully-resolved route ready for the router to forward requests. -/// -/// The router is provider-agnostic — all provider-specific decisions -/// (auth header style, default headers, passthrough headers, base URL) are made by the -/// caller during resolution. -#[derive(Clone)] -pub struct ResolvedRoute { - /// Route name used for identification (e.g. "inference.local", "sandbox-system"). - pub name: String, - pub endpoint: String, - pub model: String, - pub api_key: String, - pub protocols: Vec, - /// How to inject the API key on outgoing requests. - pub auth: AuthHeader, - /// Extra headers injected on every request (e.g. `anthropic-version`). - pub default_headers: Vec<(String, String)>, - /// Client-supplied headers that may be forwarded to the upstream backend. - pub passthrough_headers: Vec, - /// Per-request timeout for proxied inference calls. - pub timeout: Duration, - /// When true, the model identifier is embedded in the URL path (e.g. Vertex AI). - pub model_in_path: bool, - /// Optional override for the request path. When set, replaces the protocol-derived path. - /// An empty string means POST directly to `base_url/model_id` with no additional path. - pub request_path_override: Option, -} - -impl std::fmt::Debug for ResolvedRoute { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("ResolvedRoute") - .field("name", &self.name) - .field("endpoint", &self.endpoint) - .field("model", &self.model) - .field("api_key", &"[REDACTED]") - .field("protocols", &self.protocols) - .field("auth", &self.auth) - .field("default_headers", &self.default_headers) - .field("passthrough_headers", &self.passthrough_headers) - .field("timeout", &self.timeout) - .field("model_in_path", &self.model_in_path) - .field("request_path_override", &self.request_path_override) - .finish() - } -} - -impl RouterConfig { - pub fn load_from_file(path: &Path) -> Result { - let content = std::fs::read_to_string(path).map_err(|e| { - RouterError::Internal(format!( - "failed to read router config {}: {e}", - path.display() - )) - })?; - let config: Self = serde_yml::from_str(&content).map_err(|e| { - RouterError::Internal(format!( - "failed to parse router config {}: {e}", - path.display() - )) - })?; - config.resolve() - } - - fn resolve(self) -> Result { - // Validate that all routes can resolve their API keys - for route in &self.routes { - route.resolve_api_key()?; - } - Ok(self) - } - - pub fn resolve_routes(&self) -> Result, RouterError> { - self.routes.iter().map(RouteConfig::resolve).collect() - } -} - -impl RouteConfig { - fn resolve_api_key(&self) -> Result { - if let Some(key) = &self.api_key { - return Ok(key.clone()); - } - if let Some(env_var) = &self.api_key_env { - return std::env::var(env_var).map_err(|_| { - RouterError::Internal(format!( - "environment variable {env_var} not set for route '{}'", - self.name - )) - }); - } - Err(RouterError::Internal(format!( - "route '{}' has neither api_key nor api_key_env", - self.name - ))) - } - - fn resolve(&self) -> Result { - let protocols = openshell_core::inference::normalize_protocols(&self.protocols); - if protocols.is_empty() { - return Err(RouterError::Internal(format!( - "route '{}' has no protocols", - self.name - ))); - } - - let (auth, default_headers, passthrough_headers) = - route_headers_from_provider_type(self.provider_type.as_deref(), &protocols); - - Ok(ResolvedRoute { - name: self.name.clone(), - endpoint: self.endpoint.clone(), - model: self.model.clone(), - api_key: self.resolve_api_key()?, - protocols, - auth, - default_headers, - passthrough_headers, - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }) - } -} - -/// Derive auth header style, default headers, and passthrough headers from a -/// provider type string. -/// -/// Delegates to [`openshell_core::inference::route_headers_for_provider_type`] -/// which uses the centralized `InferenceProviderProfile` registry. -fn route_headers_from_provider_type( - provider_type: Option<&str>, - protocols: &[String], -) -> (AuthHeader, Vec<(String, String)>, Vec) { - openshell_core::inference::route_headers_for_route(provider_type.unwrap_or(""), protocols) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::io::Write; - - #[test] - fn load_from_file_valid_yaml_round_trip() { - let yaml = r" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] - api_key: sk-test-key - - name: inference.local - endpoint: https://api.openai.com/v1 - model: gpt-4o - protocols: [openai_chat_completions, anthropic_messages] - api_key: sk-prod-key -"; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - - let config = RouterConfig::load_from_file(f.path()).unwrap(); - assert_eq!(config.routes.len(), 2); - assert_eq!(config.routes[0].name, "inference.local"); - assert_eq!(config.routes[1].name, "inference.local"); - - let resolved = config.resolve_routes().unwrap(); - assert_eq!(resolved.len(), 2); - assert_eq!(resolved[0].api_key, "sk-test-key"); - assert_eq!(resolved[1].model, "gpt-4o"); - assert_eq!( - resolved[1].protocols, - vec!["openai_chat_completions", "anthropic_messages"] - ); - } - - #[test] - fn load_from_file_invalid_yaml_returns_error() { - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(b"not: [valid: yaml: {{{{").unwrap(); - - let err = RouterConfig::load_from_file(f.path()).unwrap_err(); - assert!( - matches!(err, RouterError::Internal(_)), - "expected Internal error, got: {err:?}" - ); - } - - #[test] - fn load_from_file_missing_api_key_returns_error() { - let yaml = r" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] -"; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - - let err = RouterConfig::load_from_file(f.path()).unwrap_err(); - assert!( - matches!(err, RouterError::Internal(_)), - "expected Internal error for missing api_key, got: {err:?}" - ); - let msg = err.to_string(); - assert!( - msg.contains("neither api_key nor api_key_env"), - "error should mention missing key: {msg}" - ); - } - - #[test] - #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 - fn load_from_file_api_key_env_resolves_from_environment() { - let yaml = r" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] - api_key_env: NAV_TEST_API_KEY_FOR_YAML_TEST -"; - // SAFETY: this test runs single-threaded; no other thread reads this var. - unsafe { std::env::set_var("NAV_TEST_API_KEY_FOR_YAML_TEST", "from-env") }; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - - let config = RouterConfig::load_from_file(f.path()).unwrap(); - let resolved = config.resolve_routes().unwrap(); - assert_eq!(resolved[0].api_key, "from-env"); - - unsafe { std::env::remove_var("NAV_TEST_API_KEY_FOR_YAML_TEST") }; - } - - #[test] - fn load_from_file_nonexistent_path_returns_error() { - let err = RouterConfig::load_from_file(Path::new("/nonexistent/routes.yaml")).unwrap_err(); - assert!( - matches!(err, RouterError::Internal(_)), - "expected Internal error, got: {err:?}" - ); - } - - #[test] - fn resolved_route_debug_redacts_api_key() { - let route = ResolvedRoute { - name: "test".to_string(), - endpoint: "https://api.example.com/v1".to_string(), - model: "test-model".to_string(), - api_key: "sk-super-secret-key-12345".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - let debug_output = format!("{route:?}"); - assert!( - !debug_output.contains("sk-super-secret-key-12345"), - "Debug output must not contain raw API key: {debug_output}" - ); - assert!( - debug_output.contains("[REDACTED]"), - "Debug output should show [REDACTED] for api_key: {debug_output}" - ); - } - - #[test] - fn auth_from_anthropic_provider_uses_custom_header() { - let (auth, headers, passthrough_headers) = route_headers_from_provider_type( - Some("anthropic"), - &["anthropic_messages".to_string()], - ); - assert_eq!(auth, AuthHeader::Custom("x-api-key")); - assert!(headers.iter().any(|(k, _)| k == "anthropic-version")); - assert!( - passthrough_headers - .iter() - .any(|name| name == "anthropic-beta") - ); - } - - #[test] - fn auth_from_openai_provider_uses_bearer() { - let (auth, headers, passthrough_headers) = route_headers_from_provider_type( - Some("openai"), - &["openai_chat_completions".to_string()], - ); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - assert!( - passthrough_headers - .iter() - .any(|name| name == "openai-organization") - ); - } - - #[test] - fn auth_from_none_defaults_to_bearer() { - let (auth, headers, passthrough_headers) = route_headers_from_provider_type(None, &[]); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - assert!(passthrough_headers.is_empty()); - } - - #[test] - fn vertex_anthropic_route_forwards_beta_only() { - let (_, headers, passthrough_headers) = route_headers_from_provider_type( - Some("google-vertex-ai"), - &["anthropic_messages".to_string()], - ); - assert!(headers.is_empty()); - assert_eq!(passthrough_headers, vec!["anthropic-beta".to_string()]); - } -} diff --git a/crates/openshell-router/src/lib.rs b/crates/openshell-router/src/lib.rs deleted file mode 100644 index 79bbfe6ca3..0000000000 --- a/crates/openshell-router/src/lib.rs +++ /dev/null @@ -1,189 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -mod backend; -pub mod config; -mod mock; - -pub use backend::{ - ProxyResponse, StreamingProxyResponse, ValidatedEndpoint, ValidationFailure, - ValidationFailureKind, verify_backend_endpoint, -}; -use config::{ResolvedRoute, RouterConfig}; -use std::time::Duration; -use tracing::info; - -#[derive(Debug, thiserror::Error)] -pub enum RouterError { - #[error("route not found for route '{0}'")] - RouteNotFound(String), - #[error("no compatible route for protocol '{0}'")] - NoCompatibleRoute(String), - #[error("unauthorized: {0}")] - Unauthorized(String), - #[error("upstream unavailable: {0}")] - UpstreamUnavailable(String), - #[error("upstream protocol error: {0}")] - UpstreamProtocol(String), - #[error("internal error: {0}")] - Internal(String), -} - -#[derive(Debug)] -pub struct Router { - routes: Vec, - client: reqwest::Client, -} - -impl Router { - pub fn new() -> Result { - let client = reqwest::Client::builder() - .connect_timeout(Duration::from_secs(30)) - .build() - .map_err(|e| RouterError::Internal(format!("failed to build HTTP client: {e}")))?; - Ok(Self { - routes: Vec::new(), - client, - }) - } - - pub fn from_config(config: &RouterConfig) -> Result { - let resolved = config.resolve_routes()?; - let mut router = Self::new()?; - router.routes = resolved; - Ok(router) - } - - /// Proxy a raw HTTP request to the first compatible route from `candidates`. - /// - /// Filters candidates by `source_protocol` compatibility (exact match against - /// one of the route's `protocols`), then forwards to the first match. - pub async fn proxy_with_candidates( - &self, - source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, - candidates: &[ResolvedRoute], - ) -> Result { - let normalized_source = source_protocol.trim().to_ascii_lowercase(); - let route = candidates - .iter() - .find(|r| r.protocols.iter().any(|p| p == &normalized_source)) - .ok_or_else(|| RouterError::NoCompatibleRoute(source_protocol.to_string()))?; - - info!( - protocols = %route.protocols.join(","), - endpoint = %route.endpoint, - method = %method, - path = %path, - "routing proxy inference request" - ); - - if mock::is_mock_route(route) { - info!(endpoint = %route.endpoint, "returning mock response"); - return Ok(mock::mock_response(route, &normalized_source)); - } - - backend::proxy_to_backend( - &self.client, - route, - &normalized_source, - method, - path, - headers, - body, - ) - .await - } - - /// Streaming variant of [`proxy_with_candidates`](Self::proxy_with_candidates). - /// - /// Returns response headers immediately without buffering the body. - /// The caller streams body chunks via [`StreamingProxyResponse::response`]. - pub async fn proxy_with_candidates_streaming( - &self, - source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, - candidates: &[ResolvedRoute], - ) -> Result { - let normalized_source = source_protocol.trim().to_ascii_lowercase(); - let route = candidates - .iter() - .find(|r| r.protocols.iter().any(|p| p == &normalized_source)) - .ok_or_else(|| RouterError::NoCompatibleRoute(source_protocol.to_string()))?; - - info!( - protocols = %route.protocols.join(","), - endpoint = %route.endpoint, - method = %method, - path = %path, - "routing proxy inference request (streaming)" - ); - - if mock::is_mock_route(route) { - info!(endpoint = %route.endpoint, "returning mock response (buffered)"); - let buffered = mock::mock_response(route, &normalized_source); - return Ok(StreamingProxyResponse::from_buffered(buffered)); - } - - backend::proxy_to_backend_streaming( - &self.client, - route, - &normalized_source, - method, - path, - headers, - body, - ) - .await - } -} - -#[cfg(test)] -mod tests { - use super::*; - use config::{RouteConfig, RouterConfig}; - - fn test_config() -> RouterConfig { - RouterConfig { - routes: vec![RouteConfig { - name: "inference.local".to_string(), - endpoint: "http://localhost:8000/v1".to_string(), - model: "meta/llama-3.1-8b-instruct".to_string(), - provider_type: None, - protocols: vec!["openai_chat_completions".to_string()], - api_key: Some("test-key".to_string()), - api_key_env: None, - }], - } - } - - #[test] - fn router_resolves_routes_from_config() { - let router = Router::from_config(&test_config()).unwrap(); - assert_eq!(router.routes.len(), 1); - assert_eq!(router.routes[0].protocols, vec!["openai_chat_completions"]); - } - - #[test] - fn config_missing_api_key_returns_error() { - let config = RouterConfig { - routes: vec![RouteConfig { - name: "inference.local".to_string(), - endpoint: "http://localhost".to_string(), - model: "test-model".to_string(), - provider_type: None, - protocols: vec!["openai_chat_completions".to_string()], - api_key: None, - api_key_env: None, - }], - }; - let err = Router::from_config(&config).unwrap_err(); - assert!(matches!(err, RouterError::Internal(_))); - } -} diff --git a/crates/openshell-router/src/mock.rs b/crates/openshell-router/src/mock.rs deleted file mode 100644 index 946839bae5..0000000000 --- a/crates/openshell-router/src/mock.rs +++ /dev/null @@ -1,263 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use crate::backend::ProxyResponse; -use crate::config::ResolvedRoute; - -const MOCK_SCHEME: &str = "mock://"; - -/// Returns true if the route's endpoint uses the `mock://` scheme, -/// indicating the router should return a canned response. -pub fn is_mock_route(route: &ResolvedRoute) -> bool { - route.endpoint.starts_with(MOCK_SCHEME) -} - -/// Generate a canned HTTP response appropriate for the route's protocol. -/// -/// The response is protocol-aware: for `openai_chat_completions` it returns -/// a valid `OpenAI` chat completion JSON, for `anthropic_messages` a valid -/// Anthropic response, etc. The route's `model` field is echoed in the response. -pub fn mock_response(route: &ResolvedRoute, source_protocol: &str) -> ProxyResponse { - tracing::warn!( - endpoint = %route.endpoint, - "Serving mock response — mock:// routes should only be used in development/testing" - ); - - let protocol = if route.protocols.iter().any(|p| p == source_protocol) { - source_protocol - } else { - route.protocols.first().map_or("", String::as_str) - }; - - let body = match protocol { - "openai_chat_completions" => openai_chat_completion_body(&route.model), - "openai_completions" => openai_completion_body(&route.model), - "openai_embeddings" => openai_embeddings_body(&route.model), - "anthropic_messages" => anthropic_messages_body(&route.model), - _ => generic_body(&route.model), - }; - - let body_bytes = bytes::Bytes::from(body); - ProxyResponse { - status: 200, - headers: vec![ - ("content-type".to_string(), "application/json".to_string()), - ("x-openshell-mock".to_string(), "true".to_string()), - ], - body: body_bytes, - route_model: Some(route.model.clone()), - route_endpoint: Some(route.endpoint.clone()), - } -} - -fn openai_chat_completion_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "id": "mock-chatcmpl-001", - "object": "chat.completion", - "created": 1_700_000_000_i64, - "model": model, - "choices": [{ - "index": 0, - "message": { - "role": "assistant", - "content": "Hello from openshell mock backend" - }, - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 1, - "completion_tokens": 5, - "total_tokens": 6 - } - })) - .expect("static JSON must serialize") -} - -fn openai_completion_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "id": "mock-cmpl-001", - "object": "text_completion", - "created": 1_700_000_000_i64, - "model": model, - "choices": [{ - "index": 0, - "text": "Hello from openshell mock backend", - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 1, - "completion_tokens": 5, - "total_tokens": 6 - } - })) - .expect("static JSON must serialize") -} - -fn openai_embeddings_body(model: &str) -> Vec { - // Shape must match the OpenAI embeddings response (`object: "list"` with a - // `data` array of `{object, index, embedding}`) so callers that deserialize - // into an embeddings type get a structurally valid — if canned — vector. - serde_json::to_vec(&serde_json::json!({ - "object": "list", - "data": [{ - "object": "embedding", - "index": 0, - "embedding": [0.0_f32, 0.0_f32, 0.0_f32] - }], - "model": model, - "usage": { - "prompt_tokens": 1, - "total_tokens": 1 - } - })) - .expect("static JSON must serialize") -} - -fn anthropic_messages_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "id": "mock-msg-001", - "type": "message", - "role": "assistant", - "model": model, - "content": [{ - "type": "text", - "text": "Hello from openshell mock backend" - }], - "stop_reason": "end_turn", - "usage": { - "input_tokens": 1, - "output_tokens": 5 - } - })) - .expect("static JSON must serialize") -} - -fn generic_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "mock": true, - "model": model, - "message": "Hello from openshell mock backend" - })) - .expect("static JSON must serialize") -} - -#[cfg(test)] -mod tests { - use super::*; - - fn make_route(endpoint: &str, protocols: &[&str], model: &str) -> ResolvedRoute { - ResolvedRoute { - name: "test".to_string(), - endpoint: endpoint.to_string(), - model: model.to_string(), - api_key: "key".to_string(), - protocols: protocols.iter().map(ToString::to_string).collect(), - auth: crate::config::AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: crate::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - #[test] - fn is_mock_route_detects_mock_scheme() { - assert!(is_mock_route(&make_route( - "mock://test", - &["openai_chat_completions"], - "m" - ))); - assert!(is_mock_route(&make_route( - "mock://", - &["openai_chat_completions"], - "m" - ))); - } - - #[test] - fn is_mock_route_rejects_real_urls() { - assert!(!is_mock_route(&make_route( - "https://api.openai.com", - &["openai_chat_completions"], - "m" - ))); - assert!(!is_mock_route(&make_route( - "http://localhost:8000", - &["openai_chat_completions"], - "m" - ))); - } - - #[test] - fn mock_openai_chat_completion() { - let route = make_route("mock://test", &["openai_chat_completions"], "gpt-4"); - let resp = mock_response(&route, "openai_chat_completions"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["model"], "gpt-4"); - assert_eq!(body["object"], "chat.completion"); - assert_eq!( - body["choices"][0]["message"]["content"], - "Hello from openshell mock backend" - ); - } - - #[test] - fn mock_anthropic_messages() { - let route = make_route("mock://test", &["anthropic_messages"], "claude-3"); - let resp = mock_response(&route, "anthropic_messages"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["model"], "claude-3"); - assert_eq!(body["type"], "message"); - assert_eq!( - body["content"][0]["text"], - "Hello from openshell mock backend" - ); - } - - #[test] - fn mock_openai_embeddings() { - let route = make_route( - "mock://test", - &["openai_embeddings"], - "text-embedding-3-small", - ); - let resp = mock_response(&route, "openai_embeddings"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["object"], "list"); - assert_eq!(body["model"], "text-embedding-3-small"); - assert_eq!(body["data"][0]["object"], "embedding"); - assert!( - body["data"][0]["embedding"].is_array(), - "embedding must be a numeric array, got: {body}" - ); - } - - #[test] - fn mock_generic_protocol() { - let route = make_route("mock://test", &["unknown_protocol"], "some-model"); - let resp = mock_response(&route, "unknown_protocol"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["mock"], true); - assert_eq!(body["model"], "some-model"); - } - - #[test] - fn mock_response_includes_marker_header() { - let route = make_route("mock://test", &["openai_chat_completions"], "m"); - let resp = mock_response(&route, "openai_chat_completions"); - assert!( - resp.headers - .iter() - .any(|(k, v)| k == "x-openshell-mock" && v == "true") - ); - } -} diff --git a/crates/openshell-router/tests/backend_integration.rs b/crates/openshell-router/tests/backend_integration.rs deleted file mode 100644 index fe5c2dd0bc..0000000000 --- a/crates/openshell-router/tests/backend_integration.rs +++ /dev/null @@ -1,994 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use openshell_router::Router; -use openshell_router::config::{AuthHeader, ResolvedRoute, RouteConfig, RouterConfig}; -use wiremock::matchers::{bearer_token, body_partial_json, header, method, path}; -use wiremock::{Match, Mock, MockServer, Request, ResponseTemplate}; - -fn mock_candidates(base_url: &str) -> Vec { - vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: base_url.to_string(), - model: "meta/llama-3.1-8b-instruct".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["openai-organization".to_string(), "x-model-id".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }] -} - -#[tokio::test] -async fn proxy_forwards_request_to_backend() { - let mock_server = MockServer::start().await; - - let response_body = serde_json::json!({ - "id": "chatcmpl-123", - "object": "chat.completion", - "created": 1_700_000_000_i64, - "model": "meta/llama-3.1-8b-instruct", - "choices": [{ - "index": 0, - "message": { - "role": "assistant", - "content": "Hello! How can I help you?" - }, - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 10, - "completion_tokens": 8, - "total_tokens": 18 - } - }); - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .respond_with(ResponseTemplate::new(200).set_body_json(&response_body)) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "test", - "messages": [{"role": "user", "content": "Hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["id"], "chatcmpl-123"); -} - -#[tokio::test] -async fn proxy_upstream_401_returns_error() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(401).set_body_json(serde_json::json!({ - "error": { "message": "Invalid API key" } - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap(); - - // Raw proxy returns the actual HTTP status, not a RouterError - assert_eq!(response.status, 401); -} - -#[tokio::test] -async fn proxy_no_compatible_route_returns_error() { - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "http://localhost:1234".to_string(), - model: "test".to_string(), - api_key: "key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let err = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap_err(); - - assert!( - matches!(err, openshell_router::RouterError::NoCompatibleRoute(_)), - "expected NoCompatibleRoute, got: {err:?}" - ); -} - -#[tokio::test] -async fn proxy_strips_auth_header() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - // Client sends its own Authorization header — should be stripped and replaced - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("authorization".to_string(), "Bearer client-key".to_string())], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_forwards_openai_organization_header() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .and(header("openai-organization", "org_123")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![ - ("openai-organization".to_string(), "org_123".to_string()), - ("cookie".to_string(), "session=abc".to_string()), - ], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_mock_route_returns_canned_response() { - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "mock://test".to_string(), - model: "mock/test-model".to_string(), - api_key: "unused".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "mock/test-model", - "messages": [{"role": "user", "content": "hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["model"], "mock/test-model"); - assert_eq!( - resp_body["choices"][0]["message"]["content"], - "Hello from openshell mock backend" - ); -} - -#[tokio::test] -async fn proxy_overrides_model_in_request_body() { - let mock_server = MockServer::start().await; - - // The mock expects the route's model, NOT the client's original model - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "model": "meta/llama-3.1-8b-instruct" - }))) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - // Client sends "gpt-4o-mini" but route is configured with "meta/llama-3.1-8b-instruct" - let body = serde_json::to_vec(&serde_json::json!({ - "model": "gpt-4o-mini", - "messages": [{"role": "user", "content": "Hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_inserts_model_when_absent_from_body() { - let mock_server = MockServer::start().await; - - // The mock expects the route's model to be inserted even though the client didn't send one - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "model": "meta/llama-3.1-8b-instruct" - }))) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - // Client omits "model" entirely - let body = serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "Hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_uses_x_api_key_for_anthropic_route() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "test-anthropic-key")) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_123", - "type": "message", - "role": "assistant", - "content": [{"type": "text", "text": "Hello"}], - "model": "claude-sonnet-4-20250514", - "stop_reason": "end_turn", - "usage": {"input_tokens": 10, "output_tokens": 5} - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "test-anthropic-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-20250514", - "max_tokens": 1, - "messages": [{"role": "user", "content": "hi"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ("anthropic-version".to_string(), "2023-06-01".to_string()), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["type"], "message"); -} - -#[tokio::test] -async fn proxy_anthropic_does_not_send_bearer_auth() { - let mock_server = MockServer::start().await; - - // This mock rejects requests that have a Bearer token — ensuring we DON'T send one - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "anthropic-key")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - // Also mount a catch-all that returns 401 if Bearer is used - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(bearer_token("anthropic-key")) - .respond_with(ResponseTemplate::new(401).set_body_string("should not use bearer")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "anthropic-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(b"{}".to_vec()), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -/// Regression test: when the client sends `anthropic-version`, the header must -/// reach the upstream. Previously, the header was added to the strip list -/// (because it appeared in `default_headers`) AND the default injection was -/// skipped (because `already_sent` checked the *original* input), so neither -/// the client's value nor the default reached the backend. -#[tokio::test] -async fn proxy_forwards_client_anthropic_version_header() { - let mock_server = MockServer::start().await; - - // The upstream requires anthropic-version — wiremock will reject if missing. - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "test-anthropic-key")) - .and(header("anthropic-version", "2024-10-22")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "test-anthropic-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-20250514", - "max_tokens": 1, - "messages": [{"role": "user", "content": "hi"}] - })) - .unwrap(); - - // Client explicitly sends anthropic-version: 2024-10-22 — this value should - // reach the upstream, NOT be silently dropped. - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ("anthropic-version".to_string(), "2024-10-22".to_string()), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!( - response.status, 200, - "upstream should have received anthropic-version header" - ); -} - -#[tokio::test] -async fn proxy_vertex_gemini_route_uses_chat_completions_override() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path( - "/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi/chat/completions", - )) - .and(bearer_token("ya29.test-token")) - .and(body_partial_json(serde_json::json!({ - "model": "gemini-2.0-flash-001", - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "chatcmpl-vertex", - "object": "chat.completion", - "created": 1_700_000_000_i64, - "model": "gemini-2.0-flash-001", - "choices": [{ - "index": 0, - "message": { "role": "assistant", "content": "pong" }, - "finish_reason": "stop" - }] - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!( - "{}/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi", - mock_server.uri() - ), - model: "gemini-2.0-flash-001".to_string(), - api_key: "ya29.test-token".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("/chat/completions".to_string()), - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "client-model", - "messages": [{"role": "user", "content": "ping"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_vertex_anthropic_route_uses_model_path_suffix() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path( - "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet@20241022:rawPredict", - )) - .and(bearer_token("ya29.vertex-token")) - .and(body_partial_json(serde_json::json!({ - "anthropic_version": "vertex-2023-10-16", - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_vertex_1", - "type": "message", - "role": "assistant", - "model": "claude-3-5-sonnet@20241022", - "content": [{"type": "text", "text": "pong"}] - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!( - "{}/v1/projects/my-project/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.vertex-token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - // Include "model" in the body, as Claude Code and other Anthropic SDK - // clients always do. The router must strip it for Vertex AI rawPredict. - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-3-5-sonnet-20241022", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32 - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ("anthropic-beta".to_string(), "tools-2024-05-16".to_string()), - ("anthropic-version".to_string(), "2023-06-01".to_string()), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert_eq!( - received_body["anthropic_version"], - serde_json::json!("vertex-2023-10-16") - ); - // "model" must be stripped: Vertex AI encodes the model in the URL path - // and rejects "model" in the body with "Extra inputs are not permitted". - assert!( - received_body.get("model").is_none(), - "Vertex Anthropic requests must not have model in the body, got: {received_body}" - ); - // anthropic-beta must be stripped: Vertex AI rejects unknown beta values - // with HTTP 400 (e.g. prompt-caching-scope-2026-01-05). - assert!( - !received[0].headers.contains_key("anthropic-beta"), - "anthropic-beta must not reach the Vertex AI backend" - ); - assert!( - !received[0].headers.contains_key("anthropic-version"), - "anthropic-version must be converted to body anthropic_version, not forwarded as a header" - ); -} - -/// Fields that Vertex AI rawPredict actually accepts for Anthropic models. -const VERTEX_ACCEPTED_FIELDS: &[&str] = &[ - "anthropic_version", - "messages", - "max_tokens", - "stop_sequences", - "stream", - "system", - "temperature", - "thinking", - "tool_choice", - "tools", - "top_k", - "top_p", - "metadata", -]; - -/// Simulates Vertex AI's strict pydantic validation: rejects any body field not -/// in the known Anthropic Messages API schema. This is what causes the real 400 -/// error described in #2444. -struct VertexStrictBodyValidator; - -impl Match for VertexStrictBodyValidator { - fn matches(&self, request: &Request) -> bool { - let Ok(body) = serde_json::from_slice::(&request.body) else { - return false; - }; - let Some(obj) = body.as_object() else { - return false; - }; - obj.keys() - .all(|k| VERTEX_ACCEPTED_FIELDS.contains(&k.as_str())) - } -} - -/// End-to-end reproduction of #2444: Claude Code sends `context_management` in -/// the body, the router must strip it before it reaches Vertex. The mock uses -/// strict body validation (like Vertex's pydantic) so the request only succeeds -/// if the field was actually removed. -#[tokio::test] -async fn proxy_vertex_strips_beta_fields_e2e() { - let mock_server = MockServer::start().await; - - let model_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let model = "claude-sonnet-4-6@20250514"; - - // This mock only matches if the body passes strict validation — - // context_management would cause a mismatch (simulating a real 400). - Mock::given(method("POST")) - .and(path(format!("{model_path}/{model}:rawPredict"))) - .and(VertexStrictBodyValidator) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_1", - "type": "message", - "role": "assistant", - "model": model, - "content": [{"type": "text", "text": "Pong"}] - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "vertex-test".to_string(), - endpoint: format!("{}{model_path}", mock_server.uri()), - model: model.to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - // Exact payload Claude Code v2.1.156+ sends: includes model, - // context_management, and anthropic-beta header. - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-6-20250514", - "messages": [{"role": "user", "content": "say pong"}], - "max_tokens": 128, - "context_management": {"enabled": true}, - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ( - "anthropic-beta".to_string(), - "context-management-2025-06-27".to_string(), - ), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - // If context_management leaked through, the strict mock wouldn't match - // and wiremock would return 404. - assert_eq!( - response.status, 200, - "request must succeed — context_management should have been stripped. \ - A 404 here means the strict body validator rejected an unsupported field." - ); -} - -#[tokio::test] -async fn proxy_vertex_anthropic_streaming_route_uses_stream_rawpredict() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path( - "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet@20241022:streamRawPredict", - )) - .and(bearer_token("ya29.vertex-token")) - .respond_with(ResponseTemplate::new(200).set_body_string("{\"id\":\"msg_vertex_stream\"}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!( - "{}/v1/projects/my-project/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.vertex-token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "stream": true - })) - .unwrap(); - - let mut response = router - .proxy_with_candidates_streaming( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let first_chunk = response.next_chunk().await.unwrap(); - assert!( - first_chunk.is_some(), - "streaming response should yield a body chunk" - ); -} - -#[test] -fn config_resolves_routes_with_protocol() { - let config = RouterConfig { - routes: vec![RouteConfig { - name: "inference.local".to_string(), - endpoint: "http://localhost:8000".to_string(), - model: "test-model".to_string(), - provider_type: None, - protocols: vec!["openai_chat_completions".to_string()], - api_key: Some("key".to_string()), - api_key_env: None, - }], - }; - let routes = config.resolve_routes().unwrap(); - assert_eq!(routes[0].protocols, vec!["openai_chat_completions"]); -} - -/// Streaming proxy must not apply a total request timeout to the body stream. -/// -/// The backend delays its response longer than the route timeout. With the old -/// code this would fail (reqwest's total `.timeout()` fires), but the streaming -/// path now omits that timeout — only the client-level `connect_timeout` and -/// the sandbox idle timeout govern liveness. -#[tokio::test] -async fn streaming_proxy_completes_despite_exceeding_route_timeout() { - use std::time::Duration; - - let mock_server = MockServer::start().await; - - let sse_body = concat!( - "data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\n", - "data: {\"choices\":[{\"delta\":{\"content\":\" world\"}}]}\n\n", - "data: [DONE]\n\n", - ); - - // Delay the response 3s — longer than the 1s route timeout. - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .respond_with( - ResponseTemplate::new(200) - .append_header("content-type", "text/event-stream") - .set_body_string(sse_body) - .set_delay(Duration::from_secs(3)), - ) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "test-model".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - // Route timeout shorter than the backend delay — streaming must - // NOT be constrained by this. - timeout: Duration::from_secs(1), - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "test-model", - "messages": [{"role": "user", "content": "hi"}], - "stream": true - })) - .unwrap(); - - // The streaming path should succeed despite the 3s delay exceeding - // the 1s route timeout. - let mut resp = router - .proxy_with_candidates_streaming( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .expect("streaming proxy should not be killed by route timeout"); - - assert_eq!(resp.status, 200); - - // Drain all chunks to verify the full body is received. - let mut total_bytes = 0; - while let Ok(Some(chunk)) = resp.next_chunk().await { - total_bytes += chunk.len(); - } - assert!(total_bytes > 0, "should have received body chunks"); -} - -/// Non-streaming (buffered) proxy must still enforce the route timeout. -#[tokio::test] -async fn buffered_proxy_enforces_route_timeout() { - use std::time::Duration; - - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_string("{}") - // Delay longer than the route timeout. - .set_delay(Duration::from_secs(5)), - ) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "test-model".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: Duration::from_secs(1), - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "test-model", - "messages": [{"role": "user", "content": "hi"}] - })) - .unwrap(); - - let result = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await; - - assert!(result.is_err(), "buffered proxy should timeout"); - let err = result.unwrap_err().to_string(); - assert!( - err.contains("timed out"), - "error should mention timeout, got: {err}" - ); -} diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index 7afae200b5..d86c94b97b 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -122,7 +122,6 @@ pub async fn run_sandbox( ssh_socket_path: Option, _health_check: bool, _health_port: u16, - inference_routes: Option, ocsf_enabled: Arc, ocsf_schema_version: Arc>, network_enabled: bool, @@ -548,7 +547,6 @@ pub async fn run_sandbox( sandbox_id.as_deref(), sandbox_name_for_agg.as_deref(), openshell_endpoint_for_proxy.as_deref(), - inference_routes.as_deref(), denial_tx, activity_tx, agent_proposals.clone(), @@ -4914,8 +4912,8 @@ filesystem_policy: #[test] fn discover_policy_restrictive_default_blocks_network() { - // In cluster mode we keep proxy mode enabled so `inference.local` - // can always be routed through proxy/OPA controls. + // Cluster sandboxes keep proxy mode enabled so egress is evaluated by + // the network policy engine. let proto = openshell_policy::restrictive_default_policy(); let local_policy = SandboxPolicy::try_from(proto).expect("conversion should succeed"); assert!(matches!(local_policy.network.mode, NetworkMode::Proxy)); diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index 1ad69e1070..a74b533cac 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -165,12 +165,6 @@ struct Args { #[arg(long, env = openshell_core::sandbox_env::SSH_SOCKET_PATH)] ssh_socket_path: Option, - /// Path to YAML inference routes for standalone routing. - /// When set, inference routes are loaded from this file instead of - /// fetching a bundle from the gateway. - #[arg(long, env = "OPENSHELL_INFERENCE_ROUTES")] - inference_routes: Option, - /// Enable health check endpoint. #[arg(long)] health_check: bool, @@ -717,7 +711,6 @@ fn main() -> Result<()> { args.ssh_socket_path, args.health_check, args.health_port, - args.inference_routes, ocsf_enabled, ocsf_schema_version, args.mode.network, diff --git a/crates/openshell-sdk/README.md b/crates/openshell-sdk/README.md index cb42e12dc1..c773de627d 100644 --- a/crates/openshell-sdk/README.md +++ b/crates/openshell-sdk/README.md @@ -12,7 +12,7 @@ gateway-name resolution. CRUD, reusable sandbox template CRUD, readiness/deletion waits, and non-streaming exec. - `raw` — direct access to the generated tonic clients for RPCs the curated - surface doesn't yet cover (inference, providers, policy, logs, settings, SSH, + surface doesn't yet cover (providers, policy, logs, settings, SSH, forwarding). ## Auth and refresh @@ -21,9 +21,8 @@ The curated surface drives OIDC refresh automatically: proactively before a request and reactively on `Unauthenticated`. Refreshes are single-flight, so only one is in flight at a time. -The plain `raw_grpc`/`raw_inference` accessors do not refresh; they return a -client bound to the current token. When a refresher is wired, use -`raw_grpc_fresh`/`raw_inference_fresh` to refresh before the call, and +The plain `raw_grpc` accessor does not refresh; it returns a client bound to +the current token. When a refresher is wired, use `raw_grpc_fresh` to refresh before the call, and `force_refresh` to recover after a raw RPC returns `Unauthenticated`. The SDK consumes a `Refresh` trait that the caller implements; it does not run diff --git a/crates/openshell-sdk/src/client.rs b/crates/openshell-sdk/src/client.rs index b5486812f7..cd48c41558 100644 --- a/crates/openshell-sdk/src/client.rs +++ b/crates/openshell-sdk/src/client.rs @@ -4,14 +4,14 @@ //! High-level async client over the gateway gRPC surface. //! //! Covers the sandbox-focused MVP slice: health, sandbox CRUD, readiness / -//! deletion waits, and non-streaming exec. Other RPCs (inference, providers, -//! policy, logs, settings, SSH, forwarding) are reachable via -//! [`OpenShellClient::raw_grpc`] / [`OpenShellClient::raw_inference`]. +//! deletion waits, and non-streaming exec. Other RPCs (providers, policy, +//! logs, settings, SSH, and forwarding) are reachable via +//! [`OpenShellClient::raw_grpc`]. use crate::auth::{BearerSlot, EdgeAuthInterceptor, bearer_metadata}; use crate::config::{AuthConfig, ClientConfig}; use crate::error::{Result, SdkError}; -use crate::raw::{AuthedGrpcClient, AuthedInferenceClient}; +use crate::raw::AuthedGrpcClient; use crate::refresh::{RefreshedToken, TokenSource}; use crate::transport; use crate::types::{ @@ -114,24 +114,6 @@ impl OpenShellClient { Ok(self.raw_grpc()) } - /// Authenticated gRPC client for the inference service. - /// - /// Like [`OpenShellClient::raw_grpc`], this does not drive OIDC refresh; - /// use [`OpenShellClient::raw_inference_fresh`] when a refresher is wired. - pub fn raw_inference(&self) -> AuthedInferenceClient { - proto::inference_client::InferenceClient::with_interceptor( - self.channel.clone(), - self.interceptor.clone(), - ) - } - - /// Like [`OpenShellClient::raw_inference`], but proactively refreshes the - /// bearer token first (see [`OpenShellClient::raw_grpc_fresh`]). - pub async fn raw_inference_fresh(&self) -> Result { - self.ensure_fresh().await?; - Ok(self.raw_inference()) - } - /// Force an OIDC refresh and write the new token into the live bearer /// slot, regardless of expiry. Returns `true` when a refresher is wired /// and a fresh token was minted, `false` for static auth. Use after a diff --git a/crates/openshell-sdk/src/config.rs b/crates/openshell-sdk/src/config.rs index 27a7411734..bc4945c8ee 100644 --- a/crates/openshell-sdk/src/config.rs +++ b/crates/openshell-sdk/src/config.rs @@ -56,8 +56,7 @@ impl AuthConfig { /// Configuration for opening a gRPC channel to an `OpenShell` gateway. /// -/// Consumed by `openshell_sdk::transport::grpc_client` and the -/// inference-client equivalent. One `ClientConfig` per logical connection; +/// Consumed by `openshell_sdk::transport::grpc_client`. One `ClientConfig` per logical connection; /// callers that want connection pooling cache the resulting `tonic::Channel`. // // NOTE: diff --git a/crates/openshell-sdk/src/lib.rs b/crates/openshell-sdk/src/lib.rs index 985c7ecc05..8dfb549294 100644 --- a/crates/openshell-sdk/src/lib.rs +++ b/crates/openshell-sdk/src/lib.rs @@ -9,8 +9,8 @@ //! health, sandbox CRUD, reusable sandbox templates, readiness/deletion //! waits, and non-streaming exec. //! - [`raw`] — direct access to the generated tonic clients for RPCs the -//! curated surface doesn't yet cover (inference, providers, policy, logs, -//! settings, SSH, forwarding). +//! curated surface doesn't yet cover (providers, policy, logs, settings, +//! SSH, forwarding). //! //! Owns the gRPC transport stack — channel construction, TLS material //! handling, request interceptors, OIDC token refresh, and the Cloudflare diff --git a/crates/openshell-sdk/src/raw.rs b/crates/openshell-sdk/src/raw.rs index 35d91f3325..3b1bb6605a 100644 --- a/crates/openshell-sdk/src/raw.rs +++ b/crates/openshell-sdk/src/raw.rs @@ -6,8 +6,8 @@ //! //! Use this module when the curated high-level surface in //! [`crate::client::OpenShellClient`] doesn't expose the RPC or field you -//! need. The high-level surface is sandbox-focused for MVP; inference, -//! providers, policy, logs, settings, SSH, and forwarding all live here. +//! need. The high-level surface is sandbox-focused for MVP; providers, +//! policy, logs, settings, SSH, and forwarding all live here. //! //! ```ignore //! use openshell_sdk::{ClientConfig, OpenShellClient}; @@ -19,7 +19,6 @@ //! ``` pub use openshell_core::proto; -pub use openshell_core::proto::inference_client::InferenceClient; pub use openshell_core::proto::open_shell_client::OpenShellClient as GrpcClient; pub use openshell_core::proto::{ CreateSandboxRequest, CreateSandboxTemplateRequest, CreateWorkspaceRequest, @@ -40,11 +39,3 @@ pub type AuthedGrpcClient = GrpcClient< crate::EdgeAuthInterceptor, >, >; - -/// Type alias for the inference client wrapped in the SDK's auth interceptor. -pub type AuthedInferenceClient = InferenceClient< - tonic::service::interceptor::InterceptedService< - tonic::transport::Channel, - crate::EdgeAuthInterceptor, - >, ->; diff --git a/crates/openshell-server-macros/src/lib.rs b/crates/openshell-server-macros/src/lib.rs index a698ae6623..e85d316293 100644 --- a/crates/openshell-server-macros/src/lib.rs +++ b/crates/openshell-server-macros/src/lib.rs @@ -199,7 +199,7 @@ fn snake_to_pascal(ident: &str) -> String { /// Name of the per-service const emitted alongside the impl block. The /// service module is what disambiguates between services — every impl /// lives in its own module (`crate::grpc::AUTH_METADATA`, -/// `crate::inference::AUTH_METADATA`), so a fixed name reads more +/// another service's `AUTH_METADATA`), so a fixed name reads more /// naturally than `OPENSHELL_AUTH_METADATA` / `INFERENCE_AUTH_METADATA`. const AUTH_METADATA_CONST: &str = "AUTH_METADATA"; diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index 2619fee5cc..a4b9c5076c 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -23,7 +23,6 @@ openshell-otel = { path = "../openshell-otel" } openshell-policy = { path = "../openshell-policy" } openshell-prover = { path = "../openshell-prover" } openshell-providers = { path = "../openshell-providers" } -openshell-router = { path = "../openshell-router" } openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } diff --git a/crates/openshell-server/migrations/postgres/007_remove_inference_routes.sql b/crates/openshell-server/migrations/postgres/007_remove_inference_routes.sql new file mode 100644 index 0000000000..291eeda398 --- /dev/null +++ b/crates/openshell-server/migrations/postgres/007_remove_inference_routes.sql @@ -0,0 +1,2 @@ +-- Managed inference routes were removed in favor of sandbox-scoped provider attachments. +DELETE FROM objects WHERE object_type = 'inference_route'; diff --git a/crates/openshell-server/migrations/sqlite/007_remove_inference_routes.sql b/crates/openshell-server/migrations/sqlite/007_remove_inference_routes.sql new file mode 100644 index 0000000000..291eeda398 --- /dev/null +++ b/crates/openshell-server/migrations/sqlite/007_remove_inference_routes.sql @@ -0,0 +1,2 @@ +-- Managed inference routes were removed in favor of sandbox-scoped provider attachments. +DELETE FROM objects WHERE object_type = 'inference_route'; diff --git a/crates/openshell-server/src/auth/descriptor_authz.rs b/crates/openshell-server/src/auth/descriptor_authz.rs index dbb9fa7ca2..baea879bb7 100644 --- a/crates/openshell-server/src/auth/descriptor_authz.rs +++ b/crates/openshell-server/src/auth/descriptor_authz.rs @@ -17,7 +17,7 @@ use super::method_authz::{AuthMode, Role}; const AUTHORIZATION_EXTENSION: &str = "openshell.options.v1.authorization"; /// Gateway-served protobuf packages. -const GATEWAY_PACKAGES: &[&str] = &["openshell.v1", "openshell.inference.v1"]; +const GATEWAY_PACKAGES: &[&str] = &["openshell.v1"]; /// Bearer-authenticated methods that deliberately require no role or scope. /// diff --git a/crates/openshell-server/src/auth/method_authz.rs b/crates/openshell-server/src/auth/method_authz.rs index 71eb7acac6..35ff82e925 100644 --- a/crates/openshell-server/src/auth/method_authz.rs +++ b/crates/openshell-server/src/auth/method_authz.rs @@ -139,9 +139,6 @@ mod tests { "/openshell.v1.OpenShell/ConnectSupervisor" )); assert!(!is_user_callable("/openshell.v1.OpenShell/RelayStream")); - assert!(!is_user_callable( - "/openshell.inference.v1.Inference/GetInferenceBundle" - )); // Unauthenticated methods are not "user callable" — they're // intercepted before principal evaluation. assert!(!is_user_callable("/openshell.v1.OpenShell/Health")); diff --git a/crates/openshell-server/src/auth/sandbox_methods.rs b/crates/openshell-server/src/auth/sandbox_methods.rs index 89f34d1253..f368298dd3 100644 --- a/crates/openshell-server/src/auth/sandbox_methods.rs +++ b/crates/openshell-server/src/auth/sandbox_methods.rs @@ -29,9 +29,6 @@ mod tests { assert!(is_sandbox_callable( "/openshell.v1.OpenShell/GetSandboxConfig" )); - assert!(is_sandbox_callable( - "/openshell.inference.v1.Inference/GetInferenceBundle" - )); assert!(is_sandbox_callable( "/openshell.v1.OpenShell/ExchangeProviderSubjectToken" )); @@ -53,11 +50,5 @@ mod tests { assert!(!is_sandbox_callable( "/openshell.v1.OpenShell/ApproveDraftChunk" )); - assert!(!is_sandbox_callable( - "/openshell.inference.v1.Inference/GetInferenceRoute" - )); - assert!(!is_sandbox_callable( - "/openshell.inference.v1.Inference/SetInferenceRoute" - )); } } diff --git a/crates/openshell-server/src/grpc/mod.rs b/crates/openshell-server/src/grpc/mod.rs index 8143e6058e..620fc3ed7a 100644 --- a/crates/openshell-server/src/grpc/mod.rs +++ b/crates/openshell-server/src/grpc/mod.rs @@ -189,19 +189,6 @@ enum StoredSettingValue { // Utility // --------------------------------------------------------------------------- -/// Validate that object metadata is present and contains required fields. -/// -/// This is a crate-level helper that wraps the validation module's implementation. -/// Use this from modules outside of `grpc` that need to validate metadata. -// `tonic::Status` is large but is the API surface of gRPC handlers. -#[allow(clippy::result_large_err)] -pub fn validate_object_metadata( - metadata: Option<&openshell_core::proto::datamodel::v1::ObjectMeta>, - resource_type: &str, -) -> Result<(), Status> { - validation::validate_object_metadata(metadata, resource_type) -} - // --------------------------------------------------------------------------- // Service struct // --------------------------------------------------------------------------- diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index 0e4c74af95..629ba5d24b 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -2808,11 +2808,6 @@ async fn provider_policy_context_with_catalog( continue; }; - if !super::provider::provider_profile_endpoints_are_active(&profile, &provider) { - endpointless_provider_names.insert(name.clone()); - continue; - } - let rule_name = openshell_policy::provider_rule_name(provider.object_name()); let mut rule = profile.network_policy_rule(&rule_name); if rule.endpoints.is_empty() { @@ -8549,36 +8544,6 @@ mod tests { ); } - #[tokio::test] - async fn provider_policy_layers_skip_public_vendor_endpoints_for_alternate_upstreams() { - let store = test_store().await; - let mut openai = test_provider("alternate-openai", "openai"); - openai.config.insert( - "OPENAI_BASE_URL".to_string(), - "https://api.example.com/v1".to_string(), - ); - let mut anthropic = test_provider("alternate-anthropic", "anthropic"); - anthropic.config.insert( - "ANTHROPIC_BASE_URL".to_string(), - "https://api.example.com/v1".to_string(), - ); - store.put_message(&openai).await.unwrap(); - store.put_message(&anthropic).await.unwrap(); - - let layers = profile_provider_policy_layers( - &store, - "default", - &[ - "alternate-openai".to_string(), - "alternate-anthropic".to_string(), - ], - ) - .await - .unwrap(); - - assert!(layers.is_empty()); - } - #[tokio::test] async fn provider_policy_layers_respect_profile_workspace_scope() { let store = test_store().await; diff --git a/crates/openshell-server/src/grpc/provider.rs b/crates/openshell-server/src/grpc/provider.rs index 764c0bbfde..310d6b026a 100644 --- a/crates/openshell-server/src/grpc/provider.rs +++ b/crates/openshell-server/src/grpc/provider.rs @@ -44,7 +44,7 @@ const GATEWAY_SPIFFE_WORKLOAD_API_SOCKET: &str = "OPENSHELL_GATEWAY_SPIFFE_WORKL /// Redact credential values from a provider before returning it in a gRPC /// response. Key names are preserved so callers can display credential counts -/// and key listings. Internal server paths (inference routing, sandbox env +/// and key listings. Internal server paths (sandbox env /// injection) read credentials from the store directly and are unaffected. fn redact_provider_credentials(mut provider: Provider) -> Provider { for value in provider.credentials.values_mut() { @@ -2293,8 +2293,7 @@ fn provider_credential_not_expired(provider: &Provider, key: &str, now_ms: i64) } fn is_non_injectable_provider_credential(provider: &Provider, key: &str) -> bool { - openshell_core::inference::normalize_inference_provider_type(&provider.r#type) - == Some("google-vertex-ai") + normalize_provider_type(&provider.r#type) == Some("google-vertex-ai") && key == "GOOGLE_SERVICE_ACCOUNT_KEY" } @@ -2974,6 +2973,9 @@ pub(super) fn get_provider_type_profile_for_scope( catalog.get_type_profile_for_scope(id, profile_workspace) } +/// Prevent a legacy alternate-upstream provider from binding its credential to +/// the built-in public vendor endpoint. Alternate endpoints must be expressed +/// by an explicitly imported endpoint-bearing profile. pub(super) fn provider_profile_endpoints_are_active( profile: &ProviderTypeProfile, provider: &Provider, @@ -2981,24 +2983,24 @@ pub(super) fn provider_profile_endpoints_are_active( if profile.source != "builtin" { return true; } - let Some(inference_profile) = openshell_core::inference::profile_for(&profile.id) else { - return true; + + let (base_url_key, default_base_url) = match profile.id.as_str() { + "openai" => ("OPENAI_BASE_URL", "https://api.openai.com/v1"), + "anthropic" => ("ANTHROPIC_BASE_URL", "https://api.anthropic.com/v1"), + _ => return true, }; - if !matches!(inference_profile.provider_type, "openai" | "anthropic") { - return true; - } - let configured_base_url = inference_profile - .base_url_config_keys - .iter() - .find_map(|key| provider.config.get(*key)) - .map(|value| value.trim()) - .filter(|value| !value.is_empty()); - configured_base_url.is_none_or(|configured| { - configured - .trim_end_matches('/') - .eq_ignore_ascii_case(inference_profile.default_base_url.trim_end_matches('/')) - }) + provider + .config + .get(base_url_key) + .map(String::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + .is_none_or(|configured| { + configured + .trim_end_matches('/') + .eq_ignore_ascii_case(default_base_url.trim_end_matches('/')) + }) } #[cfg(test)] diff --git a/crates/openshell-server/src/grpc/workspace.rs b/crates/openshell-server/src/grpc/workspace.rs index 7446938156..30ca756a1b 100644 --- a/crates/openshell-server/src/grpc/workspace.rs +++ b/crates/openshell-server/src/grpc/workspace.rs @@ -12,11 +12,11 @@ use openshell_core::proto::datamodel::v1::{ObjectMeta, WorkspacePhase, Workspace use openshell_core::proto::{ AddWorkspaceMemberRequest, AddWorkspaceMemberResponse, CreateWorkspaceRequest, CreateWorkspaceResponse, DeleteWorkspaceRequest, DeleteWorkspaceResponse, GetWorkspaceRequest, - GetWorkspaceResponse, InferenceRoute, ListWorkspaceMembersRequest, - ListWorkspaceMembersResponse, ListWorkspacesRequest, ListWorkspacesResponse, Provider, - RemoveWorkspaceMemberRequest, RemoveWorkspaceMemberResponse, Sandbox, SandboxWorkloadTemplate, - ServiceEndpoint, SshSession, StoredProviderCredentialRefreshState, StoredProviderProfile, - Workspace, WorkspaceMember, WorkspaceRole, + GetWorkspaceResponse, ListWorkspaceMembersRequest, ListWorkspaceMembersResponse, + ListWorkspacesRequest, ListWorkspacesResponse, Provider, RemoveWorkspaceMemberRequest, + RemoveWorkspaceMemberResponse, Sandbox, SandboxWorkloadTemplate, ServiceEndpoint, SshSession, + StoredProviderCredentialRefreshState, StoredProviderProfile, Workspace, WorkspaceMember, + WorkspaceRole, }; use prost::Message; use tonic::{Request, Response, Status}; @@ -411,13 +411,7 @@ pub(super) async fn handle_delete_workspace( // Cascade-delete non-blocking resources before the final CAS delete. // This is safe without a transaction: the workspace is Terminating, so // ensure_active rejects new resource creation. If delete_if conflicts - // below, the retry will find no routes/members to delete and succeed. - state - .store - .delete_all_in_workspace(InferenceRoute::object_type(), &name) - .await - .map_err(|e| Status::internal(format!("delete inference routes failed: {e}")))?; - + // below, the retry will find no members to delete and succeed. state .store .delete_all_in_workspace(WorkspaceMember::object_type(), &name) @@ -1525,64 +1519,6 @@ mod tests { assert!(err.message().contains("being deleted")); } - #[tokio::test] - async fn delete_workspace_cascade_deletes_inference_routes() { - let state = test_server_state().await; - - handle_create_workspace( - &state, - Request::new(CreateWorkspaceRequest { - name: "route-test".to_string(), - labels: HashMap::new(), - }), - ) - .await - .unwrap(); - - let route = InferenceRoute { - metadata: Some(ObjectMeta { - id: "route-1".to_string(), - name: "inference.local".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - annotations: HashMap::new(), - resource_version: 0, - workspace: "route-test".to_string(), - deletion_timestamp_ms: 0, - }), - config: Some(openshell_core::proto::InferenceRouteConfig { - provider_name: "test-provider".to_string(), - model_id: "gpt-4o".to_string(), - timeout_secs: 0, - }), - version: 1, - }; - state.store.put_message(&route).await.unwrap(); - - // Inference route should NOT block workspace deletion. - let resp = handle_delete_workspace( - &state, - Request::new(DeleteWorkspaceRequest { - name: "route-test".to_string(), - }), - ) - .await - .unwrap() - .into_inner(); - assert!(resp.deleted); - - // Inference route should have been cascade-deleted. - let remaining: Vec = state - .store - .list_messages("route-test", 100, 0) - .await - .unwrap(); - assert!( - remaining.is_empty(), - "inference routes should be cascade-deleted with workspace" - ); - } - /// Non-member callers must receive `PERMISSION_DENIED` — not `NOT_FOUND` — /// when targeting a workspace that does not exist. Returning `NOT_FOUND` /// would create a CWE-203 workspace-name oracle. diff --git a/crates/openshell-server/src/inference.rs b/crates/openshell-server/src/inference.rs deleted file mode 100644 index b83fd6be4f..0000000000 --- a/crates/openshell-server/src/inference.rs +++ /dev/null @@ -1,3839 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![allow(clippy::result_large_err)] // gRPC handlers return Result, Status> - -use openshell_core::inference::{ - VERTEX_AI_PROJECT_ID_KEY, VERTEX_AI_PUBLISHER_KEY, VERTEX_AI_REGION_KEY, -}; -use openshell_core::proto::{ - DeleteInferenceRouteRequest, DeleteInferenceRouteResponse, GetInferenceBundleRequest, - GetInferenceBundleResponse, GetInferenceRouteRequest, GetInferenceRouteResponse, - InferenceRoute, InferenceRouteConfig, Provider, ResolvedRoute, Sandbox, - SetInferenceRouteRequest, SetInferenceRouteResponse, ValidatedEndpoint, - inference_server::Inference, -}; -use openshell_core::{ObjectId, ObjectLabels, ObjectWorkspace}; -use openshell_providers::normalize_provider_type; -use openshell_router::config::ResolvedRoute as RouterResolvedRoute; -use openshell_router::{ValidationFailureKind, verify_backend_endpoint}; -use prost::Message as _; -use std::collections::HashMap; -use std::sync::Arc; -use std::time::Duration; -use tonic::{Request, Response, Status}; - -use crate::{ - ServerState, - auth::workspace_authz::{MinWorkspaceRole, authorize_workspace}, - persistence::{ObjectName, ObjectType, Store, WriteCondition, current_time_ms}, -}; - -#[derive(Debug)] -pub struct InferenceService { - state: Arc, -} - -impl InferenceService { - pub fn new(state: Arc) -> Self { - Self { state } - } -} - -const CLUSTER_INFERENCE_ROUTE_NAME: &str = "inference.local"; -const SANDBOX_SYSTEM_ROUTE_NAME: &str = "sandbox-system"; - -/// Map a request `route_name` to the canonical store key. -/// -/// Empty string defaults to `CLUSTER_INFERENCE_ROUTE_NAME` for backward compat. -fn effective_route_name(name: &str) -> Result<&str, Status> { - match name.trim() { - "" | "inference.local" => Ok(CLUSTER_INFERENCE_ROUTE_NAME), - "sandbox-system" => Ok(SANDBOX_SYSTEM_ROUTE_NAME), - other => Err(Status::invalid_argument(format!( - "unknown route_name '{other}'; expected 'inference.local' or 'sandbox-system'" - ))), - } -} - -impl ObjectType for InferenceRoute { - fn object_type() -> &'static str { - "inference_route" - } -} - -#[tonic::async_trait] -impl Inference for InferenceService { - async fn get_inference_bundle( - &self, - request: Request, - ) -> Result, Status> { - let sandbox_id = authorize_inference_bundle( - request - .extensions() - .get::(), - )?; - let sandbox: Sandbox = self - .state - .store - .get_message::(&sandbox_id) - .await - .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? - .ok_or_else(|| Status::not_found(format!("sandbox '{sandbox_id}' not found")))?; - let workspace = sandbox.object_workspace(); - resolve_inference_bundle_with_credentials( - self.state.store.as_ref(), - workspace, - Some(&self.state.credentials), - ) - .await - .map(Response::new) - } - - async fn set_inference_route( - &self, - request: Request, - ) -> Result, Status> { - let principal = crate::grpc::extract_principal(&request)?; - let req = request.into_inner(); - let authz = authorize_workspace( - &self.state.store, - &self.state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::Admin, - ) - .await?; - let workspace = - crate::grpc::workspace::resolve_workspace(self.state.store.as_ref(), &authz.workspace) - .await? - .ensure_active()?; - let route_name = effective_route_name(&req.route_name)?; - let verify = !req.no_verify; - let route = upsert_cluster_inference_route_with_credentials( - self.state.store.as_ref(), - &workspace, - Some(&self.state.credentials), - route_name, - &req.provider_name, - &req.model_id, - req.timeout_secs, - verify, - ) - .await?; - - let config = route - .route - .config - .as_ref() - .ok_or_else(|| Status::internal("managed route missing config"))?; - - Ok(Response::new(SetInferenceRouteResponse { - provider_name: config.provider_name.clone(), - model_id: config.model_id.clone(), - version: route.route.version, - route_name: route_name.to_string(), - validation_performed: !route.validation.is_empty(), - validated_endpoints: route.validation, - timeout_secs: config.timeout_secs, - workspace, - })) - } - - async fn get_inference_route( - &self, - request: Request, - ) -> Result, Status> { - let principal = crate::grpc::extract_principal(&request)?; - let req = request.into_inner(); - let authz = authorize_workspace( - &self.state.store, - &self.state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::User, - ) - .await?; - let workspace = - crate::grpc::workspace::resolve_workspace(self.state.store.as_ref(), &authz.workspace) - .await? - .name; - let route_name = effective_route_name(&req.route_name)?; - let route = self - .state - .store - .get_message_by_name::(&workspace, route_name) - .await - .map_err(|e| Status::internal(format!("fetch route failed: {e}")))? - .ok_or_else(|| { - Status::not_found(format!( - "inference route '{route_name}' is not configured in workspace '{workspace}'; run 'openshell inference set --provider --model '" - )) - })?; - - let config = route - .config - .as_ref() - .ok_or_else(|| Status::internal("managed route missing config"))?; - - if config.provider_name.trim().is_empty() || config.model_id.trim().is_empty() { - return Err(Status::failed_precondition( - "managed route is missing provider/model metadata", - )); - } - - Ok(Response::new(GetInferenceRouteResponse { - provider_name: config.provider_name.clone(), - model_id: config.model_id.clone(), - version: route.version, - route_name: route_name.to_string(), - timeout_secs: config.timeout_secs, - workspace, - })) - } - - async fn delete_inference_route( - &self, - request: Request, - ) -> Result, Status> { - let principal = crate::grpc::extract_principal(&request)?; - let req = request.into_inner(); - let authz = authorize_workspace( - &self.state.store, - &self.state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::Admin, - ) - .await?; - let workspace = - crate::grpc::workspace::resolve_workspace(self.state.store.as_ref(), &authz.workspace) - .await? - .name; - let route_name = effective_route_name(&req.route_name)?; - let deleted = self - .state - .store - .delete_by_name(InferenceRoute::object_type(), &workspace, route_name) - .await - .map_err(|e| Status::internal(format!("delete route failed: {e}")))?; - Ok(Response::new(DeleteInferenceRouteResponse { deleted })) - } -} - -#[cfg(test)] -async fn upsert_cluster_inference_route( - store: &Store, - workspace: &str, - route_name: &str, - provider_name: &str, - model_id: &str, - timeout_secs: u64, - verify: bool, -) -> Result { - upsert_cluster_inference_route_with_credentials( - store, - workspace, - None, - route_name, - provider_name, - model_id, - timeout_secs, - verify, - ) - .await -} - -#[cfg(test)] -async fn upsert_inference_route( - store: &Store, - workspace: &str, - route_name: &str, - provider_name: &str, - model_id: &str, - timeout_secs: u64, - verify: bool, -) -> Result { - upsert_cluster_inference_route( - store, - workspace, - route_name, - provider_name, - model_id, - timeout_secs, - verify, - ) - .await -} - -#[allow(clippy::too_many_arguments)] -async fn upsert_cluster_inference_route_with_credentials( - store: &Store, - workspace: &str, - credentials: Option<&crate::credentials::CredentialRuntime>, - route_name: &str, - provider_name: &str, - model_id: &str, - timeout_secs: u64, - verify: bool, -) -> Result { - if provider_name.trim().is_empty() { - return Err(Status::invalid_argument("provider_name is required")); - } - if model_id.trim().is_empty() { - return Err(Status::invalid_argument("model_id is required")); - } - - let provider = store - .get_message_by_name::(workspace, provider_name) - .await - .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))? - .ok_or_else(|| { - Status::failed_precondition(format!( - "provider '{provider_name}' not found in workspace '{workspace}'" - )) - })?; - let provider = resolve_provider_credentials(provider, credentials).await?; - - let resolved = resolve_provider_route(&provider, model_id)?; - let validation = if verify { - vec![verify_provider_endpoint(provider.object_name(), model_id, &resolved).await?] - } else { - Vec::new() - }; - - let config = build_inference_route_config(&provider, model_id, timeout_secs); - - let existing = store - .get_message_by_name::(workspace, route_name) - .await - .map_err(|e| Status::internal(format!("fetch route failed: {e}")))?; - - let now_ms = current_time_ms(); - - let (id, metadata, new_version, condition) = if let Some(existing) = existing { - let resource_version = existing.metadata.as_ref().map_or(0, |m| m.resource_version); - ( - existing.object_id().to_string(), - existing.metadata.clone(), - existing.version.saturating_add(1), - WriteCondition::MatchResourceVersion(resource_version), - ) - } else { - let new_id = uuid::Uuid::new_v4().to_string(); - let new_metadata = Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: new_id.clone(), - name: route_name.to_string(), - created_at_ms: now_ms, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }); - (new_id, new_metadata, 1, WriteCondition::MustCreate) - }; - - let route = InferenceRoute { - metadata, - config: Some(config), - version: new_version, - }; - - crate::grpc::validate_object_metadata(route.metadata.as_ref(), "inference_route")?; - - let labels_map = route.object_labels(); - let labels_json = if labels_map.as_ref().is_none_or(HashMap::is_empty) { - None - } else { - Some( - serde_json::to_string(&labels_map) - .map_err(|e| Status::internal(format!("failed to serialize labels: {e}")))?, - ) - }; - store - .put_if( - InferenceRoute::object_type(), - &id, - route_name, - workspace, - &route.encode_to_vec(), - labels_json.as_deref(), - condition, - ) - .await - .map_err(|e| crate::grpc::persistence_error_to_status(e, "upsert inference route"))?; - - Ok(UpsertedInferenceRoute { route, validation }) -} - -fn build_inference_route_config( - provider: &Provider, - model_id: &str, - timeout_secs: u64, -) -> InferenceRouteConfig { - InferenceRouteConfig { - provider_name: provider.object_name().to_string(), - model_id: model_id.to_string(), - timeout_secs, - } -} - -#[derive(Debug)] -struct ResolvedProviderRoute { - provider_type: String, - route: RouterResolvedRoute, -} - -#[derive(Debug)] -struct UpsertedInferenceRoute { - route: InferenceRoute, - validation: Vec, -} - -/// Infer the Vertex AI publisher segment from a model identifier. -/// -/// Currently only the `"anthropic"` result is consumed by -/// `resolve_vertex_ai_route` to select between the native Anthropic -/// Messages API (`rawPredict`) and the OpenAI-compatible endpoint. -/// Non-Anthropic publisher mappings (`meta`, `mistralai`, `ai21`, -/// `deepseek`, `google`) are maintained for forward compatibility -/// and documentation value — all non-Anthropic models route to the -/// same OpenAI-compatible endpoint regardless of publisher. -/// -/// Returns `None` for unrecognized models, which causes resolution to -/// fall back to the OpenAI-compatible endpoint -/// (`v1beta1/.../endpoints/openapi`). -fn infer_vertex_publisher(model_id: &str) -> Option<&'static str> { - if model_id.starts_with("claude-") { - Some("anthropic") - } else if model_id.starts_with("gemini-") - || model_id.starts_with("text-bison-") - || model_id.starts_with("chat-bison-") - { - Some("google") - } else if model_id.starts_with("llama-") { - Some("meta") - } else if model_id.starts_with("mistral-") || model_id.starts_with("codestral-") { - Some("mistralai") - } else if model_id.starts_with("jamba-") { - Some("ai21") - } else if model_id.starts_with("deepseek-") { - Some("deepseek") - } else { - None - } -} - -/// Return a required Vertex AI config value, or a `FailedPrecondition` status. -fn required_vertex_config<'a>( - config: &'a HashMap, - key: &str, -) -> Result<&'a str, Status> { - config - .get(key) - .map(String::as_str) - .filter(|v| !v.trim().is_empty()) - .ok_or_else(|| { - Status::failed_precondition(format!("Vertex AI provider requires {key} config")) - }) -} - -/// Validate a GCP project ID against the documented format. -/// -/// GCP project IDs must be 6–30 characters, start with a lowercase letter, -/// contain only lowercase letters, digits, and hyphens, and not end with a hyphen. -fn validate_gcp_project_id(value: &str) -> Result<(), Status> { - let valid = value.len() >= 6 - && value.len() <= 30 - && value.starts_with(|c: char| c.is_ascii_lowercase()) - && !value.ends_with('-') - && value - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'); - if valid { - Ok(()) - } else { - Err(Status::invalid_argument(format!( - "VERTEX_AI_PROJECT_ID has invalid format: {value:?}. \ - GCP project IDs must be 6-30 characters, start with a lowercase letter, \ - contain only lowercase letters, digits, and hyphens, and not end with a hyphen." - ))) - } -} - -/// Validate a GCP region/location value. -/// -/// Accepts the special keywords `global`, `us`, and `eu`, plus standard -/// regional patterns like `us-central1`, `europe-west4`, `us-east4-a`. -fn validate_gcp_region(value: &str) -> Result<(), Status> { - let lower = value.trim().to_ascii_lowercase(); - let valid = matches!(lower.as_str(), "global" | "us" | "eu") - || (lower - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') - && lower.contains('-') - && !lower.starts_with('-') - && !lower.ends_with('-')); - if valid { - Ok(()) - } else { - Err(Status::invalid_argument(format!( - "VERTEX_AI_REGION has invalid format: {value:?}. \ - Expected a GCP region (e.g. us-central1, europe-west4) \ - or one of: global, us, eu." - ))) - } -} - -/// Resolve the Vertex AI API host and normalized location from a configured region. -fn vertex_location_and_host(region: &str) -> (String, String) { - let location = region.trim().to_ascii_lowercase(); - let host = match location.as_str() { - "global" => "aiplatform.googleapis.com".to_string(), - "us" | "eu" => format!("aiplatform.{location}.rep.googleapis.com"), - _ => format!("{location}-aiplatform.googleapis.com"), - }; - (location, host) -} - -/// Reject Bedrock model ids that would produce ambiguous or malformed -/// upstream URL paths. -/// -/// AWS Bedrock encodes the model in `/model//invoke`, so the value -/// is interpolated directly into a URL path segment. Without -/// validation, a value containing `/`, `\`, percent escapes, query or -/// fragment delimiters, traversal segments, whitespace, or control -/// characters could break out of the path segment, smuggle a different -/// upstream route, or produce ambiguous/malformed paths upstream. -/// -/// Mirrors [`validate_vertex_model_id`] — Bedrock has the same exposure -/// for the same reason, and the contract is enforced again at the -/// router layer (`is_valid_bedrock_model_id`) as defense-in-depth. -fn validate_aws_bedrock_model_id(value: &str) -> Result<(), Status> { - let trimmed = value.trim(); - if trimmed.is_empty() { - return Err(Status::invalid_argument("model_id is required")); - } - if value != trimmed { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not include leading or trailing whitespace: {value:?}" - ))); - } - if value.contains('/') || value.contains('\\') { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain path separators: {value:?}" - ))); - } - if value.chars().any(|c| matches!(c, '?' | '#' | '%')) { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain URL delimiters or percent escapes: {value:?}" - ))); - } - if value.contains("..") { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain traversal segments: {value:?}" - ))); - } - if value.chars().any(|c| c.is_control() || c.is_whitespace()) { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain whitespace or control characters: {value:?}" - ))); - } - Ok(()) -} - -fn validate_vertex_model_id(value: &str) -> Result<(), Status> { - let trimmed = value.trim(); - if trimmed.is_empty() { - return Err(Status::invalid_argument("model_id is required")); - } - if value != trimmed { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not include leading or trailing whitespace: {value:?}" - ))); - } - if value.contains('/') || value.contains('\\') { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain path separators: {value:?}" - ))); - } - if value.chars().any(|c| matches!(c, '?' | '#' | '%')) { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain URL delimiters or percent escapes: {value:?}" - ))); - } - if value.contains("..") { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain traversal segments: {value:?}" - ))); - } - if value.chars().any(|c| c.is_control() || c.is_whitespace()) { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain whitespace or control characters: {value:?}" - ))); - } - Ok(()) -} - -fn is_allowed_vertex_override_host(host: &str) -> bool { - matches!( - host, - "aiplatform.googleapis.com" - | "aiplatform.us.rep.googleapis.com" - | "aiplatform.eu.rep.googleapis.com" - ) || host.ends_with("-aiplatform.googleapis.com") -} - -fn validate_vertex_base_url(value: &str) -> Result { - let trimmed = value.trim(); - let url = url::Url::parse(trimmed).map_err(|err| { - Status::invalid_argument(format!("Vertex AI base URL override is invalid: {err}")) - })?; - - if url.scheme() != "https" { - return Err(Status::invalid_argument( - "Vertex AI base URL override must use https".to_string(), - )); - } - if !url.username().is_empty() || url.password().is_some() { - return Err(Status::invalid_argument( - "Vertex AI base URL override must not include userinfo".to_string(), - )); - } - if url.query().is_some() || url.fragment().is_some() { - return Err(Status::invalid_argument( - "Vertex AI base URL override must not include query or fragment components".to_string(), - )); - } - if let Some(port) = url.port() - && port != 443 - { - return Err(Status::invalid_argument(format!( - "Vertex AI base URL override must use port 443 when an explicit port is set, got {port}" - ))); - } - - match url.host() { - Some(url::Host::Domain(host)) if is_allowed_vertex_override_host(host) => {} - Some(url::Host::Domain(host)) => { - return Err(Status::invalid_argument(format!( - "Vertex AI base URL override must target an official Vertex AI hostname, got {host:?}" - ))); - } - Some(url::Host::Ipv4(_) | url::Host::Ipv6(_)) => { - return Err(Status::invalid_argument(format!( - "Vertex AI base URL override must not use IP literal hosts: {}", - url.host_str().unwrap_or("") - ))); - } - None => { - return Err(Status::invalid_argument( - "Vertex AI base URL override must include a host".to_string(), - )); - } - } - - Ok(trimmed.to_string()) -} - -/// Build a [`RouterResolvedRoute`] for Vertex AI without duplicating the 15-field struct. -#[allow(clippy::too_many_arguments)] -fn build_vertex_route( - route_name: &str, - endpoint: String, - model_id: &str, - api_key: &str, - protocols: Vec, - profile: &openshell_core::inference::InferenceProviderProfile, - model_in_path: bool, - request_path_override: Option, -) -> RouterResolvedRoute { - RouterResolvedRoute { - name: route_name.to_string(), - endpoint, - model: model_id.to_string(), - api_key: api_key.to_string(), - protocols, - auth: profile.auth.clone(), - default_headers: profile - .default_headers - .iter() - .map(|(k, v)| ((*k).to_string(), (*v).to_string())) - .collect(), - passthrough_headers: profile - .passthrough_headers - .iter() - .map(|p| (*p).to_string()) - .collect(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path, - request_path_override, - } -} - -/// Resolve a Vertex AI route given provider config, model, and bearer token. -fn resolve_vertex_ai_route( - config: &HashMap, - model_id: &str, - route_name: &str, - api_key: &str, - profile: &openshell_core::inference::InferenceProviderProfile, -) -> Result { - // Validate model_id early — it appears in URL paths for Anthropic routes - // and in JSON request bodies for all routes. Rejecting path separators, - // traversal segments, and control characters up front is defense-in-depth. - validate_vertex_model_id(model_id)?; - - // Determine if this is an Anthropic model. - // Explicit VERTEX_AI_PUBLISHER=anthropic overrides inference. - // All non-Anthropic models route to the OpenAI-compatible endpoint. - let explicit_publisher = config - .get(VERTEX_AI_PUBLISHER_KEY) - .map(String::as_str) - .filter(|v| !v.trim().is_empty()); - - let is_anthropic = explicit_publisher.map_or_else( - || infer_vertex_publisher(model_id) == Some("anthropic"), - |p| p.eq_ignore_ascii_case("anthropic"), - ); - - // Vertex's OpenAI-compatible endpoint requires the request body's `model` - // field to carry a publisher prefix: `/` (e.g. - // `google/gemini-2.5-flash`). The publisher is taken from the explicit - // VERTEX_AI_PUBLISHER config value (when set to a non-Anthropic value) or - // inferred from the model name. For unrecognised models with no explicit - // publisher, the bare model ID is forwarded unchanged; Vertex will return - // a 400 in that case, which is the correct observable signal to the caller. - // Anthropic rawPredict routes encode the model in the URL path, not the - // body, so they are unaffected. - let body_model_id: String = if is_anthropic { - model_id.to_string() - } else { - let publisher = explicit_publisher.or_else(|| infer_vertex_publisher(model_id)); - publisher.map_or_else(|| model_id.to_string(), |p| format!("{p}/{model_id}")) - }; - - // Escape hatch: caller-supplied full base URL still uses the model-derived - // protocol and path contract, but only for the OpenAI-compatible Vertex surface. - // Anthropic-on-Vertex needs model-path shaping and body adaptation that a fully - // caller-controlled URL cannot safely preserve. - if let Some(base_url) = profile - .base_url_config_keys - .iter() - .find_map(|key| config.get(*key).filter(|v| !v.trim().is_empty())) - .map(String::as_str) - { - if is_anthropic { - return Err(Status::invalid_argument( - "Vertex AI base URL overrides are not supported for Anthropic models. \ - Remove GOOGLE_VERTEX_AI_BASE_URL / VERTEX_AI_BASE_URL and configure \ - VERTEX_AI_PROJECT_ID + VERTEX_AI_REGION instead." - .to_string(), - )); - } - let base_url = validate_vertex_base_url(base_url)?; - - return Ok(build_vertex_route( - route_name, - base_url, - &body_model_id, - api_key, - vec!["openai_chat_completions".to_string()], - profile, - false, - Some("/chat/completions".to_string()), - )); - } - - let project = required_vertex_config(config, VERTEX_AI_PROJECT_ID_KEY)?; - validate_gcp_project_id(project)?; - let region = config - .get(VERTEX_AI_REGION_KEY) - .map(String::as_str) - .filter(|v| !v.trim().is_empty()) - .unwrap_or("us-central1"); - validate_gcp_region(region)?; - let (location, host) = vertex_location_and_host(region); - - if is_anthropic { - // Native Anthropic Messages API via rawPredict. - // model_id is NOT embedded in the endpoint — it is carried in route.model - // and appended with the suffix by build_provider_url(). The router upgrades - // `:rawPredict` to `:streamRawPredict` only for streaming proxy calls. - let endpoint = format!( - "https://{host}/v1/projects/{project}/locations/{location}/publishers/anthropic/models" - ); - let protocols = vec!["anthropic_messages".to_string()]; - Ok(build_vertex_route( - route_name, - endpoint, - model_id, - api_key, - protocols, - profile, - true, - Some(":rawPredict".to_string()), - )) - } else { - // OpenAI-compatible endpoint for all non-Anthropic models - // (Gemini, Llama, Mistral, unknown, etc.). Vertex's OpenAI-compatible - // surface uses `/chat/completions` under the `.../endpoints/openapi` - // base, so we pin the route to that path instead of appending the - // router's default `/v1/...` protocol path. - let endpoint = format!( - "https://{host}/v1beta1/projects/{project}/locations/{location}/endpoints/openapi" - ); - let protocols = vec!["openai_chat_completions".to_string()]; - Ok(build_vertex_route( - route_name, - endpoint, - &body_model_id, - api_key, - protocols, - profile, - false, - Some("/chat/completions".to_string()), - )) - } -} - -fn resolve_provider_route( - provider: &Provider, - model_id: &str, -) -> Result { - let raw_provider_type = provider.r#type.trim(); - let provider_type = normalize_provider_type(raw_provider_type) - .map_or_else(|| raw_provider_type.to_ascii_lowercase(), str::to_string); - - let profile = openshell_core::inference::profile_for(&provider_type).ok_or_else(|| { - Status::invalid_argument(format!( - "provider '{name}' has unsupported type '{raw_provider_type}' for cluster inference \ - (supported: openai, anthropic, nvidia, deepinfra, google-vertex-ai, aws-bedrock)", - name = provider.object_name() - )) - })?; - - // Profiles with `auth: None` are bridge-fronted — the upstream - // authenticates itself, so the router doesn't need a credential at - // route-resolution time. Today this is `aws-bedrock`. - let api_key = if matches!(profile.auth, openshell_core::inference::AuthHeader::None) { - String::new() - } else { - find_provider_api_key( - provider, - profile.credential_key_names, - if provider_type == "google-vertex-ai" { - CredentialLookup::PreferredOnly - } else { - CredentialLookup::PreferredThenAny - }, - ) - .ok_or_else(|| { - Status::invalid_argument(format!( - "provider '{name}' has no usable API key credential", - name = provider.object_name() - )) - })? - }; - - // Vertex AI requires a model-aware URL; delegate to specialised resolver. - if provider_type == "google-vertex-ai" { - let route = resolve_vertex_ai_route( - &provider.config, - model_id, - provider.object_name(), - &api_key, - profile, - )?; - return Ok(ResolvedProviderRoute { - provider_type, - route, - }); - } - - // AWS Bedrock encodes the model in the URL path - // (`/model//invoke`), so the model id is interpolated directly - // into a path segment by the router. Validate up front so the route - // store cannot hold a model id that would produce ambiguous or - // malformed upstream paths. Defense-in-depth: the router enforces - // the same contract again before constructing an upstream URL. - if provider_type == "aws-bedrock" { - validate_aws_bedrock_model_id(model_id)?; - } - - let base_url = find_provider_config_value(provider, profile.base_url_config_keys) - .unwrap_or_else(|| profile.default_base_url.to_string()) - .trim() - .to_string(); - - if base_url.is_empty() { - return Err(Status::invalid_argument(format!( - "provider '{name}' resolved to empty base_url", - name = provider.object_name() - ))); - } - - Ok(ResolvedProviderRoute { - provider_type, - route: RouterResolvedRoute { - name: provider.object_name().to_string(), - endpoint: base_url, - model: model_id.to_string(), - api_key, - protocols: profile.protocols.iter().map(|p| (*p).to_string()).collect(), - auth: profile.auth.clone(), - default_headers: profile - .default_headers - .iter() - .map(|(name, value)| ((*name).to_string(), (*value).to_string())) - .collect(), - passthrough_headers: profile - .passthrough_headers - .iter() - .map(|name| (*name).to_string()) - .collect(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }, - }) -} - -fn validation_failure( - provider_name: &str, - model_id: &str, - base_url: &str, - details: &str, - next_steps: &str, -) -> Status { - Status::failed_precondition(format!( - "failed to verify inference endpoint for provider '{provider_name}' and model '{model_id}' at '{base_url}': {details}. Next steps: {next_steps}, or retry with '--no-verify' if you want to skip verification" - )) -} - -fn validation_next_steps(kind: ValidationFailureKind) -> &'static str { - match kind { - ValidationFailureKind::Credentials => { - "verify the provider API key and any required auth headers" - } - ValidationFailureKind::RateLimited => { - "retry later or verify quota/limits on the upstream provider" - } - ValidationFailureKind::RequestShape => { - "confirm the provider type, base URL, and model identifier" - } - ValidationFailureKind::Connectivity => { - "check that the service is running, confirm the base URL and protocol, and verify credentials" - } - ValidationFailureKind::UpstreamHealth => { - "check whether the endpoint is healthy and serving requests" - } - ValidationFailureKind::Unexpected => { - "confirm the endpoint URL, protocol, credentials, and model identifier" - } - } -} - -async fn verify_provider_endpoint( - provider_name: &str, - model_id: &str, - route: &ResolvedProviderRoute, -) -> Result { - let client = reqwest::Client::builder() - .timeout(Duration::from_secs(30)) - .build() - .map_err(|err| Status::internal(format!("build validation client failed: {err}")))?; - verify_backend_endpoint(&client, &route.route) - .await - .map(|validated| ValidatedEndpoint { - url: validated.url, - protocol: validated.protocol, - }) - .map_err(|err| { - validation_failure( - provider_name, - model_id, - &route.route.endpoint, - &err.details, - validation_next_steps(err.kind), - ) - }) -} - -/// Controls whether `find_provider_api_key` is allowed to fall back to any -/// non-empty credential when the preferred key names produce no match. -/// -/// `PreferredOnly` is used for providers like Vertex AI where the fallback -/// would pick up JSON bootstrap material (e.g. service account keys) that -/// are not valid bearer tokens. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum CredentialLookup { - /// Only search `preferred_key_names`. Return `None` if none match. - PreferredOnly, - /// Search `preferred_key_names` first, then fall back to any non-empty credential. - PreferredThenAny, -} - -fn find_provider_api_key( - provider: &Provider, - preferred_key_names: &[&str], - lookup: CredentialLookup, -) -> Option { - for key in preferred_key_names { - if let Some(value) = provider.credentials.get(*key) - && !value.trim().is_empty() - { - return Some(value.clone()); - } - } - - if lookup == CredentialLookup::PreferredOnly { - return None; - } - - let mut keys = provider.credentials.keys().collect::>(); - keys.sort(); - for key in keys { - if let Some(value) = provider.credentials.get(key) - && !value.trim().is_empty() - { - return Some(value.clone()); - } - } - - None -} - -fn find_provider_config_value(provider: &Provider, preferred_keys: &[&str]) -> Option { - for key in preferred_keys { - if let Some(value) = provider.config.get(*key) - && !value.trim().is_empty() - { - return Some(value.clone()); - } - } - None -} - -fn authorize_inference_bundle( - principal: Option<&crate::auth::principal::Principal>, -) -> Result { - match principal { - Some(crate::auth::principal::Principal::Sandbox(s)) => Ok(s.sandbox_id.clone()), - Some(crate::auth::principal::Principal::User(_)) => Err(Status::permission_denied( - "GetInferenceBundle requires a sandbox principal", - )), - Some(crate::auth::principal::Principal::Anonymous) | None => Err(Status::unauthenticated( - "GetInferenceBundle requires an authenticated sandbox principal", - )), - } -} - -/// Resolve the inference bundle (all managed routes + revision hash). -#[cfg(test)] -async fn resolve_inference_bundle( - store: &Store, - workspace: &str, -) -> Result { - resolve_inference_bundle_with_credentials(store, workspace, None).await -} - -async fn resolve_inference_bundle_with_credentials( - store: &Store, - workspace: &str, - credentials: Option<&crate::credentials::CredentialRuntime>, -) -> Result { - let mut routes = Vec::new(); - if let Some(r) = resolve_route_by_name_with_credentials( - store, - workspace, - credentials, - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await? - { - routes.push(r); - } - if let Some(r) = resolve_route_by_name_with_credentials( - store, - workspace, - credentials, - SANDBOX_SYSTEM_ROUTE_NAME, - ) - .await? - { - routes.push(r); - } - - let now_ms = i64::try_from( - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_millis(), - ) - .unwrap_or(i64::MAX); - - // Compute a simple revision from route contents for cache freshness checks. - let revision = { - use std::hash::{Hash, Hasher}; - let mut hasher = std::collections::hash_map::DefaultHasher::new(); - for r in &routes { - r.name.hash(&mut hasher); - r.base_url.hash(&mut hasher); - r.model_id.hash(&mut hasher); - r.api_key.hash(&mut hasher); - r.protocols.hash(&mut hasher); - r.provider_type.hash(&mut hasher); - r.timeout_secs.hash(&mut hasher); - r.model_in_path.hash(&mut hasher); - r.request_path_override.hash(&mut hasher); - } - format!("{:016x}", hasher.finish()) - }; - - Ok(GetInferenceBundleResponse { - routes, - revision, - generated_at_ms: now_ms, - }) -} - -#[cfg(test)] -async fn resolve_route_by_name( - store: &Store, - workspace: &str, - route_name: &str, -) -> Result, Status> { - resolve_route_by_name_with_credentials(store, workspace, None, route_name).await -} - -async fn resolve_route_by_name_with_credentials( - store: &Store, - workspace: &str, - credentials: Option<&crate::credentials::CredentialRuntime>, - route_name: &str, -) -> Result, Status> { - let route = store - .get_message_by_name::(workspace, route_name) - .await - .map_err(|e| Status::internal(format!("fetch route failed: {e}")))?; - - let Some(route) = route else { - return Ok(None); - }; - - let Some(config) = route.config.as_ref() else { - return Ok(None); - }; - - if config.provider_name.trim().is_empty() { - return Err(Status::failed_precondition(format!( - "route '{route_name}' is missing provider_name" - ))); - } - - if config.model_id.trim().is_empty() { - return Err(Status::failed_precondition(format!( - "route '{route_name}' is missing model_id" - ))); - } - - let provider = store - .get_message_by_name::(workspace, &config.provider_name) - .await - .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))? - .ok_or_else(|| { - Status::failed_precondition(format!( - "configured provider '{}' was not found in workspace '{workspace}'", - config.provider_name - )) - })?; - let provider = resolve_provider_credentials(provider, credentials).await?; - - let resolved = resolve_provider_route(&provider, &config.model_id)?; - - Ok(Some(ResolvedRoute { - name: route_name.to_string(), - base_url: resolved.route.endpoint, - model_id: resolved.route.model.clone(), - api_key: resolved.route.api_key, - protocols: resolved.route.protocols, - provider_type: resolved.provider_type, - timeout_secs: config.timeout_secs, - model_in_path: resolved.route.model_in_path, - request_path_override: resolved.route.request_path_override, - })) -} - -async fn resolve_provider_credentials( - mut provider: Provider, - credentials: Option<&crate::credentials::CredentialRuntime>, -) -> Result { - if provider.credential_handles.is_empty() { - return Ok(provider); - } - - let credentials = credentials.ok_or_else(|| { - Status::failed_precondition(format!( - "provider '{}' stores credentials as handles, but credential storage is unavailable", - provider.object_name() - )) - })?; - let resolved = credentials - .resolve_provider_handles(&provider, current_time_ms()) - .await?; - provider.credentials.extend(resolved.values); - - // Merge expiration times, keeping the earliest non-zero value - for (key, driver_expires_at_ms) in resolved.expires_at_ms { - let provider_expires_at_ms = provider - .credential_expires_at_ms - .get(&key) - .copied() - .unwrap_or(0); - - let effective_expires_at_ms = match (provider_expires_at_ms, driver_expires_at_ms) { - (0, driver) => driver, - (provider, 0) => provider, - (provider, driver) => provider.min(driver), - }; - - if effective_expires_at_ms > 0 { - provider - .credential_expires_at_ms - .insert(key, effective_expires_at_ms); - } - } - Ok(provider) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::auth::identity::{Identity, IdentityProvider}; - use crate::auth::principal::{ - Principal, SandboxIdentitySource, SandboxPrincipal, UserPrincipal, - }; - use openshell_core::ObjectId; - use wiremock::matchers::{body_partial_json, header, method, path}; - use wiremock::{Mock, MockServer, ResponseTemplate}; - - async fn test_store() -> Store { - crate::persistence::test_store().await - } - - fn test_user_principal() -> Principal { - Principal::User(UserPrincipal { - identity: Identity { - subject: "user-a".to_string(), - display_name: None, - roles: vec!["openshell-user".to_string()], - scopes: vec![], - provider: IdentityProvider::Oidc, - }, - }) - } - - fn test_sandbox_principal() -> Principal { - Principal::Sandbox(SandboxPrincipal { - sandbox_id: "sandbox-a".to_string(), - source: SandboxIdentitySource::BootstrapJwt { - issuer: "openshell-gateway:test".to_string(), - }, - trust_domain: Some("openshell".to_string()), - }) - } - - fn make_route(name: &str, provider_name: &str, model_id: &str) -> InferenceRoute { - InferenceRoute { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("id-{name}"), - name: name.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - config: Some(InferenceRouteConfig { - provider_name: provider_name.to_string(), - model_id: model_id.to_string(), - timeout_secs: 0, - }), - version: 0, - } - } - - fn make_provider(name: &str, provider_type: &str, key_name: &str, key_value: &str) -> Provider { - Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("provider-{name}"), - name: name.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: provider_type.to_string(), - credentials: std::iter::once((key_name.to_string(), key_value.to_string())).collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - } - } - - fn make_provider_with_base_url( - name: &str, - provider_type: &str, - key_name: &str, - key_value: &str, - base_url_key: &str, - base_url: &str, - ) -> Provider { - Provider { - config: std::iter::once((base_url_key.to_string(), base_url.to_string())).collect(), - ..make_provider(name, provider_type, key_name, key_value) - } - } - - #[test] - fn resolve_vertex_ai_route_handles_empty_base_url_keys() { - let config = HashMap::new(); - let profile = openshell_core::inference::InferenceProviderProfile { - provider_type: "google_vertex_ai", - default_base_url: "https://example.com", - protocols: &[], - credential_key_names: &[], - base_url_config_keys: &[], - auth: openshell_core::inference::AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &[], - }; - let result = resolve_vertex_ai_route(&config, "model-id", "route", "api-key", &profile); - // Empty base_url_config_keys must not panic. The route still errors because - // the minimal Vertex config is missing, so we only assert reachability. - assert!( - result.is_err(), - "expected missing config to error, got {result:?}" - ); - } - - #[test] - fn resolve_vertex_ai_route_skips_blank_preferred_for_fallback() { - let mut config = HashMap::new(); - config.insert("GOOGLE_VERTEX_AI_BASE_URL".to_string(), " ".to_string()); - config.insert( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com".to_string(), - ); - config.insert( - VERTEX_AI_PROJECT_ID_KEY.to_string(), - "my-project".to_string(), - ); - config.insert(VERTEX_AI_REGION_KEY.to_string(), "us-central1".to_string()); - - let profile = openshell_core::inference::InferenceProviderProfile { - provider_type: "google-vertex-ai", - default_base_url: "", - protocols: &[], - credential_key_names: &[], - base_url_config_keys: &["GOOGLE_VERTEX_AI_BASE_URL", "VERTEX_AI_BASE_URL"], - auth: openshell_core::inference::AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &[], - }; - - let result = resolve_vertex_ai_route(&config, "model-id", "route", "api-key", &profile); - assert!( - result.is_ok(), - "blank preferred key should fall back to valid alias: {result:?}" - ); - let route = result.unwrap(); - assert_eq!( - route.endpoint, - "https://us-central1-aiplatform.googleapis.com" - ); - } - - #[test] - fn inference_bundle_requires_sandbox_principal() { - let sandbox = test_sandbox_principal(); - assert!(authorize_inference_bundle(Some(&sandbox)).is_ok()); - - let user = test_user_principal(); - let err = authorize_inference_bundle(Some(&user)).expect_err("users cannot fetch bundle"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - - let err = authorize_inference_bundle(None).expect_err("missing principal rejected"); - assert_eq!(err.code(), tonic::Code::Unauthenticated); - } - - #[tokio::test] - async fn upsert_cluster_route_creates_and_increments_version() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let first = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - .expect("first set should succeed"); - assert_eq!(first.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); - - let second = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4.1", - 0, - false, - ) - .await - .expect("second set should succeed"); - assert_eq!(second.route.object_id(), first.route.object_id()); - - let config = second.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "openai-dev"); - assert_eq!(config.model_id, "gpt-4.1"); - } - - #[tokio::test] - async fn upsert_cluster_route_succeeds_for_aws_bedrock_with_bridge_url() { - // aws-bedrock is registered with `auth: AuthHeader::None` (the - // bridge-fronted shape) so route resolution does NOT require a - // real API key — but `provider create` still requires a - // non-empty credentials map at the gRPC layer, so operators - // pass a placeholder credential per the docs. The router - // ignores it on the outbound path. - // - // The other half of the contract is `BEDROCK_BASE_URL`: with - // `default_base_url: ""` in the core profile, providers - // without it fail route resolution rather than silently - // forwarding prompts to AWS Bedrock with no usable auth. This - // test pins down the success path. - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-bedrock-bridge".to_string(), - name: "bedrock-bridge".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws-bedrock".to_string(), - // Placeholder credential — the router ignores it because - // auth: None skips header injection. Mirrors the - // doc-recommended `--credential AWS_ACCESS_KEY_ID=unused-bridge-fronted-shape`. - credentials: std::iter::once(( - "AWS_ACCESS_KEY_ID".to_string(), - "unused-bridge-fronted-shape".to_string(), - )) - .collect(), - config: std::iter::once(( - "BEDROCK_BASE_URL".to_string(), - "http://bedrock-bridge.demo.svc.cluster.local:8080".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let upserted = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "bedrock-bridge", - "anthropic.claude-3-5-sonnet-20241022-v2:0", - 0, - false, - ) - .await - .expect("upsert should succeed for aws-bedrock provider"); - - assert_eq!(upserted.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); - let config = upserted.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "bedrock-bridge"); - assert_eq!(config.model_id, "anthropic.claude-3-5-sonnet-20241022-v2:0"); - - // Verify the resolved route metadata reflects bridge-fronted - // auth (empty api_key + provider_type = "aws-bedrock"). Note - // the api_key is empty even though the provider has a - // credential — auth: None skips api-key lookup entirely. - let managed = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - assert_eq!(managed.provider_type, "aws-bedrock"); - assert_eq!( - managed.base_url, - "http://bedrock-bridge.demo.svc.cluster.local:8080" - ); - assert_eq!(managed.api_key, ""); - } - - #[tokio::test] - async fn upsert_cluster_route_rejects_aws_bedrock_without_bedrock_base_url() { - // The companion to upsert_cluster_route_succeeds_for_aws_bedrock_with_bridge_url: - // an aws-bedrock provider without BEDROCK_BASE_URL must be - // rejected at route resolution. This pins down the safety - // contract johntmyers asked for — until the SigV4 follow-up - // lands, the router must NOT silently forward prompts to AWS - // with auth: None. - // - // Mechanism: AWS_BEDROCK_PROFILE.default_base_url is "". When - // the provider has no BEDROCK_BASE_URL config, base_url - // resolves to empty, triggering the existing - // empty-base_url check in resolve_provider_route. - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-bedrock-misconfigured".to_string(), - name: "bedrock-misconfigured".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws-bedrock".to_string(), - credentials: std::iter::once(( - "AWS_ACCESS_KEY_ID".to_string(), - "unused-bridge-fronted-shape".to_string(), - )) - .collect(), - // Intentionally no BEDROCK_BASE_URL. - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let err = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "bedrock-misconfigured", - "anthropic.claude-3-5-sonnet-20241022-v2:0", - 0, - false, - ) - .await - .expect_err("upsert should reject aws-bedrock provider without BEDROCK_BASE_URL"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("empty base_url"), - "error should name the missing base_url, got: {}", - err.message() - ); - } - - /// Bedrock route resolution must reject model ids that would - /// produce ambiguous or malformed upstream URL paths. The Vertex - /// suite has equivalent coverage; this is the Bedrock companion. - #[tokio::test] - async fn upsert_cluster_route_rejects_aws_bedrock_unsafe_model_id() { - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-bedrock-bridge".to_string(), - name: "bedrock-bridge".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws-bedrock".to_string(), - credentials: HashMap::new(), - config: std::iter::once(( - "BEDROCK_BASE_URL".to_string(), - "http://bedrock-bridge.demo.svc.cluster.local:8080".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - for unsafe_model in [ - "anthropic.claude/../../etc/passwd", - "back\\slash-id", - "model?injected=1", - "model#fragment", - "percent%2fencoded", - "model..v2", - " leading-space", - "trailing-space ", - "tab\there", - "newline\nhere", - ] { - let err = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "bedrock-bridge", - unsafe_model, - 0, - false, - ) - .await - .expect_err(unsafe_model); - assert_eq!( - err.code(), - tonic::Code::InvalidArgument, - "{unsafe_model:?} should fail with InvalidArgument" - ); - assert!( - err.message().contains("AWS Bedrock model_id"), - "error must name AWS Bedrock model_id for {unsafe_model:?}, got: {}", - err.message() - ); - } - } - - #[tokio::test] - async fn resolve_managed_route_returns_none_when_missing() { - let store = test_store().await; - - let route = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("resolution should not fail"); - assert!(route.is_none()); - } - - #[tokio::test] - async fn bundle_happy_path_returns_managed_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = make_route(CLUSTER_INFERENCE_ROUTE_NAME, "openai-dev", "mock/model-a"); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - assert_eq!(resp.routes[0].name, CLUSTER_INFERENCE_ROUTE_NAME); - assert_eq!(resp.routes[0].model_id, "mock/model-a"); - assert_eq!(resp.routes[0].provider_type, "openai"); - assert_eq!(resp.routes[0].api_key, "sk-test"); - assert_eq!(resp.routes[0].base_url, "https://api.openai.com/v1"); - assert!(!resp.revision.is_empty()); - assert!(resp.generated_at_ms > 0); - } - - #[tokio::test] - async fn bundle_vertex_ai_anthropic_route_preserves_model_path_and_rawpredict() { - let store = test_store().await; - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - store - .put_message(&provider) - .await - .expect("persist provider"); - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-dev", - "claude-3-5-sonnet@20241022", - ); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - let route = &resp.routes[0]; - assert_eq!(route.provider_type, "google-vertex-ai"); - assert_eq!(route.api_key, "ya29.test-token"); - assert_eq!(route.protocols, vec!["anthropic_messages"]); - assert!(route.model_in_path); - assert_eq!(route.request_path_override, Some(":rawPredict".to_string())); - assert_eq!(route.model_id, "claude-3-5-sonnet@20241022"); - assert_eq!( - route.base_url, - "https://us-central1-aiplatform.googleapis.com/v1/projects/my-gcp-project/locations/us-central1/publishers/anthropic/models" - ); - } - - #[tokio::test] - async fn bundle_vertex_ai_gemini_route_preserves_chat_completions_override() { - let store = test_store().await; - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - store - .put_message(&provider) - .await - .expect("persist provider"); - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-dev", - "gemini-2.0-flash-001", - ); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - let route = &resp.routes[0]; - assert_eq!(route.provider_type, "google-vertex-ai"); - assert_eq!(route.api_key, "ya29.test-token"); - assert_eq!(route.protocols, vec!["openai_chat_completions"]); - assert!(!route.model_in_path); - assert_eq!( - route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert_eq!(route.model_id, "google/gemini-2.0-flash-001"); - assert_eq!( - route.base_url, - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-gcp-project/locations/us-central1/endpoints/openapi" - ); - } - - #[tokio::test] - async fn bundle_vertex_ai_non_anthropic_model_id_carries_publisher_prefix() { - // Regression test: the bundle's model_id must carry the publisher prefix - // so the router sends e.g. "google/gemini-2.5-flash" in the request body, - // not the bare "gemini-2.5-flash" that Vertex AI rejects with HTTP 400. - let store = test_store().await; - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - store - .put_message(&provider) - .await - .expect("persist provider"); - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-dev", - "gemini-2.5-flash", - ); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - assert_eq!( - resp.routes[0].model_id, "google/gemini-2.5-flash", - "bundle model_id must carry publisher prefix for non-Anthropic Vertex routes" - ); - } - - #[tokio::test] - async fn bundle_without_cluster_route_returns_empty_routes() { - let store = test_store().await; - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - assert!(resp.routes.is_empty()); - } - - #[tokio::test] - async fn bundle_revision_is_stable_for_same_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "mock/model-stable", - ); - store.put_message(&route).await.expect("persist route"); - - let resp1 = resolve_inference_bundle(&store, "default") - .await - .expect("first resolve"); - let resp2 = resolve_inference_bundle(&store, "default") - .await - .expect("second resolve"); - - assert_eq!( - resp1.revision, resp2.revision, - "same route should produce same revision" - ); - } - - #[tokio::test] - async fn resolve_managed_route_derives_from_provider() { - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-1".to_string(), - name: "openai-dev".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "openai".to_string(), - credentials: std::iter::once(("OPENAI_API_KEY".to_string(), "sk-test".to_string())) - .collect(), - config: std::iter::once(( - "OPENAI_BASE_URL".to_string(), - "https://station.example.com/v1".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let route = InferenceRoute { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "r-1".to_string(), - name: CLUSTER_INFERENCE_ROUTE_NAME.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - config: Some(InferenceRouteConfig { - provider_name: "openai-dev".to_string(), - model_id: "test/model".to_string(), - timeout_secs: 0, - }), - version: 1, - }; - store - .put_message(&route) - .await - .expect("route should persist"); - - let managed = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - - assert_eq!(managed.base_url, "https://station.example.com/v1"); - assert_eq!(managed.api_key, "sk-test"); - assert_eq!(managed.provider_type, "openai"); - assert_eq!( - managed.protocols, - vec![ - "openai_chat_completions".to_string(), - "openai_completions".to_string(), - "openai_responses".to_string(), - "openai_embeddings".to_string(), - "model_discovery".to_string(), - ] - ); - } - - #[tokio::test] - async fn managed_route_resolves_default_credential_handles() { - let store = test_store().await; - let credentials = crate::credentials::CredentialRuntime::from_config_with_store( - &openshell_core::Config::new(None), - Arc::new(store.clone()), - ) - .expect("credential runtime should connect to default encrypted store"); - let handles = credentials - .store_provider_credentials( - "openai-dev", - "default", - "provider-1", - &HashMap::from([("OPENAI_API_KEY".to_string(), "sk-encrypted".to_string())]), - &HashMap::new(), - ) - .await - .expect("credential should be stored"); - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-1".to_string(), - name: "openai-dev".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - workspace: "default".to_string(), - ..Default::default() - }), - r#type: "openai".to_string(), - credentials: HashMap::new(), - config: std::iter::once(( - "OPENAI_BASE_URL".to_string(), - "https://station.example.com/v1".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - credential_handles: handles, - profile_workspace: String::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - upsert_cluster_inference_route_with_credentials( - &store, - "default", - Some(&credentials), - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "test/model", - 0, - false, - ) - .await - .expect("route should be created from handle-backed provider"); - - let managed = resolve_route_by_name_with_credentials( - &store, - "default", - Some(&credentials), - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("route should resolve") - .expect("managed route should exist"); - - assert_eq!(managed.base_url, "https://station.example.com/v1"); - assert_eq!(managed.api_key, "sk-encrypted"); - } - - #[tokio::test] - async fn resolve_managed_route_reflects_provider_key_rotation() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-initial"); - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let route = make_route(CLUSTER_INFERENCE_ROUTE_NAME, "openai-dev", "test/model"); - store - .put_message(&route) - .await - .expect("route should persist"); - - let first = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - assert_eq!(first.api_key, "sk-initial"); - - let rotated_provider = Provider { - metadata: provider.metadata.clone(), - r#type: provider.r#type.clone(), - credentials: std::iter::once(("OPENAI_API_KEY".to_string(), "sk-rotated".to_string())) - .collect(), - config: provider.config.clone(), - credential_expires_at_ms: provider.credential_expires_at_ms.clone(), - profile_workspace: provider.profile_workspace.clone(), - credential_handles: HashMap::new(), - }; - store - .put_message(&rotated_provider) - .await - .expect("provider rotation should persist"); - - let second = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - assert_eq!(second.api_key, "sk-rotated"); - } - - #[tokio::test] - async fn upsert_system_route_creates_with_correct_name() { - let store = test_store().await; - - let provider = make_provider("anthropic-dev", "anthropic", "ANTHROPIC_API_KEY", "sk-ant"); - store.put_message(&provider).await.expect("persist"); - - let route = upsert_inference_route( - &store, - "default", - SANDBOX_SYSTEM_ROUTE_NAME, - "anthropic-dev", - "claude-sonnet-4-20250514", - 0, - false, - ) - .await - .expect("should succeed"); - - assert_eq!(route.route.object_name(), SANDBOX_SYSTEM_ROUTE_NAME); - let config = route.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "anthropic-dev"); - assert_eq!(config.model_id, "claude-sonnet-4-20250514"); - } - - #[tokio::test] - async fn upsert_inference_route_vertex_ai_anthropic_sets_model_in_path() { - let store = test_store().await; - - // Build a Vertex AI provider with the required config and a minted access token. - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-vertex-test".to_string(), - name: "vertex-test".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "google-vertex-ai".to_string(), - credentials: std::iter::once(( - "GOOGLE_VERTEX_AI_TOKEN".to_string(), - "ya29.test-access-token".to_string(), - )) - .collect(), - config: [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("persist provider"); - - let result = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-test", - "claude-3-5-sonnet@20241022", - 0, - false, // skip verification — no live endpoint - ) - .await - .expect("upsert should succeed for Vertex AI Anthropic model"); - - // Confirm the route was persisted with correct metadata - assert_eq!(result.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); - let config = result.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "vertex-test"); - assert_eq!(config.model_id, "claude-3-5-sonnet@20241022"); - - // Resolve the persisted route and assert Vertex AI Anthropic path contract - let resolved = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("resolve should not fail") - .expect("route should exist after upsert"); - - assert!( - resolved.model_in_path, - "Anthropic-on-Vertex routes must set model_in_path=true" - ); - assert_eq!( - resolved.request_path_override, - Some(":rawPredict".to_string()), - "Anthropic-on-Vertex routes must persist the rawPredict suffix" - ); - assert_eq!(resolved.provider_type, "google-vertex-ai"); - assert!( - resolved.base_url.contains("publishers/anthropic/models"), - "endpoint must end with /publishers/anthropic/models, got: {}", - resolved.base_url - ); - assert!( - !resolved.base_url.contains("claude-3-5-sonnet"), - "model_id must not be embedded in the endpoint, got: {}", - resolved.base_url - ); - } - - #[tokio::test] - async fn bundle_includes_both_user_and_system_routes() { - let store = test_store().await; - - let openai = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-oai"); - store.put_message(&openai).await.expect("persist openai"); - let anthropic = make_provider("anthropic-dev", "anthropic", "ANTHROPIC_API_KEY", "sk-ant"); - store - .put_message(&anthropic) - .await - .expect("persist anthropic"); - - let user_route = make_route(CLUSTER_INFERENCE_ROUTE_NAME, "openai-dev", "gpt-4o"); - store - .put_message(&user_route) - .await - .expect("persist user route"); - let system_route = make_route( - SANDBOX_SYSTEM_ROUTE_NAME, - "anthropic-dev", - "claude-sonnet-4-20250514", - ); - store - .put_message(&system_route) - .await - .expect("persist system route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 2); - assert_eq!(resp.routes[0].name, CLUSTER_INFERENCE_ROUTE_NAME); - assert_eq!(resp.routes[0].model_id, "gpt-4o"); - assert_eq!(resp.routes[1].name, SANDBOX_SYSTEM_ROUTE_NAME); - assert_eq!(resp.routes[1].model_id, "claude-sonnet-4-20250514"); - } - - #[tokio::test] - async fn bundle_with_only_system_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - let system_route = make_route(SANDBOX_SYSTEM_ROUTE_NAME, "openai-dev", "gpt-4o-mini"); - store.put_message(&system_route).await.expect("persist"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - assert_eq!(resp.routes[0].name, SANDBOX_SYSTEM_ROUTE_NAME); - assert_eq!(resp.routes[0].model_id, "gpt-4o-mini"); - } - - #[tokio::test] - async fn get_returns_system_route_when_requested() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - upsert_inference_route( - &store, - "default", - SANDBOX_SYSTEM_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - false, - ) - .await - .expect("upsert should succeed"); - - let route = store - .get_message_by_name::("default", SANDBOX_SYSTEM_ROUTE_NAME) - .await - .expect("fetch should succeed") - .expect("route should exist"); - - assert_eq!(route.object_name(), SANDBOX_SYSTEM_ROUTE_NAME); - let config = route.config.as_ref().expect("config"); - assert_eq!(config.model_id, "gpt-4o-mini"); - } - - #[tokio::test] - async fn upsert_cluster_route_verifies_endpoint_when_requested() { - let store = test_store().await; - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(header("authorization", "Bearer sk-test")) - .and(header("content-type", "application/json")) - .and(body_partial_json(serde_json::json!({ - "model": "gpt-4o-mini", - "max_completion_tokens": 32, - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "chatcmpl-123", - "object": "chat.completion", - "choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}], - "model": "gpt-4o-mini" - }))) - .mount(&mock_server) - .await; - - let provider = make_provider_with_base_url( - "openai-dev", - "openai", - "OPENAI_API_KEY", - "sk-test", - "OPENAI_BASE_URL", - &mock_server.uri(), - ); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - true, - ) - .await - .expect("validation should succeed"); - - assert_eq!(route.route.version, 1); - assert_eq!(route.validation.len(), 1); - assert_eq!(route.validation[0].protocol, "openai_chat_completions"); - } - - #[tokio::test] - async fn upsert_cluster_route_rejects_failed_validation() { - let store = test_store().await; - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(401).set_body_string("bad key")) - .mount(&mock_server) - .await; - - let provider = make_provider_with_base_url( - "openai-dev", - "openai", - "OPENAI_API_KEY", - "sk-test", - "OPENAI_BASE_URL", - &mock_server.uri(), - ); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let err = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - true, - ) - .await - .expect_err("validation should fail"); - - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert!( - err.message() - .contains("failed to verify inference endpoint") - ); - assert!(err.message().contains("verify the provider API key")); - assert!(err.message().contains("--no-verify")); - - let persisted = store - .get_message_by_name::("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch route") - .is_none(); - assert!(persisted, "route should not persist on failed validation"); - } - - #[tokio::test] - async fn upsert_cluster_route_skips_validation_by_default() { - let store = test_store().await; - let provider = make_provider_with_base_url( - "openai-dev", - "openai", - "OPENAI_API_KEY", - "sk-test", - "OPENAI_BASE_URL", - "http://127.0.0.1:9", - ); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - false, - ) - .await - .expect("non-verified route should persist"); - - assert_eq!(route.route.version, 1); - assert!(route.validation.is_empty()); - } - - // ------------------------------------------------------------------------- - // infer_vertex_publisher tests - // ------------------------------------------------------------------------- - - #[test] - fn infer_vertex_publisher_anthropic() { - assert_eq!( - infer_vertex_publisher("claude-3-5-sonnet@20241022"), - Some("anthropic") - ); - assert_eq!(infer_vertex_publisher("claude-opus-4"), Some("anthropic")); - } - - #[test] - fn infer_vertex_publisher_gemini() { - assert_eq!(infer_vertex_publisher("gemini-pro"), Some("google")); - assert_eq!(infer_vertex_publisher("gemini-1.5-flash"), Some("google")); - assert_eq!(infer_vertex_publisher("text-bison-001"), Some("google")); - assert_eq!(infer_vertex_publisher("chat-bison-001"), Some("google")); - } - - #[test] - fn infer_vertex_publisher_unknown() { - assert_eq!(infer_vertex_publisher("some-unknown-model"), None); - assert_eq!(infer_vertex_publisher("gpt-4o"), None); - } - - #[test] - fn infer_vertex_publisher_other_publishers() { - assert_eq!(infer_vertex_publisher("llama-3-70b"), Some("meta")); - assert_eq!(infer_vertex_publisher("mistral-large"), Some("mistralai")); - assert_eq!(infer_vertex_publisher("codestral-22b"), Some("mistralai")); - assert_eq!(infer_vertex_publisher("jamba-1.5-large"), Some("ai21")); - assert_eq!(infer_vertex_publisher("deepseek-r1"), Some("deepseek")); - } - - // ------------------------------------------------------------------------- - // resolve_vertex_ai_route tests - // ------------------------------------------------------------------------- - - fn make_vertex_provider_with_config(name: &str, config: HashMap) -> Provider { - Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("provider-{name}"), - name: name.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 1, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "google-vertex-ai".to_string(), - credentials: std::iter::once(( - "GOOGLE_VERTEX_AI_TOKEN".to_string(), - "ya29.test-token".to_string(), - )) - .collect(), - config, - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - } - } - - #[test] - fn resolve_vertex_ai_route_anthropic_model() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "my-project".to_string()), - ("VERTEX_AI_REGION".to_string(), "us-east1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect("should resolve"); - - assert_eq!(resolved.provider_type, "google-vertex-ai"); - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - // model_id must NOT be embedded in the endpoint — it travels via route.model - assert!( - !resolved.route.endpoint.contains("claude-3-5-sonnet"), - "model_id must not be in endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .endpoint - .ends_with("/publishers/anthropic/models"), - "endpoint should end with /publishers/anthropic/models, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .endpoint - .starts_with("https://us-east1-aiplatform.googleapis.com/"), - "expected regional Vertex host, got: {}", - resolved.route.endpoint - ); - assert!( - resolved.route.endpoint.contains("my-project"), - "expected project in URL" - ); - assert!( - resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "expected anthropic_messages protocol" - ); - assert_eq!(resolved.route.model, "claude-3-5-sonnet@20241022"); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom", config); - - let resolved = resolve_provider_route(&provider, "any-model").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi" - ); - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert_eq!( - resolved.route.protocols, - vec!["openai_chat_completions".to_string()] - ); - assert_eq!(resolved.route.model, "any-model"); - } - - #[test] - fn resolve_vertex_ai_route_google_prefixed_base_url_override() { - // GOOGLE_VERTEX_AI_BASE_URL (the preferred key) must work on its own. - let config = std::iter::once(( - "GOOGLE_VERTEX_AI_BASE_URL".to_string(), - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-google", config); - - let resolved = resolve_provider_route(&provider, "any-model").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi" - ); - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_priority_google_wins() { - // When both override keys are set, GOOGLE_VERTEX_AI_BASE_URL takes priority. - let config = [ - ( - "GOOGLE_VERTEX_AI_BASE_URL".to_string(), - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi".to_string(), - ), - ( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - ), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-priority", config); - - let resolved = resolve_provider_route(&provider, "any-model").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi", - "GOOGLE_VERTEX_AI_BASE_URL must win over VERTEX_AI_BASE_URL" - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_anthropic_models() { - let config = std::iter::once(( - "GOOGLE_VERTEX_AI_BASE_URL".to_string(), - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-anthropic", config); - - let err = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect_err("anthropic overrides should fail closed"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("base URL overrides are not supported") - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_non_vertex_host() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://custom.example.com/v1".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-invalid-host", config); - - let err = resolve_provider_route(&provider, "gemini-pro") - .expect_err("non-Vertex hosts must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("must target an official Vertex AI hostname") - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_non_https() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "http://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-http", config); - - let err = resolve_provider_route(&provider, "gemini-pro") - .expect_err("non-https overrides must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("must use https")); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_ip_literal() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://127.0.0.1/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi" - .to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-ip", config); - - let err = resolve_provider_route(&provider, "gemini-pro") - .expect_err("IP literal overrides must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("must not use IP literal hosts")); - } - - #[test] - fn resolve_vertex_ai_route_gemini_model() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-123".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-gemini", config); - - let resolved = resolve_provider_route(&provider, "gemini-pro").expect("should resolve"); - - // Gemini routes to OpenAI-compatible endpoint, not publisher endpoint - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert!( - resolved.route.endpoint.contains("v1beta1"), - "gemini should use v1beta1 endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved.route.endpoint.contains("endpoints/openapi"), - "gemini should use openapi endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - !resolved.route.endpoint.contains("publishers/google"), - "gemini must not embed publisher in endpoint, got: {}", - resolved.route.endpoint - ); - // Default region - assert!(resolved.route.endpoint.contains("us-central1")); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()), - "expected openai_chat_completions protocol" - ); - assert!( - !resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "must not have anthropic_messages protocol for gemini" - ); - // Vertex OpenAI-compatible endpoint requires publisher prefix in body model field - assert_eq!( - resolved.route.model, "google/gemini-pro", - "Vertex non-Anthropic body model must carry publisher prefix" - ); - } - - #[test] - fn resolve_vertex_ai_route_unknown_model_uses_openai_compat() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-compat", config); - - let resolved = - resolve_provider_route(&provider, "some-unknown-model").expect("should resolve"); - - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert!( - resolved.route.endpoint.contains("v1beta1"), - "unknown model should use v1beta1 endpoint" - ); - assert!( - resolved.route.endpoint.contains("endpoints/openapi"), - "unknown model should use openapi endpoint" - ); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()), - "expected openai_chat_completions protocol for unknown model" - ); - assert!( - !resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "must not have anthropic_messages for unknown model" - ); - // Unknown models have no inferred publisher; body model ID is unchanged - assert_eq!(resolved.route.model, "some-unknown-model"); - } - - #[test] - fn resolve_vertex_ai_route_non_anthropic_publisher_prefix_gemini() { - // Gemini models must get `google/` in route.model so the - // OpenAI-compatible Vertex endpoint accepts the request body. - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-gemini-flash", config); - - let resolved = - resolve_provider_route(&provider, "gemini-2.5-flash").expect("should resolve"); - - assert_eq!(resolved.route.model, "google/gemini-2.5-flash"); - } - - #[test] - fn resolve_vertex_ai_route_non_anthropic_publisher_prefix_llama() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-llama", config); - - let resolved = resolve_provider_route(&provider, "llama-3-70b").expect("should resolve"); - - assert_eq!(resolved.route.model, "meta/llama-3-70b"); - } - - #[test] - fn resolve_vertex_ai_route_explicit_publisher_overrides_inference() { - // VERTEX_AI_PUBLISHER takes precedence over infer_vertex_publisher for - // unknown model names. - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "acme".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-explicit", config); - - let resolved = - resolve_provider_route(&provider, "some-acme-model").expect("should resolve"); - - assert_eq!(resolved.route.model, "acme/some-acme-model"); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_gemini_gets_publisher_prefix() { - // Publisher prefix must also be applied when a base URL override is used. - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-base-url-gemini", config); - - let resolved = - resolve_provider_route(&provider, "gemini-2.0-flash").expect("should resolve"); - - assert_eq!(resolved.route.model, "google/gemini-2.0-flash"); - } - - #[test] - fn resolve_vertex_ai_route_global_region_uses_global_host() { - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "proj-global".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "GLOBAL".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-global", config); - - let resolved = - resolve_provider_route(&provider, "claude-opus-4-7").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.googleapis.com/v1/projects/proj-global/locations/global/publishers/anthropic/models" - ); - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_us_multiregion_uses_rep_host() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-us".to_string()), - ("VERTEX_AI_REGION".to_string(), "us".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-us", config); - - let resolved = resolve_provider_route(&provider, "gemini-pro").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.us.rep.googleapis.com/v1beta1/projects/proj-us/locations/us/endpoints/openapi" - ); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_eu_multiregion_uses_rep_host() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-eu".to_string()), - ("VERTEX_AI_REGION".to_string(), "eu".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-eu", config); - - let resolved = resolve_provider_route(&provider, "gemini-pro").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.eu.rep.googleapis.com/v1beta1/projects/proj-eu/locations/eu/endpoints/openapi" - ); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_explicit_publisher_anthropic_override() { - // Explicit VERTEX_AI_PUBLISHER=anthropic → Anthropic Messages API path - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "my-proj".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-pub-anthropic", config); - - let resolved = resolve_provider_route(&provider, "some-model").expect("should resolve"); - - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - assert!( - resolved - .route - .endpoint - .ends_with("/publishers/anthropic/models"), - "expected anthropic publisher endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - !resolved.route.endpoint.contains("some-model"), - "model must not be in endpoint" - ); - assert!( - resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "expected anthropic_messages protocol" - ); - } - - #[test] - fn resolve_vertex_ai_route_explicit_publisher_non_anthropic_uses_openai_compat() { - // Explicit VERTEX_AI_PUBLISHER=google (any non-anthropic) → OpenAI-compat endpoint - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "my-proj".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "google".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-pub-google", config); - - let resolved = resolve_provider_route(&provider, "some-model").expect("should resolve"); - - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert!( - resolved.route.endpoint.contains("v1beta1"), - "non-anthropic publisher should use v1beta1 endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved.route.endpoint.contains("endpoints/openapi"), - "non-anthropic publisher should use openapi endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - !resolved.route.endpoint.contains("publishers/google"), - "must not embed publisher in endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()), - "expected openai_chat_completions for non-anthropic publisher" - ); - } - - #[test] - fn resolve_vertex_ai_route_missing_project_fails() { - let config = HashMap::new(); - let provider = make_vertex_provider_with_config("vertex-no-proj", config); - - let err = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect_err("should fail without project"); - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert!(err.message().contains("VERTEX_AI_PROJECT_ID")); - } - - #[test] - fn resolve_vertex_ai_route_whitespace_only_project_fails() { - // required_vertex_config rejects whitespace-only values via .filter(|v| !v.trim().is_empty()) - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), " ".to_string()), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let result = resolve_vertex_ai_route( - &config, - "claude-3-5-sonnet@20241022", - "test-route", - "dummy-token", - openshell_core::inference::profile_for("google-vertex-ai").unwrap(), - ); - assert!( - result.is_err(), - "whitespace-only project should fail, got: {result:?}" - ); - let status = result.unwrap_err(); - assert_eq!(status.code(), tonic::Code::FailedPrecondition); - } - - #[test] - fn resolve_vertex_ai_route_requires_minted_access_token() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let provider = Provider { - credentials: std::iter::once(( - "GOOGLE_SERVICE_ACCOUNT_KEY".to_string(), - "{\"type\":\"service_account\"}".to_string(), - )) - .collect(), - config, - ..make_vertex_provider_with_config("vertex-bootstrap-only", HashMap::new()) - }; - - let err = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect_err("bootstrap JSON must not be treated as a bearer token"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("no usable API key credential")); - } - - #[test] - fn resolve_vertex_ai_route_alias_canonicalizes_provider_type() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let mut provider = make_vertex_provider_with_config("vertex-alias", config); - provider.r#type = "vertex-ai".to_string(); - - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect("alias should resolve through Vertex routing"); - - assert_eq!(resolved.provider_type, "google-vertex-ai"); - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_anthropic_protocols() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let provider = make_vertex_provider_with_config("v", config); - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022").unwrap(); - assert!( - resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()) - ); - assert!( - !resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()) - ); - assert_eq!( - resolved.route.protocols, - vec!["anthropic_messages".to_string()] - ); - } - - #[test] - fn resolve_vertex_ai_route_openai_compat_protocols() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let provider = make_vertex_provider_with_config("v", config); - let resolved = resolve_provider_route(&provider, "gemini-pro").unwrap(); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()) - ); - assert!( - resolved - .route - .protocols - .iter() - .all(|protocol| protocol == "openai_chat_completions") - ); - } - - #[test] - fn resolve_vertex_ai_route_model_not_in_endpoint() { - // model_id must NOT appear in the endpoint URL — it travels via route.model - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_REGION".to_string(), "us-east1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("v", config); - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022").unwrap(); - assert!( - !resolved.route.endpoint.contains("claude-3-5-sonnet"), - "model_id must not be in endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .endpoint - .ends_with("/publishers/anthropic/models") - ); - } - - #[test] - fn resolve_vertex_ai_route_rejects_model_ids_with_path_separators() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-bad-model", config); - - let err = resolve_provider_route(&provider, "claude/3-sonnet") - .expect_err("path-like model IDs must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("must not contain path separators")); - } - - #[test] - fn resolve_vertex_ai_route_rejects_model_ids_with_url_delimiters() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-bad-model-url", config); - - for model_id in ["claude?alt=1", "claude#fragment", "claude%2Fbad"] { - let err = resolve_provider_route(&provider, model_id) - .expect_err("URL delimiter-bearing model IDs must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("must not contain URL delimiters or percent escapes"), - "unexpected error for {model_id:?}: {}", - err.message() - ); - } - } - - #[test] - fn resolve_vertex_ai_route_accepts_versioned_claude_model_id() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-good-model", config); - - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect("versioned Claude model IDs must remain valid"); - - assert!(resolved.route.model_in_path); - assert_eq!(resolved.route.model, "claude-3-5-sonnet@20241022"); - } - - #[test] - fn resolve_vertex_ai_route_rejects_model_ids_with_whitespace() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-bad-model-whitespace", config); - - let err = resolve_provider_route(&provider, "some model") - .expect_err("whitespace in Anthropic Vertex model IDs must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("must not contain whitespace or control characters") - ); - } - - #[test] - fn validate_gcp_project_id_accepts_valid() { - assert!(validate_gcp_project_id("my-project").is_ok()); - assert!(validate_gcp_project_id("my-project-123").is_ok()); - assert!(validate_gcp_project_id("abcdef").is_ok()); // min length 6 - } - - #[test] - fn validate_gcp_project_id_rejects_invalid() { - assert!(validate_gcp_project_id("").is_err()); // empty - assert!(validate_gcp_project_id("ab").is_err()); // too short - assert!(validate_gcp_project_id("../admin").is_err()); // path traversal - assert!(validate_gcp_project_id("MY-PROJECT").is_err()); // uppercase - assert!(validate_gcp_project_id("my-project-").is_err()); // trailing hyphen - assert!(validate_gcp_project_id("1my-project").is_err()); // starts with digit - } - - #[test] - fn validate_gcp_region_accepts_valid() { - assert!(validate_gcp_region("us-central1").is_ok()); - assert!(validate_gcp_region("europe-west4").is_ok()); - assert!(validate_gcp_region("global").is_ok()); - assert!(validate_gcp_region("us").is_ok()); - assert!(validate_gcp_region("eu").is_ok()); - assert!(validate_gcp_region("us-east4-a").is_ok()); // zone-like - } - - #[test] - fn validate_gcp_region_rejects_invalid() { - assert!(validate_gcp_region("").is_err()); - assert!(validate_gcp_region("../../etc").is_err()); // path traversal - assert!(validate_gcp_region("us central1").is_err()); // space - assert!(validate_gcp_region("-us-central1").is_err()); // leading hyphen - assert!(validate_gcp_region("us-central1-").is_err()); // trailing hyphen - } - - // ------------------------------------------------------------------------- - // validate_vertex_base_url edge-case tests - // ------------------------------------------------------------------------- - - #[test] - fn validate_vertex_base_url_rejects_ipv6_literal() { - let err = validate_vertex_base_url( - "https://[::1]/v1beta1/projects/p/locations/l/endpoints/openapi", - ) - .expect_err("IPv6 literals must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("IP literal"), - "expected IP literal error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_userinfo() { - let err = - validate_vertex_base_url("https://user:pass@us-central1-aiplatform.googleapis.com/v1") - .expect_err("userinfo must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("userinfo"), - "expected userinfo error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_query_string() { - let err = - validate_vertex_base_url("https://us-central1-aiplatform.googleapis.com/v1?key=val") - .expect_err("query string must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("query or fragment"), - "expected query/fragment error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_fragment() { - let err = - validate_vertex_base_url("https://us-central1-aiplatform.googleapis.com/v1#section") - .expect_err("fragment must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("query or fragment"), - "expected query/fragment error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_non_443_port() { - let err = validate_vertex_base_url("https://us-central1-aiplatform.googleapis.com:8443/v1") - .expect_err("non-443 port must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("443"), - "expected port 443 error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_model_id_rejects_double_dot_traversal() { - // ".." without a slash should still be rejected as a path traversal segment. - let err = validate_vertex_model_id("model..v2") - .expect_err("double-dot traversal must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("traversal"), - "expected path traversal error, got: {}", - err.message() - ); - } - - /// Bedrock model ids appear as a URL path segment in - /// `/model//invoke`. Mirrors the Vertex validation suite. - #[test] - fn validate_aws_bedrock_model_id_accepts_well_formed_ids() { - // Real Bedrock model ids: provider-prefixed, dotted, hyphenated, - // possibly versioned with `:0` suffix. - validate_aws_bedrock_model_id("anthropic.claude-opus-4-7").expect("dotted id"); - validate_aws_bedrock_model_id("anthropic.claude-3-5-sonnet-20241022-v2:0") - .expect("versioned id"); - validate_aws_bedrock_model_id("meta.llama3-70b-instruct-v1:0").expect("meta id"); - validate_aws_bedrock_model_id("mistral.mixtral-8x7b-instruct-v0:1").expect("mistral id"); - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_empty() { - let err = validate_aws_bedrock_model_id("").expect_err("empty must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("required")); - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_path_separators() { - for value in ["foo/bar", "anthropic.claude/../passwd", "back\\slash"] { - let err = validate_aws_bedrock_model_id(value).expect_err(value); - assert!( - err.message().contains("path separators"), - "expected path-separator error for {value:?}, got: {}", - err.message() - ); - } - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_url_delimiters() { - for value in ["model?injected=1", "model#fragment", "percent%2fencoded"] { - let err = validate_aws_bedrock_model_id(value).expect_err(value); - assert!( - err.message().contains("URL delimiters"), - "expected URL-delimiter error for {value:?}, got: {}", - err.message() - ); - } - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_traversal() { - let err = validate_aws_bedrock_model_id("model..v2") - .expect_err("double-dot traversal must be rejected"); - assert!( - err.message().contains("traversal"), - "expected path traversal error, got: {}", - err.message() - ); - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_whitespace_and_control() { - for value in [ - " leading", - "trailing ", - "in middle", - "tab\tin", - "newline\nin", - ] { - let err = validate_aws_bedrock_model_id(value).expect_err(value); - assert!( - err.message().contains("whitespace") || err.message().contains("control"), - "expected whitespace/control error for {value:?}, got: {}", - err.message() - ); - } - } - - #[test] - fn effective_route_name_defaults_empty_to_inference_local() { - assert_eq!( - effective_route_name("").unwrap(), - CLUSTER_INFERENCE_ROUTE_NAME - ); - assert_eq!( - effective_route_name(" ").unwrap(), - CLUSTER_INFERENCE_ROUTE_NAME - ); - assert_eq!( - effective_route_name("inference.local").unwrap(), - CLUSTER_INFERENCE_ROUTE_NAME - ); - } - - #[test] - fn effective_route_name_accepts_sandbox_system() { - assert_eq!( - effective_route_name("sandbox-system").unwrap(), - SANDBOX_SYSTEM_ROUTE_NAME - ); - } - - #[test] - fn effective_route_name_rejects_unknown_name() { - let err = effective_route_name("unknown-route").unwrap_err(); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - } - - #[tokio::test] - async fn concurrent_upsert_route_create_uses_must_create() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - // Spawn two concurrent upsert calls for the same route (create path) - let store1 = store.clone(); - let handle1 = tokio::spawn(async move { - upsert_inference_route( - &store1, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - }); - - let store2 = store.clone(); - let handle2 = tokio::spawn(async move { - upsert_inference_route( - &store2, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4.1", - 0, - false, - ) - .await - }); - - let result1 = handle1.await.unwrap(); - let result2 = handle2.await.unwrap(); - - // If both tasks observe a missing route before either insert commits, MustCreate - // should let exactly one win. If the scheduler serializes them, the second call - // may legitimately observe the new route and take the update path. - let successes = [&result1, &result2].iter().filter(|r| r.is_ok()).count(); - let failures = [&result1, &result2] - .iter() - .filter(|r| { - r.as_ref().is_err_and(|e| { - // Accept either ABORTED (from CAS) or Internal (from DB unique constraint) - e.code() == tonic::Code::Aborted - || (e.code() == tonic::Code::Internal - && e.message().contains("unique violation")) - }) - }) - .count(); - - assert!( - successes == 1 || successes == 2, - "one racing create should succeed, or both serialized upserts should succeed, got: {result1:?}, {result2:?}" - ); - if successes == 1 { - assert_eq!( - failures, 1, - "the losing racing create should fail, got: {result1:?}, {result2:?}" - ); - } else { - assert_eq!( - failures, 0, - "serialized upserts should not fail, got: {result1:?}, {result2:?}" - ); - let mut versions = [&result1, &result2] - .into_iter() - .map(|result| result.as_ref().expect("success").route.version) - .collect::>(); - versions.sort_unstable(); - assert_eq!( - versions, - vec![1, 2], - "serialized create-then-update should return versions 1 and 2" - ); - } - - // Only one route should exist. - let route = store - .get_message_by_name::("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch") - .expect("route should exist"); - let expected_version = if successes == 1 { 1 } else { 2 }; - assert_eq!(route.version, expected_version); - } - - #[tokio::test] - async fn concurrent_upsert_route_update_uses_cas() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - // Create initial route - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-3.5", - 0, - false, - ) - .await - .expect("initial create should succeed"); - - // Spawn two concurrent updates - let store1 = store.clone(); - let handle1 = tokio::spawn(async move { - upsert_inference_route( - &store1, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - }); - - let store2 = store.clone(); - let handle2 = tokio::spawn(async move { - upsert_inference_route( - &store2, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4.1", - 0, - false, - ) - .await - }); - - let result1 = handle1.await.unwrap(); - let result2 = handle2.await.unwrap(); - - // One should succeed, one may fail with ABORTED due to CAS conflict - let successes = [&result1, &result2].iter().filter(|r| r.is_ok()).count(); - - assert!( - successes >= 1, - "at least one update should succeed, got: {result1:?}, {result2:?}" - ); - - // The route should have one of the new model values and version 2 - let route = store - .get_message_by_name::("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch") - .expect("route should exist"); - let config = route.config.expect("config"); - assert!( - config.model_id == "gpt-4o" || config.model_id == "gpt-4.1", - "model should be one of the updated values, got {}", - config.model_id - ); - assert_ne!( - config.model_id, "gpt-3.5", - "model should not be the original value" - ); - assert!( - route.version >= 2 && route.version <= 3, - "version should be 2 (one update won, one conflicted) or 3 (both succeeded sequentially), got {}", - route.version - ); - } - - // ------------------------------------------------------------------------- - // Workspace isolation tests - // ------------------------------------------------------------------------- - - #[tokio::test] - async fn inference_bundle_resolves_workspace_scoped_route() { - let store = test_store().await; - - let alpha_provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-alpha".to_string(), - name: "openai-alpha".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - annotations: HashMap::new(), - resource_version: 0, - workspace: "alpha".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "openai".to_string(), - credentials: std::iter::once(( - "OPENAI_API_KEY".to_string(), - "sk-alpha-key".to_string(), - )) - .collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&alpha_provider) - .await - .expect("persist alpha provider"); - - let beta_provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-beta".to_string(), - name: "anthropic-beta".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - annotations: HashMap::new(), - resource_version: 0, - workspace: "beta".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "anthropic".to_string(), - credentials: std::iter::once(( - "ANTHROPIC_API_KEY".to_string(), - "sk-beta-key".to_string(), - )) - .collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&beta_provider) - .await - .expect("persist beta provider"); - - upsert_inference_route( - &store, - "alpha", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-alpha", - "gpt-4", - 0, - false, - ) - .await - .expect("set alpha route"); - - upsert_inference_route( - &store, - "beta", - CLUSTER_INFERENCE_ROUTE_NAME, - "anthropic-beta", - "claude-sonnet-4-20250514", - 0, - false, - ) - .await - .expect("set beta route"); - - let alpha_bundle = resolve_inference_bundle(&store, "alpha") - .await - .expect("alpha bundle should resolve"); - assert_eq!(alpha_bundle.routes.len(), 1); - assert_eq!(alpha_bundle.routes[0].api_key, "sk-alpha-key"); - assert_eq!(alpha_bundle.routes[0].model_id, "gpt-4"); - assert_eq!(alpha_bundle.routes[0].provider_type, "openai"); - - let beta_bundle = resolve_inference_bundle(&store, "beta") - .await - .expect("beta bundle should resolve"); - assert_eq!(beta_bundle.routes.len(), 1); - assert_eq!(beta_bundle.routes[0].api_key, "sk-beta-key"); - assert_eq!(beta_bundle.routes[0].model_id, "claude-sonnet-4-20250514"); - assert_eq!(beta_bundle.routes[0].provider_type, "anthropic"); - } - - #[tokio::test] - async fn inference_bundle_empty_for_workspace_without_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("persist provider"); - - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4", - 0, - false, - ) - .await - .expect("set default route"); - - let other_bundle = resolve_inference_bundle(&store, "other-workspace") - .await - .expect("bundle should resolve"); - assert!( - other_bundle.routes.is_empty(), - "workspace with no route should get empty bundle, not inherit from another workspace" - ); - } - - #[tokio::test] - async fn delete_route_removes_existing_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - .expect("upsert should succeed"); - - let deleted = store - .delete_by_name( - InferenceRoute::object_type(), - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("delete should succeed"); - assert!(deleted, "route should have been deleted"); - - let route: Option = store - .get_message_by_name("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch should succeed"); - assert!(route.is_none(), "route should no longer exist"); - } - - #[tokio::test] - async fn delete_route_returns_false_when_not_found() { - let store = test_store().await; - - let deleted = store - .delete_by_name( - InferenceRoute::object_type(), - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("delete should succeed"); - assert!(!deleted, "nothing to delete"); - } - - #[tokio::test] - async fn delete_route_bundle_becomes_empty() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - .expect("upsert should succeed"); - - let bundle = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - assert_eq!(bundle.routes.len(), 1); - - store - .delete_by_name( - InferenceRoute::object_type(), - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("delete should succeed"); - - let bundle = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - assert!( - bundle.routes.is_empty(), - "bundle should be empty after route deletion" - ); - } - - /// Non-member callers must receive `PERMISSION_DENIED` — not `NOT_FOUND` — - /// when targeting a workspace that does not exist. Returning `NOT_FOUND` - /// would create a CWE-203 workspace-name oracle. - #[tokio::test] - async fn non_member_gets_permission_denied_not_workspace_oracle() { - use crate::grpc::test_support::test_server_state; - use crate::inference::InferenceService; - use openshell_core::proto::inference_server::Inference; - - fn non_member_request(inner: T) -> Request { - let mut req = Request::new(inner); - req.extensions_mut().insert(Principal::User(UserPrincipal { - identity: Identity { - subject: "non-member".to_string(), - display_name: None, - roles: vec![], - scopes: vec![], - provider: IdentityProvider::Oidc, - }, - })); - req - } - - let mut state = test_server_state().await; - Arc::get_mut(&mut state).unwrap().admin_role = "openshell-admin".to_string(); - - let svc = InferenceService::new(state.clone()); - - let err = svc - .set_inference_route(non_member_request(SetInferenceRouteRequest { - workspace: "no-such-ws".into(), - ..Default::default() - })) - .await - .unwrap_err(); - assert_eq!( - err.code(), - tonic::Code::PermissionDenied, - "set_inference_route should return PermissionDenied, got {:?}", - err.code() - ); - - let err = svc - .get_inference_route(non_member_request(GetInferenceRouteRequest { - workspace: "no-such-ws".into(), - ..Default::default() - })) - .await - .unwrap_err(); - assert_eq!( - err.code(), - tonic::Code::PermissionDenied, - "get_inference_route should return PermissionDenied, got {:?}", - err.code() - ); - - let err = svc - .delete_inference_route(non_member_request(DeleteInferenceRouteRequest { - workspace: "no-such-ws".into(), - ..Default::default() - })) - .await - .unwrap_err(); - assert_eq!( - err.code(), - tonic::Code::PermissionDenied, - "delete_inference_route should return PermissionDenied, got {:?}", - err.code() - ); - } -} diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index a8c8afdf08..ed72fcf4ac 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -23,7 +23,6 @@ mod defaults; mod gateway_listener; mod grpc; mod http; -mod inference; mod middleware; mod multiplex; mod otel_tracing; diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 3ef774e4cb..21315f9830 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -17,9 +17,7 @@ use hyper_util::{ service::TowerToHyperService, }; use metrics::{counter, histogram}; -use openshell_core::proto::{ - inference_server::InferenceServer, open_shell_server::OpenShellServer, -}; +use openshell_core::proto::open_shell_server::OpenShellServer; use openshell_core::{ Config, proto::{Provider, UpdateProviderRequest}, @@ -52,9 +50,7 @@ use crate::{ auth::principal::{Principal, UserPrincipal}, auth::workspace_authz::{MinWorkspaceRole, authorize_workspace}, gateway_listener::GatewayListenerScope, - http_router, - inference::InferenceService, - service_http_router, + http_router, service_http_router, }; /// Request-ID generator that produces a UUID v4 for each inbound request. @@ -277,8 +273,6 @@ impl MultiplexService { self.state.gateway_interceptors.clone(), Some(self.state.clone()), ); - let inference = InferenceServer::new(InferenceService::new(self.state.clone())) - .max_decoding_message_size(MAX_GRPC_DECODE_SIZE); let authz_policy = self.state.config.oidc.as_ref().map(|oidc| AuthzPolicy { admin_role: oidc.admin_role.clone(), user_role: oidc.user_role.clone(), @@ -286,7 +280,7 @@ impl MultiplexService { }); let authenticator_chain = build_authenticator_chain(&self.state); let grpc_service = AuthGrpcRouter::with_peer_identity( - GrpcRouter::new(openshell, inference), + openshell, authenticator_chain, authz_policy, self.state @@ -920,59 +914,6 @@ where } } -/// Combined gRPC service that routes between `OpenShell` and Inference services -/// based on the request path prefix. -#[derive(Clone)] -pub struct GrpcRouter { - openshell: N, - inference: I, -} - -impl GrpcRouter { - fn new(openshell: N, inference: I) -> Self { - Self { - openshell, - inference, - } - } -} - -const INFERENCE_PATH_PREFIX: &str = "/openshell.inference.v1.Inference/"; - -impl tower::Service> for GrpcRouter -where - N: tower::Service> + Clone + Send + 'static, - N::Response: Send, - N::Future: Send, - N::Error: Send, - I: tower::Service, Response = N::Response, Error = N::Error> - + Clone - + Send - + 'static, - I::Future: Send, - B: Send + 'static, -{ - type Response = N::Response; - type Error = N::Error; - type Future = Pin> + Send>>; - - fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll> { - Poll::Ready(Ok(())) - } - - fn call(&mut self, req: Request) -> Self::Future { - let is_inference = req.uri().path().starts_with(INFERENCE_PATH_PREFIX); - - if is_inference { - let mut svc = self.inference.clone(); - Box::pin(async move { svc.ready().await?.call(req).await }) - } else { - let mut svc = self.openshell.clone(); - Box::pin(async move { svc.ready().await?.call(req).await }) - } - } -} - /// Assemble the authenticator chain for the gateway. /// /// Chain order (first-match-wins): @@ -1488,7 +1429,6 @@ mod tests { "/openshell.v1.OpenShell/GetSandboxProviderEnvironment", "/openshell.v1.OpenShell/SubmitPolicyAnalysis", "/openshell.v1.OpenShell/RefreshSandboxToken", - "/openshell.inference.v1.Inference/GetInferenceBundle", ]; for path in callback_paths { @@ -1521,8 +1461,6 @@ mod tests { "/openshell.v1.OpenShell/ListSandboxes", "/openshell.v1.OpenShell/DeleteSandbox", "/openshell.v1.OpenShell/CreateProvider", - "/openshell.inference.v1.Inference/GetInferenceRoute", - "/openshell.inference.v1.Inference/SetInferenceRoute", ]; for path in rejected_grpc_paths { @@ -1541,7 +1479,6 @@ mod tests { let paths = [ "/grpc.health.v1.Health/Check", "/openshell.v1.OpenShell/ListSandboxes", - "/openshell.inference.v1.Inference/GetInferenceRoute", "/health", "/service", ]; @@ -2492,7 +2429,6 @@ mod tests { "/openshell.v1.OpenShell/CreateSandbox", "/openshell.v1.OpenShell/ListSandboxes", "/openshell.v1.OpenShell/DeleteSandbox", - "/openshell.inference.v1.Inference/GetInferenceBundle", "/metrics", ]; @@ -2515,7 +2451,6 @@ mod tests { let expected = [ "GET", - "openshell.inference.v1.Inference/GetInferenceBundle", "openshell.v1.OpenShell/CreateSandbox", "openshell.v1.OpenShell/DeleteSandbox", "openshell.v1.OpenShell/ListSandboxes", @@ -2542,13 +2477,6 @@ mod tests { otel_span_name(&http::Method::POST, "/openshell.v1.OpenShell/CreateSandbox"), "openshell.v1.OpenShell/CreateSandbox" ); - assert_eq!( - otel_span_name( - &http::Method::POST, - "/openshell.inference.v1.Inference/GetInferenceBundle" - ), - "openshell.inference.v1.Inference/GetInferenceBundle" - ); } /// Non-RPC paths use a low-cardinality method-only name because sandbox @@ -2575,14 +2503,6 @@ mod tests { ); } - #[test] - fn grpc_method_extracts_inference_service() { - assert_eq!( - grpc_method_from_path("/openshell.inference.v1.Inference/GetInferenceBundle"), - "GetInferenceBundle" - ); - } - #[test] fn grpc_method_handles_bare_path() { assert_eq!(grpc_method_from_path("Health"), "Health"); @@ -2910,27 +2830,6 @@ mod tests { )); } - #[tokio::test] - async fn sandbox_principal_can_fetch_inference_bundle() { - let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal())))); - let chain = AuthenticatorChain::new(vec![mock]); - let (recorder, seen) = PrincipalRecorder::new(); - let mut router = AuthGrpcRouter::new(recorder, Some(chain), None); - - let res = router - .call(empty_request( - "/openshell.inference.v1.Inference/GetInferenceBundle", - )) - .await - .unwrap(); - - assert_eq!(res.status(), 200); - assert!(matches!( - seen.lock().unwrap().as_ref(), - Some(Principal::Sandbox(_)) - )); - } - /// A user principal — even one carrying `openshell:all` and the /// admin role — must not reach a `sandbox`-annotated method. The /// router enforces this from the per-handler auth-mode declarations @@ -2964,7 +2863,6 @@ mod tests { "/openshell.v1.OpenShell/RelayStream", "/openshell.v1.OpenShell/IssueSandboxToken", "/openshell.v1.OpenShell/RefreshSandboxToken", - "/openshell.inference.v1.Inference/GetInferenceBundle", ] { let mock = Arc::new(MockAuthenticator::returning(Ok(Some(admin_user())))); let chain = AuthenticatorChain::new(vec![mock]); @@ -3021,8 +2919,6 @@ mod tests { "/openshell.v1.OpenShell/DeleteSandbox", "/openshell.v1.OpenShell/CreateProvider", "/openshell.v1.OpenShell/ApproveDraftChunk", - "/openshell.inference.v1.Inference/GetInferenceRoute", - "/openshell.inference.v1.Inference/SetInferenceRoute", ] { let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal())))); let chain = AuthenticatorChain::new(vec![mock]); diff --git a/crates/openshell-server/src/persistence/postgres.rs b/crates/openshell-server/src/persistence/postgres.rs index 19c50c6187..449f8f6df3 100644 --- a/crates/openshell-server/src/persistence/postgres.rs +++ b/crates/openshell-server/src/persistence/postgres.rs @@ -18,6 +18,14 @@ use sqlx::{Connection, PgPool, Postgres, QueryBuilder, Row}; static POSTGRES_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/postgres"); +#[cfg(test)] +pub(super) fn embedded_migration_sql(version: i64) -> Option<&'static str> { + POSTGRES_MIGRATOR + .iter() + .find(|migration| migration.version == version) + .map(|migration| migration.sql.as_ref()) +} + use super::{DELETE_MANY_BATCH_SIZE, DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}; #[derive(Debug, Clone)] diff --git a/crates/openshell-server/src/persistence/sqlite.rs b/crates/openshell-server/src/persistence/sqlite.rs index 3e96040e34..924a38af14 100644 --- a/crates/openshell-server/src/persistence/sqlite.rs +++ b/crates/openshell-server/src/persistence/sqlite.rs @@ -23,6 +23,14 @@ use std::sync::atomic::{AtomicU64, Ordering}; use tokio::sync::Mutex; static SQLITE_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/sqlite"); + +#[cfg(test)] +pub(super) fn embedded_migration_sql(version: i64) -> Option<&'static str> { + SQLITE_MIGRATOR + .iter() + .find(|migration| migration.version == version) + .map(|migration| migration.sql.as_ref()) +} static IN_MEMORY_DB_SEQUENCE: AtomicU64 = AtomicU64::new(0); use super::{DELETE_MANY_BATCH_SIZE, DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}; diff --git a/crates/openshell-server/src/persistence/tests.rs b/crates/openshell-server/src/persistence/tests.rs index 37fc469d66..169fa81538 100644 --- a/crates/openshell-server/src/persistence/tests.rs +++ b/crates/openshell-server/src/persistence/tests.rs @@ -132,6 +132,56 @@ async fn sqlite_connect_runs_embedded_migrations() { assert!(records.is_empty()); } +#[tokio::test] +async fn sqlite_inference_route_removal_migration_deletes_only_managed_routes() { + use sqlx::{Connection, SqliteConnection}; + + let migration = super::sqlite::embedded_migration_sql(7) + .expect("SQLite migrator must embed removal migration 007"); + let mut connection = SqliteConnection::connect("sqlite::memory:") + .await + .expect("connect to migration test database"); + sqlx::raw_sql( + "CREATE TABLE objects (object_type TEXT NOT NULL, id TEXT NOT NULL);\ + INSERT INTO objects VALUES ('inference_route', 'managed-route');\ + INSERT INTO objects VALUES ('sandbox', 'preserved-sandbox');", + ) + .execute(&mut connection) + .await + .expect("seed pre-migration objects"); + + sqlx::raw_sql(migration) + .execute(&mut connection) + .await + .expect("run SQLite removal migration"); + + let remaining: Vec<(String, String)> = + sqlx::query_as("SELECT object_type, id FROM objects ORDER BY object_type, id") + .fetch_all(&mut connection) + .await + .expect("read migrated objects"); + assert_eq!( + remaining, + vec![("sandbox".to_string(), "preserved-sandbox".to_string())], + "removal migration must purge managed routes without touching other objects" + ); +} + +#[test] +fn embedded_migrators_include_inference_route_removal() { + for (backend, migration) in [ + ("sqlite", super::sqlite::embedded_migration_sql(7)), + ("postgres", super::postgres::embedded_migration_sql(7)), + ] { + let sql = + migration.unwrap_or_else(|| panic!("{backend} migrator is missing migration 007")); + assert!( + sql.contains("DELETE FROM objects WHERE object_type = 'inference_route'"), + "{backend} migration 007 must purge managed inference route objects" + ); + } +} + #[tokio::test] async fn sqlite_in_memory_store_survives_pool_connection_replacement() { for url in ["sqlite::memory:", "sqlite://?mode=memory"] { diff --git a/crates/openshell-supervisor-network/Cargo.toml b/crates/openshell-supervisor-network/Cargo.toml index 34d9c32a47..edc5658c51 100644 --- a/crates/openshell-supervisor-network/Cargo.toml +++ b/crates/openshell-supervisor-network/Cargo.toml @@ -3,7 +3,7 @@ [package] name = "openshell-supervisor-network" -description = "Network component of the OpenShell supervisor: proxy, L7 enforcement, OPA, inference routing, denial aggregator" +description = "Network component of the OpenShell supervisor: proxy, L7 enforcement, OPA, and denial aggregation" version.workspace = true edition.workspace = true license.workspace = true @@ -14,7 +14,6 @@ rust-version.workspace = true openshell-core = { path = "../openshell-core", features = ["oauth"] } openshell-ocsf = { path = "../openshell-ocsf" } openshell-policy = { path = "../openshell-policy" } -openshell-router = { path = "../openshell-router" } openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } apollo-parser = { workspace = true } diff --git a/crates/openshell-supervisor-network/src/inference_routes.rs b/crates/openshell-supervisor-network/src/inference_routes.rs deleted file mode 100644 index 22b406b8dd..0000000000 --- a/crates/openshell-supervisor-network/src/inference_routes.rs +++ /dev/null @@ -1,751 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Inference route bundle resolution and refresh. -//! -//! Resolves inference routes from one of two sources at sandbox startup: -//! a local YAML file (`--inference-routes`) or a cluster bundle fetched via -//! gRPC. Builds the [`InferenceContext`] consumed by the proxy's L7 layer -//! and spawns a background refresh loop in cluster mode so route changes -//! propagate without restarting the sandbox. -//! -//! Distinct from [`crate::l7::inference`], which parses HTTP requests and -//! matches them against API patterns at request time. -//! -//! [`InferenceContext`]: crate::proxy::InferenceContext - -use std::sync::Arc; -use std::time::Duration; - -use miette::Result; -use tracing::{info, trace, warn}; - -use openshell_ocsf::{ - ConfigStateChangeBuilder, SeverityId, StateId, StatusId, ctx::ctx as ocsf_ctx, ocsf_emit, -}; - -/// Default interval (seconds) for re-fetching the inference route bundle from -/// the gateway in cluster mode. -/// -/// Override at runtime with the `OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS` -/// environment variable. File-based routes (`--inference-routes`) are loaded -/// once at startup and never refreshed. -pub const DEFAULT_ROUTE_REFRESH_INTERVAL_SECS: u64 = 5; - -/// Route name for the sandbox system inference route. -const SANDBOX_SYSTEM_ROUTE_NAME: &str = "sandbox-system"; - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum InferenceRouteSource { - File, - Cluster, - None, -} - -pub fn infer_route_source( - sandbox_id: Option<&str>, - openshell_endpoint: Option<&str>, - inference_routes: Option<&str>, -) -> InferenceRouteSource { - if inference_routes.is_some() { - InferenceRouteSource::File - } else if sandbox_id.is_some() && openshell_endpoint.is_some() { - InferenceRouteSource::Cluster - } else { - InferenceRouteSource::None - } -} - -pub fn disable_inference_on_empty_routes(source: InferenceRouteSource) -> bool { - !matches!(source, InferenceRouteSource::Cluster) -} - -pub fn route_refresh_interval_secs() -> u64 { - let Ok(value) = std::env::var("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS") else { - return DEFAULT_ROUTE_REFRESH_INTERVAL_SECS; - }; - match value.parse::() { - Ok(interval) if interval > 0 => interval, - Ok(_) => { - warn!( - default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, - "Ignoring zero route refresh interval" - ); - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - } - Err(error) => { - warn!( - interval = %value, - error = %error, - default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, - "Ignoring invalid route refresh interval" - ); - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - } - } -} - -/// Build an [`InferenceContext`](crate::proxy::InferenceContext) by resolving -/// inference routes from either a local YAML file or the gateway bundle. -/// -/// If both a routes file and cluster credentials are provided, the routes file -/// wins and the cluster bundle is not fetched. -/// -/// Returns `None` if neither source is configured (inference routing disabled). -/// -/// # Errors -/// -/// Returns an error if loading the routes file fails or the file's routes -/// cannot be resolved. gRPC errors are swallowed (logged) and produce -/// `Ok(None)` so a missing cluster bundle disables inference routing rather -/// than aborting sandbox startup. -// `routes`/`router` are intentionally distinct nouns (the route list vs the -// router that consumes them); both names are clearer than alternatives. -#[allow(clippy::similar_names)] -pub async fn build_inference_context( - sandbox_id: Option<&str>, - openshell_endpoint: Option<&str>, - inference_routes: Option<&str>, -) -> Result>> { - use openshell_router::Router; - use openshell_router::config::RouterConfig; - - let source = infer_route_source(sandbox_id, openshell_endpoint, inference_routes); - - // Captured during the initial cluster bundle fetch so the background refresh - // loop can skip no-op updates from the very first tick. - let mut initial_revision: Option = None; - - let routes = match source { - InferenceRouteSource::File => { - let Some(path) = inference_routes else { - return Ok(None); - }; - - // Standalone mode: load routes from file (fail-fast on errors) - if sandbox_id.is_some() { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("inference_routes", serde_json::json!(path)) - .message(format!( - "Inference routes file takes precedence over cluster bundle [path:{path}]" - )) - .build()); - } - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "loading") - .unmapped("inference_routes", serde_json::json!(path)) - .message(format!("Loading inference routes from file [path:{path}]")) - .build() - ); - let config = RouterConfig::load_from_file(std::path::Path::new(path)) - .map_err(|e| miette::miette!("failed to load inference routes {path}: {e}"))?; - config - .resolve_routes() - .map_err(|e| miette::miette!("failed to resolve routes from {path}: {e}"))? - } - InferenceRouteSource::Cluster => { - let (Some(_id), Some(endpoint)) = (sandbox_id, openshell_endpoint) else { - return Ok(None); - }; - - // Cluster mode: fetch bundle from gateway - info!(endpoint = %endpoint, "Fetching inference route bundle from gateway"); - match openshell_core::grpc_client::fetch_inference_bundle(endpoint).await { - Ok(bundle) => { - initial_revision = Some(bundle.revision.clone()); - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("route_count", serde_json::json!(bundle.routes.len())) - .unmapped("revision", serde_json::json!(&bundle.revision)) - .message(format!( - "Loaded inference route bundle [route_count:{} revision:{}]", - bundle.routes.len(), - bundle.revision - )) - .build() - ); - bundle_to_resolved_routes(&bundle) - } - Err(e) => { - // Distinguish expected "not configured" states from server errors. - // gRPC PermissionDenied/NotFound means inference bundle is unavailable - // for this sandbox — skip gracefully. Other errors are unexpected. - let msg = e.to_string(); - if msg.contains("permission denied") || msg.contains("not found") { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Disabled, "disabled") - .unmapped("error", serde_json::json!(e.to_string())) - .message(format!( - "Inference bundle unavailable, routing disabled [error:{e}]" - )) - .build() - ); - return Ok(None); - } - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Disabled, "disabled") - .unmapped("error", serde_json::json!(e.to_string())) - .message(format!( - "Failed to fetch inference bundle, inference routing disabled [error:{e}]" - )) - .build()); - return Ok(None); - } - } - } - InferenceRouteSource::None => { - // No route source — inference routing is not configured - return Ok(None); - } - }; - - if routes.is_empty() && disable_inference_on_empty_routes(source) { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Disabled, "disabled") - .message("No usable inference routes, inference routing disabled") - .build() - ); - return Ok(None); - } - - if routes.is_empty() { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "waiting") - .message("Inference route bundle is empty; keeping routing enabled and waiting for refresh") - .build()); - } - - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "enabled") - .unmapped("route_count", serde_json::json!(routes.len())) - .message(format!( - "Inference routing enabled with local execution [route_count:{}]", - routes.len() - )) - .build() - ); - - // Partition routes by name into user-facing and system caches. - let (user_routes, system_routes) = partition_routes(routes); - - let router = - Router::new().map_err(|e| miette::miette!("failed to initialize inference router: {e}"))?; - let patterns = crate::l7::inference::default_patterns(); - - let ctx = Arc::new(crate::proxy::InferenceContext::new( - patterns, - router, - user_routes, - system_routes, - )); - - // Spawn background route cache refresh for cluster mode at startup so - // request handling never depends on control-plane latency. - if matches!(source, InferenceRouteSource::Cluster) - && let (Some(_id), Some(endpoint)) = (sandbox_id, openshell_endpoint) - { - spawn_route_refresh( - ctx.route_cache(), - ctx.system_route_cache(), - endpoint.to_string(), - route_refresh_interval_secs(), - initial_revision, - ); - } - - Ok(Some(ctx)) -} - -/// Split resolved routes into user-facing and system caches by route name. -/// -/// Routes named `"sandbox-system"` go to the system cache; everything else -/// (including `"inference.local"` and empty names) goes to the user cache. -pub fn partition_routes( - routes: Vec, -) -> ( - Vec, - Vec, -) { - let mut user = Vec::new(); - let mut system = Vec::new(); - for r in routes { - if r.name == SANDBOX_SYSTEM_ROUTE_NAME { - system.push(r); - } else { - user.push(r); - } - } - (user, system) -} - -/// Convert a proto bundle response into resolved routes for the router. -pub fn bundle_to_resolved_routes( - bundle: &openshell_core::proto::GetInferenceBundleResponse, -) -> Vec { - bundle - .routes - .iter() - .map(|r| { - let (auth, default_headers, passthrough_headers) = - openshell_core::inference::route_headers_for_provider_type(&r.provider_type); - let timeout = if r.timeout_secs == 0 { - openshell_router::config::DEFAULT_ROUTE_TIMEOUT - } else { - Duration::from_secs(r.timeout_secs) - }; - openshell_router::config::ResolvedRoute { - name: r.name.clone(), - endpoint: r.base_url.clone(), - model: r.model_id.clone(), - api_key: r.api_key.clone(), - protocols: r.protocols.clone(), - auth, - default_headers, - passthrough_headers, - timeout, - model_in_path: r.model_in_path, - request_path_override: r.request_path_override.clone(), - } - }) - .collect() -} - -/// Spawn a background task that periodically refreshes both route caches from the gateway. -/// -/// The loop uses the bundle `revision` hash to avoid unnecessary cache writes -/// when routes haven't changed. `initial_revision` is the revision captured -/// during the startup fetch in [`build_inference_context`] so the first refresh -/// cycle can already skip a no-op update. -pub fn spawn_route_refresh( - user_cache: Arc>>, - system_cache: Arc>>, - endpoint: String, - interval_secs: u64, - initial_revision: Option, -) { - tokio::spawn(async move { - use tokio::time::{MissedTickBehavior, interval}; - - let mut current_revision = initial_revision; - - let mut tick = interval(Duration::from_secs(interval_secs)); - tick.set_missed_tick_behavior(MissedTickBehavior::Skip); - - loop { - tick.tick().await; - - match openshell_core::grpc_client::fetch_inference_bundle(&endpoint).await { - Ok(bundle) => { - if current_revision.as_deref() == Some(&bundle.revision) { - trace!(revision = %bundle.revision, "Inference bundle unchanged"); - continue; - } - - let routes = bundle_to_resolved_routes(&bundle); - let (user_routes, system_routes) = partition_routes(routes); - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "updated") - .unmapped("user_route_count", serde_json::json!(user_routes.len())) - .unmapped("system_route_count", serde_json::json!(system_routes.len())) - .unmapped("revision", serde_json::json!(&bundle.revision)) - .message(format!( - "Inference routes updated [user_route_count:{} system_route_count:{} revision:{}]", - user_routes.len(), - system_routes.len(), - bundle.revision - )) - .build()); - current_revision = Some(bundle.revision); - *user_cache.write().await = user_routes; - *system_cache.write().await = system_routes; - } - Err(e) => { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "stale") - .unmapped("error", serde_json::json!(e.to_string())) - .message(format!( - "Failed to refresh inference route cache, keeping stale routes [error:{e}]" - )) - .build()); - } - } - } - }); -} - -#[cfg(test)] -#[allow( - clippy::needless_raw_string_hashes, - clippy::similar_names, - reason = "Test code: test fixtures often use idiomatic forms not flagged in production." -)] -mod tests { - use super::*; - use std::sync::{LazyLock, Mutex}; - use temp_env::with_vars; - - static ENV_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); - - #[test] - fn bundle_to_resolved_routes_converts_all_fields() { - let bundle = openshell_core::proto::GetInferenceBundleResponse { - routes: vec![ - openshell_core::proto::ResolvedRoute { - name: "frontier".to_string(), - base_url: "https://api.example.com/v1".to_string(), - api_key: "sk-test-key".to_string(), - model_id: "gpt-4".to_string(), - protocols: vec![ - "openai_chat_completions".to_string(), - "openai_responses".to_string(), - ], - provider_type: "openai".to_string(), - timeout_secs: 0, - model_in_path: false, - request_path_override: None, - }, - openshell_core::proto::ResolvedRoute { - name: "local".to_string(), - base_url: "http://vllm:8000/v1".to_string(), - api_key: "local-key".to_string(), - model_id: "llama-3".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - provider_type: String::new(), - timeout_secs: 120, - model_in_path: false, - request_path_override: None, - }, - ], - revision: "abc123".to_string(), - generated_at_ms: 1000, - }; - - let routes = bundle_to_resolved_routes(&bundle); - - assert_eq!(routes.len(), 2); - assert_eq!(routes[0].endpoint, "https://api.example.com/v1"); - assert_eq!(routes[0].model, "gpt-4"); - assert_eq!(routes[0].api_key, "sk-test-key"); - assert_eq!( - routes[0].auth, - openshell_core::inference::AuthHeader::Bearer - ); - assert_eq!( - routes[0].protocols, - vec!["openai_chat_completions", "openai_responses"] - ); - assert_eq!( - routes[0].timeout, - openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - "timeout_secs=0 should map to default" - ); - assert_eq!(routes[1].endpoint, "http://vllm:8000/v1"); - assert_eq!( - routes[1].auth, - openshell_core::inference::AuthHeader::Bearer - ); - assert_eq!( - routes[1].timeout, - Duration::from_secs(120), - "timeout_secs=120 should map to 120s" - ); - } - - #[test] - fn bundle_to_resolved_routes_handles_empty_bundle() { - let bundle = openshell_core::proto::GetInferenceBundleResponse { - routes: vec![], - revision: "empty".to_string(), - generated_at_ms: 0, - }; - - let routes = bundle_to_resolved_routes(&bundle); - assert!(routes.is_empty()); - } - - #[test] - fn bundle_to_resolved_routes_preserves_name_field() { - let bundle = openshell_core::proto::GetInferenceBundleResponse { - routes: vec![openshell_core::proto::ResolvedRoute { - name: "sandbox-system".to_string(), - base_url: "https://api.example.com/v1".to_string(), - api_key: "key".to_string(), - model_id: "model".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - provider_type: "openai".to_string(), - timeout_secs: 0, - model_in_path: false, - request_path_override: None, - }], - revision: "rev".to_string(), - generated_at_ms: 0, - }; - - let routes = bundle_to_resolved_routes(&bundle); - assert_eq!(routes[0].name, "sandbox-system"); - } - - #[test] - fn routes_segregated_by_name() { - let routes = vec![ - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.openai.com/v1".to_string(), - model: "gpt-4o".to_string(), - api_key: "key1".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: openshell_core::inference::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }, - openshell_router::config::ResolvedRoute { - name: "sandbox-system".to_string(), - endpoint: "https://api.anthropic.com/v1".to_string(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "key2".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: openshell_core::inference::AuthHeader::Custom("x-api-key"), - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }, - ]; - - let (user, system) = partition_routes(routes); - assert_eq!(user.len(), 1); - assert_eq!(user[0].name, "inference.local"); - assert_eq!(system.len(), 1); - assert_eq!(system[0].name, "sandbox-system"); - } - - // -- build_inference_context tests -- - - #[tokio::test] - async fn build_inference_context_route_file_loads_routes() { - use std::io::Write; - - let yaml = r#" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] - api_key: test-key -"#; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - let path = f.path().to_str().unwrap(); - - let ctx = build_inference_context(None, None, Some(path)) - .await - .expect("should load routes from file"); - - let ctx = ctx.expect("context should be Some"); - let cache = ctx.route_cache(); - let routes = cache.read().await; - assert_eq!(routes.len(), 1); - assert_eq!(routes[0].endpoint, "http://localhost:8000/v1"); - } - - #[tokio::test] - async fn build_inference_context_empty_route_file_returns_none() { - use std::io::Write; - - // Route file with empty routes list → inference routing disabled (not an error) - let yaml = "routes: []\n"; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - let path = f.path().to_str().unwrap(); - - let ctx = build_inference_context(None, None, Some(path)) - .await - .expect("empty routes file should not error"); - assert!( - ctx.is_none(), - "empty routes should disable inference routing" - ); - } - - #[tokio::test] - async fn build_inference_context_no_sources_returns_none() { - let ctx = build_inference_context(None, None, None) - .await - .expect("should succeed with None"); - - assert!(ctx.is_none(), "no sources should return None"); - } - - #[tokio::test] - async fn build_inference_context_route_file_overrides_cluster() { - use std::io::Write; - - let yaml = r#" -routes: - - name: inference.local - endpoint: http://localhost:9999/v1 - model: file-model - protocols: [openai_chat_completions] - api_key: file-key -"#; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - let path = f.path().to_str().unwrap(); - - // Even with sandbox_id and endpoint, route_file takes precedence - let ctx = build_inference_context(Some("sb-1"), Some("http://localhost:50051"), Some(path)) - .await - .expect("should load from file"); - - let ctx = ctx.expect("context should be Some"); - let cache = ctx.route_cache(); - let routes = cache.read().await; - assert_eq!(routes[0].endpoint, "http://localhost:9999/v1"); - } - - #[test] - fn infer_route_source_prefers_file_mode() { - assert_eq!( - infer_route_source( - Some("sb-1"), - Some("http://localhost:50051"), - Some("routes.yaml") - ), - InferenceRouteSource::File - ); - } - - #[test] - fn infer_route_source_cluster_requires_id_and_endpoint() { - assert_eq!( - infer_route_source(Some("sb-1"), Some("http://localhost:50051"), None), - InferenceRouteSource::Cluster - ); - assert_eq!( - infer_route_source(Some("sb-1"), None, None), - InferenceRouteSource::None - ); - assert_eq!( - infer_route_source(None, Some("http://localhost:50051"), None), - InferenceRouteSource::None - ); - } - - #[test] - fn disable_inference_on_empty_routes_depends_on_source() { - assert!(disable_inference_on_empty_routes( - InferenceRouteSource::File - )); - assert!(!disable_inference_on_empty_routes( - InferenceRouteSource::Cluster - )); - assert!(disable_inference_on_empty_routes( - InferenceRouteSource::None - )); - } - - // ---- Route refresh interval + revision tests ---- - - #[test] - fn default_route_refresh_interval_is_five_seconds() { - assert_eq!(DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, 5); - } - - #[test] - fn route_refresh_interval_uses_env_override() { - let _guard = ENV_LOCK.lock().unwrap(); - with_vars( - [("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS", Some("9"))], - || { - assert_eq!(route_refresh_interval_secs(), 9); - }, - ); - } - - #[test] - fn route_refresh_interval_rejects_zero() { - let _guard = ENV_LOCK.lock().unwrap(); - with_vars( - [("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS", Some("0"))], - || { - assert_eq!( - route_refresh_interval_secs(), - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - ); - }, - ); - } - - #[test] - fn route_refresh_interval_rejects_invalid_values() { - let _guard = ENV_LOCK.lock().unwrap(); - with_vars( - [("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS", Some("abc"))], - || { - assert_eq!( - route_refresh_interval_secs(), - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - ); - }, - ); - } - - #[tokio::test] - async fn route_cache_preserves_content_when_not_written() { - use std::sync::Arc; - use tokio::sync::RwLock; - - let routes = vec![openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "http://original:8000/v1".to_string(), - model: "original-model".to_string(), - api_key: "key".to_string(), - auth: openshell_core::inference::AuthHeader::Bearer, - protocols: vec!["openai_chat_completions".to_string()], - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let cache = Arc::new(RwLock::new(routes)); - - // Verify the cache preserves its content — the revision-based skip - // logic in spawn_route_refresh ensures the cache is only written - // when the revision actually changes. - let read = cache.read().await; - assert_eq!(read.len(), 1); - assert_eq!(read[0].model, "original-model"); - } -} diff --git a/crates/openshell-supervisor-network/src/l7/inference.rs b/crates/openshell-supervisor-network/src/l7/inference.rs deleted file mode 100644 index 5aefab9e93..0000000000 --- a/crates/openshell-supervisor-network/src/l7/inference.rs +++ /dev/null @@ -1,1003 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Inference API pattern detection and gateway rerouting. -//! -//! For requests targeting `inference.local`, this module detects whether the -//! HTTP request is a known inference API call and routes it through the local -//! sandbox router. - -/// How an inference protocol delivers its response to the sandboxed client. -/// -/// `Streaming` protocols (chat completions, completions, responses, Anthropic -/// messages) emit a Server-Sent Events token stream and are served through the -/// chunked transfer-encoding path so tokens reach the client incrementally. -/// -/// `Buffered` protocols (embeddings, model discovery) return a single JSON -/// object the client parses whole. They must be served in one piece with an -/// accurate `Content-Length`. Sending them through the streaming path is -/// unsafe: a mid-body truncation (the streaming size cap or idle timeout) -/// appends an SSE error event to bytes the client decodes as one JSON object, -/// silently corrupting it. -/// -/// Framing is a property of the protocol, declared once per pattern in -/// [`default_patterns`], so the streaming-vs-buffered decision cannot drift -/// across the dispatch sites that consume it. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ResponseFraming { - /// SSE token stream, served via chunked transfer-encoding. - /// - /// The `OpenAI` completion-style protocols are classified streaming - /// unconditionally. They are dual-mode (a `stream: false` request returns - /// one buffered JSON object), but the dispatch keys framing off the - /// protocol alone and does not inspect the request body, so they are - /// streamed defensively. Their buffered responses tolerate chunked framing; - /// only the embeddings and model-discovery shapes (no streaming mode at - /// all) must be served buffered. - Streaming, - /// Single JSON object, served buffered with an accurate `Content-Length`. - Buffered, -} - -/// An inference API pattern for detecting inference calls in intercepted traffic. -#[derive(Debug, Clone)] -pub struct InferenceApiPattern { - pub method: String, - pub path_glob: String, - pub protocol: String, - pub kind: String, - /// Response delivery mode for this protocol. Selects the buffered or - /// streaming proxy path; see [`ResponseFraming`]. - pub framing: ResponseFraming, -} - -impl InferenceApiPattern { - /// Whether this protocol's response must be served buffered (one JSON - /// object framed with an accurate `Content-Length`) rather than streamed. - #[must_use] - pub fn is_buffered(&self) -> bool { - matches!(self.framing, ResponseFraming::Buffered) - } -} - -/// Default patterns for known inference APIs (`OpenAI`, Anthropic, AWS Bedrock). -pub fn default_patterns() -> Vec { - vec![ - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/chat/completions".to_string(), - protocol: "openai_chat_completions".to_string(), - kind: "chat_completion".to_string(), - framing: ResponseFraming::Streaming, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/completions".to_string(), - protocol: "openai_completions".to_string(), - kind: "completion".to_string(), - framing: ResponseFraming::Streaming, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/responses".to_string(), - protocol: "openai_responses".to_string(), - kind: "responses".to_string(), - framing: ResponseFraming::Streaming, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/embeddings".to_string(), - protocol: "openai_embeddings".to_string(), - kind: "embeddings".to_string(), - framing: ResponseFraming::Buffered, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/messages".to_string(), - protocol: "anthropic_messages".to_string(), - kind: "messages".to_string(), - framing: ResponseFraming::Streaming, - }, - // Model discovery returns one JSON object (a model list), never an SSE - // stream, so it is served buffered for the same reason as embeddings. - InferenceApiPattern { - method: "GET".to_string(), - path_glob: "/v1/models".to_string(), - protocol: "model_discovery".to_string(), - kind: "models_list".to_string(), - framing: ResponseFraming::Buffered, - }, - InferenceApiPattern { - method: "GET".to_string(), - path_glob: "/v1/models/*".to_string(), - protocol: "model_discovery".to_string(), - kind: "models_get".to_string(), - framing: ResponseFraming::Buffered, - }, - // AWS Bedrock InvokeModel. The `*` segment is the Bedrock model id - // (e.g. `anthropic.claude-opus-4-7`). - // - // InvokeModel returns ONE JSON object the client decodes whole — it - // must be served buffered with an accurate `Content-Length`, otherwise - // the streaming proxy's size-cap or idle-timeout failure mode would - // append an SSE error event to bytes the caller decodes as one JSON - // object, silently corrupting it. - // - // `InvokeModelWithResponseStream` - // (`/model/{id}/invoke-with-response-stream`) is deferred to a - // follow-up: Bedrock streams use AWS event-stream framing, but the - // shared streaming relay's truncation/timeout/upstream-error path - // emits SSE-formatted error frames, which would corrupt downstream - // event-stream parsers. The follow-up adds protocol-aware error - // termination before re-introducing the streaming pattern. - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/model/*/invoke".to_string(), - protocol: "aws_bedrock_invoke".to_string(), - kind: "messages".to_string(), - framing: ResponseFraming::Buffered, - }, - ] -} - -/// Check if an HTTP request matches a known inference API pattern. -/// -/// Path globs support two wildcard shapes (one per pattern, not both): -/// - **Trailing `/*`**: `/v1/models/*` matches `/v1/models` and any -/// `/v1/models/` (one or many path segments). -/// - **Middle `/*/`**: `/model/*/invoke` matches `/model//invoke` -/// for a single non-empty segment that contains no `/`. Used for -/// AWS Bedrock's `/model/{modelId}/invoke[-with-response-stream]`. -pub fn detect_inference_pattern<'a>( - method: &str, - path: &str, - patterns: &'a [InferenceApiPattern], -) -> Option<&'a InferenceApiPattern> { - // Strip query string for matching - let path_only = path.split('?').next().unwrap_or(path); - patterns.iter().find(|p| { - if !method.eq_ignore_ascii_case(&p.method) { - return false; - } - - if let Some(prefix) = p.path_glob.strip_suffix("/*") { - return path_only == prefix - || path_only - .strip_prefix(prefix) - .is_some_and(|suffix| suffix.starts_with('/')); - } - - if let Some((before, after)) = p.path_glob.split_once("/*/") { - let Some(rest) = path_only.strip_prefix(before) else { - return false; - }; - let Some(rest) = rest.strip_prefix('/') else { - return false; - }; - // rest must look like `/` where is non-empty - // and contains no `/` (single path segment). - let Some(slash_at) = rest.find('/') else { - return false; - }; - let Some(after_segment) = rest.get(slash_at + 1..) else { - return false; - }; - return slash_at > 0 && after_segment == after; - } - - path_only == p.path_glob - }) -} - -/// A parsed HTTP request from the intercepted tunnel. -pub struct ParsedHttpRequest { - pub method: String, - pub path: String, - pub headers: Vec<(String, String)>, - pub body: Vec, -} - -/// Result of attempting to parse an HTTP request from a buffer. -pub enum ParseResult { - /// A complete request was parsed, along with the byte count consumed. - Complete(ParsedHttpRequest, usize), - /// Headers are incomplete — caller should read more data. - Incomplete, - /// The request is malformed and must be rejected (e.g., duplicate Content-Length). - Invalid(String), -} - -/// Try to parse an HTTP/1.1 request from raw bytes. -/// -/// Returns [`ParseResult::Complete`] with the parsed request and bytes consumed, -/// or [`ParseResult::Incomplete`] if more data is needed. -pub fn try_parse_http_request(buf: &[u8]) -> ParseResult { - let Some(header_end) = buf.windows(4).position(|w| w == b"\r\n\r\n") else { - return ParseResult::Incomplete; - }; - let headers_bytes = &buf[..header_end]; - let Ok(header_str) = std::str::from_utf8(headers_bytes) else { - return ParseResult::Incomplete; - }; - let body_start = header_end + 4; - - let mut lines = header_str.split("\r\n"); - let Some(request_line) = lines.next() else { - return ParseResult::Incomplete; - }; - let mut parts = request_line.split_whitespace(); - let (Some(method), Some(path)) = (parts.next(), parts.next()) else { - return ParseResult::Incomplete; - }; - let method = method.to_string(); - let path = path.to_string(); - - let mut headers = Vec::new(); - let mut content_length: usize = 0; - let mut has_content_length = false; - let mut is_chunked = false; - for line in lines { - if line.is_empty() { - break; - } - if let Some((name, value)) = line.split_once(':') { - let name = name.trim().to_string(); - let value = value.trim().to_string(); - if name.eq_ignore_ascii_case("content-length") { - let new_len: usize = match value.parse() { - Ok(v) => v, - Err(_) => { - return ParseResult::Invalid(format!( - "invalid Content-Length value: {value}" - )); - } - }; - if has_content_length && new_len != content_length { - return ParseResult::Invalid(format!( - "duplicate Content-Length headers with differing values ({content_length} vs {new_len})" - )); - } - content_length = new_len; - has_content_length = true; - } - if name.eq_ignore_ascii_case("transfer-encoding") - && value - .split(',') - .any(|enc| enc.trim().eq_ignore_ascii_case("chunked")) - { - is_chunked = true; - } - headers.push((name, value)); - } - } - - if is_chunked && has_content_length { - return ParseResult::Invalid( - "Request contains both Transfer-Encoding and Content-Length headers".to_string(), - ); - } - - let (body, consumed) = if is_chunked { - let Some((decoded_body, consumed)) = parse_chunked_body(buf, body_start) else { - return ParseResult::Incomplete; - }; - (decoded_body, consumed) - } else { - let total_len = body_start + content_length; - if buf.len() < total_len { - return ParseResult::Incomplete; - } - (buf[body_start..total_len].to_vec(), total_len) - }; - - ParseResult::Complete( - ParsedHttpRequest { - method, - path, - headers, - body, - }, - consumed, - ) -} - -/// Maximum decoded body size from chunked transfer encoding (10 MiB). -/// Matches the caller's `MAX_INFERENCE_BUF` limit. -const MAX_CHUNKED_BODY: usize = 10 * 1024 * 1024; - -/// Maximum number of chunks to process. Normal HTTP clients send the body -/// in a handful of large chunks; thousands of tiny chunks indicate abuse. -const MAX_CHUNK_COUNT: usize = 4096; - -/// Parse an HTTP chunked body from `buf[start..]`. -/// -/// Returns `(decoded_body, total_consumed_bytes_from_buf_start)` when complete, -/// or `None` if more bytes are needed or resource limits are exceeded. -fn parse_chunked_body(buf: &[u8], start: usize) -> Option<(Vec, usize)> { - let mut pos = start; - let mut body = Vec::new(); - let mut chunk_count: usize = 0; - - loop { - chunk_count += 1; - if chunk_count > MAX_CHUNK_COUNT { - return None; - } - - let size_line_end = find_crlf(buf, pos)?; - let size_line = std::str::from_utf8(&buf[pos..size_line_end]).ok()?; - let size_token = size_line.split(';').next()?.trim(); - let chunk_size = usize::from_str_radix(size_token, 16).ok()?; - pos = size_line_end.checked_add(2)?; - - if chunk_size == 0 { - // Parse trailers (if any). Terminates on empty trailer line. - loop { - let trailer_end = find_crlf(buf, pos)?; - let trailer_line = &buf[pos..trailer_end]; - pos = trailer_end.checked_add(2)?; - if trailer_line.is_empty() { - return Some((body, pos)); - } - } - } - - // Early reject: chunk cannot possibly fit in remaining buffer. - let remaining = buf.len().saturating_sub(pos); - if chunk_size > remaining { - return None; - } - - // Reject if decoded body would exceed size limit. - if body.len().saturating_add(chunk_size) > MAX_CHUNKED_BODY { - return None; - } - - let chunk_end = pos.checked_add(chunk_size)?; - let chunk_crlf_end = chunk_end.checked_add(2)?; - if buf.len() < chunk_crlf_end { - return None; - } - if &buf[chunk_end..chunk_crlf_end] != b"\r\n" { - return None; - } - - body.extend_from_slice(&buf[pos..chunk_end]); - pos = chunk_crlf_end; - } -} - -fn find_crlf(buf: &[u8], start: usize) -> Option { - buf.get(start..)? - .windows(2) - .position(|w| w == b"\r\n") - .map(|offset| start + offset) -} - -/// Reason phrase for an HTTP status code used on the inference proxy path. -/// -/// Covers the statuses produced by the router error mapping (400/401/403/500/ -/// 502/503) and the upstream codes an inference backend can pass through -/// verbatim (404/405 on unknown model or method, 422 on malformed embeddings -/// input, 429 on rate limit). Unknown codes fall back to `"Unknown"` so the -/// status line is still well-formed. -fn http_status_text(status: u16) -> &'static str { - match status { - 200 => "OK", - 400 => "Bad Request", - 401 => "Unauthorized", - 403 => "Forbidden", - 404 => "Not Found", - 405 => "Method Not Allowed", - 411 => "Length Required", - 413 => "Payload Too Large", - 422 => "Unprocessable Entity", - 429 => "Too Many Requests", - 500 => "Internal Server Error", - 502 => "Bad Gateway", - 503 => "Service Unavailable", - _ => "Unknown", - } -} - -/// Format an HTTP/1.1 response from status, headers, and body. -pub fn format_http_response(status: u16, headers: &[(String, String)], body: &[u8]) -> Vec { - use std::fmt::Write; - - let status_text = http_status_text(status); - - let mut response = format!("HTTP/1.1 {status} {status_text}\r\n"); - let mut has_content_length = false; - for (name, value) in headers { - let _ = write!(response, "{name}: {value}\r\n"); - if name.eq_ignore_ascii_case("content-length") { - has_content_length = true; - } - } - if !has_content_length { - let _ = write!(response, "content-length: {}\r\n", body.len()); - } - response.push_str("\r\n"); - - let mut bytes = response.into_bytes(); - bytes.extend_from_slice(body); - bytes -} - -/// Format HTTP/1.1 response headers for a chunked (streaming) response. -/// -/// Emits the status line, supplied headers (stripping any `content-length` or -/// `transfer-encoding` the upstream may have sent), and a -/// `transfer-encoding: chunked` header. The body is **not** included — the -/// caller writes chunks separately via [`format_chunk`] and -/// [`format_chunk_terminator`]. -pub fn format_http_response_header(status: u16, headers: &[(String, String)]) -> Vec { - use std::fmt::Write; - - let status_text = http_status_text(status); - - let mut response = format!("HTTP/1.1 {status} {status_text}\r\n"); - for (name, value) in headers { - // Skip framing headers — we always emit chunked TE. - if name.eq_ignore_ascii_case("content-length") - || name.eq_ignore_ascii_case("transfer-encoding") - { - continue; - } - let _ = write!(response, "{name}: {value}\r\n"); - } - let _ = write!(response, "transfer-encoding: chunked\r\n"); - response.push_str("\r\n"); - response.into_bytes() -} - -/// Format a single HTTP chunked transfer-encoding segment. -/// -/// Returns `\r\n\r\n`. -pub fn format_chunk(data: &[u8]) -> Vec { - let mut buf = format!("{:x}\r\n", data.len()).into_bytes(); - buf.extend_from_slice(data); - buf.extend_from_slice(b"\r\n"); - buf -} - -/// The HTTP chunked transfer-encoding terminator: `0\r\n\r\n`. -pub fn format_chunk_terminator() -> &'static [u8] { - b"0\r\n\r\n" -} - -/// Format an SSE error event for injection into a streaming response. -/// -/// Sent just before the chunked terminator when the proxy truncates a stream -/// due to timeout, byte limit, or upstream error. Clients parsing SSE events -/// can detect this and surface the error instead of silently losing data. -/// -/// The `reason` must NOT contain internal URLs, hostnames, or credentials — -/// the OCSF log captures full detail server-side. -pub fn format_sse_error(reason: &str) -> Vec { - // Use serde_json to escape control characters, quotes, and backslashes - // correctly. A handwritten escape can't safely cover \u0000-\u001F, and - // an unescaped \n\n in `reason` would split the SSE event into two - // frames, allowing a malicious upstream to inject a forged event. - let payload = serde_json::json!({ - "error": { - "message": reason, - "type": "proxy_stream_error", - } - }); - let mut out = Vec::with_capacity(reason.len() + 64); - out.extend_from_slice(b"data: "); - // serde_json::to_writer is infallible for in-memory Vec. - serde_json::to_writer(&mut out, &payload).expect("serializing static schema cannot fail"); - out.extend_from_slice(b"\n\n"); - out -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn detect_openai_chat_completions() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/chat/completions", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "openai_chat_completions"); - } - - #[test] - fn detect_openai_responses() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/responses", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "openai_responses"); - } - - #[test] - fn detect_anthropic_messages() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/messages", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "anthropic_messages"); - } - - #[test] - fn detect_with_query_string() { - let patterns = default_patterns(); - let result = - detect_inference_pattern("POST", "/v1/chat/completions?stream=true", &patterns); - assert!(result.is_some()); - } - - #[test] - fn no_match_for_get() { - let patterns = default_patterns(); - let result = detect_inference_pattern("GET", "/v1/chat/completions", &patterns); - assert!(result.is_none()); - } - - #[test] - fn detect_get_models() { - let patterns = default_patterns(); - let result = detect_inference_pattern("GET", "/v1/models", &patterns); - assert!(result.is_some()); - let pattern = result.unwrap(); - assert_eq!(pattern.protocol, "model_discovery"); - // A model list is one JSON object; it must be served buffered, never - // through the SSE streaming path that could append an error frame. - assert!(pattern.is_buffered()); - } - - #[test] - fn detect_get_model_details() { - let patterns = default_patterns(); - let result = detect_inference_pattern("GET", "/v1/models/gpt-4.1", &patterns); - assert!(result.is_some()); - let pattern = result.unwrap(); - assert_eq!(pattern.protocol, "model_discovery"); - assert!(pattern.is_buffered()); - } - - #[test] - fn detect_openai_embeddings() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/embeddings", &patterns); - assert!(result.is_some()); - let pattern = result.unwrap(); - assert_eq!(pattern.protocol, "openai_embeddings"); - assert_eq!(pattern.kind, "embeddings"); - assert!(pattern.is_buffered()); - } - - /// Every default pattern must declare framing consistent with how its - /// protocol actually responds: single-JSON-object protocols buffered, - /// SSE token streams streaming. A wrong classification routes a response - /// through the path that can corrupt or stall it. - #[test] - fn protocol_framing_classification() { - let patterns = default_patterns(); - for pattern in &patterns { - let expected_buffered = matches!( - pattern.protocol.as_str(), - "model_discovery" | "openai_embeddings" | "aws_bedrock_invoke" - ); - assert_eq!( - pattern.is_buffered(), - expected_buffered, - "{} ({}) has wrong framing", - pattern.protocol, - pattern.path_glob - ); - } - } - - #[test] - fn detect_aws_bedrock_invoke() { - let patterns = default_patterns(); - let result = - detect_inference_pattern("POST", "/model/anthropic.claude-opus-4-7/invoke", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "aws_bedrock_invoke"); - assert_eq!(result.unwrap().kind, "messages"); - } - - /// `InvokeModelWithResponseStream` is intentionally NOT advertised by - /// the default pattern set today. The shared streaming relay's - /// truncation/timeout/upstream-error path emits SSE-formatted error - /// frames, which would corrupt the AWS event-stream framing Bedrock - /// streams use. The pattern is restored alongside protocol-aware - /// error termination in a follow-up; until then, intercepted - /// `/invoke-with-response-stream` requests fall through to the - /// non-inference path rather than being mis-routed through the - /// SSE-injecting relay. - #[test] - fn aws_bedrock_invoke_stream_pattern_is_deferred() { - let patterns = default_patterns(); - assert!( - detect_inference_pattern( - "POST", - "/model/anthropic.claude-opus-4-7/invoke-with-response-stream", - &patterns, - ) - .is_none(), - "InvokeModelWithResponseStream must not be advertised until \ - protocol-aware AWS event-stream error framing exists" - ); - assert!( - !patterns - .iter() - .any(|p| p.protocol == "aws_bedrock_invoke_stream"), - "no pattern should declare protocol=aws_bedrock_invoke_stream" - ); - } - - #[test] - fn aws_bedrock_invoke_with_query_string() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/model/foo.bar/invoke?trace=1", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "aws_bedrock_invoke"); - } - - #[test] - fn aws_bedrock_rejects_empty_model_id() { - let patterns = default_patterns(); - // `/model//invoke` — empty wildcard segment is not a valid Bedrock id. - assert!(detect_inference_pattern("POST", "/model//invoke", &patterns).is_none()); - } - - #[test] - fn aws_bedrock_rejects_multi_segment_model_id() { - let patterns = default_patterns(); - // The `*` matches a single path segment only; multi-segment ids must - // not match (would be a path-traversal liability otherwise). - assert!(detect_inference_pattern("POST", "/model/foo/bar/invoke", &patterns).is_none()); - } - - #[test] - fn aws_bedrock_rejects_get() { - let patterns = default_patterns(); - assert!( - detect_inference_pattern("GET", "/model/anthropic.claude-opus-4-7/invoke", &patterns) - .is_none() - ); - } - - #[test] - fn aws_bedrock_rejects_unknown_action() { - let patterns = default_patterns(); - assert!(detect_inference_pattern("POST", "/model/foo/converse", &patterns).is_none()); - } - - /// `InvokeModel` returns one JSON object — must be served buffered. - /// Sending it through the streaming proxy would risk truncation or an - /// appended SSE error event corrupting the JSON body the caller decodes. - #[test] - fn aws_bedrock_invoke_is_buffered() { - let patterns = default_patterns(); - let invoke = - detect_inference_pattern("POST", "/model/anthropic.claude-opus-4-7/invoke", &patterns) - .expect("InvokeModel pattern must match"); - assert_eq!(invoke.protocol, "aws_bedrock_invoke"); - assert!( - invoke.is_buffered(), - "InvokeModel must be Buffered (one JSON object, accurate Content-Length); \ - streaming would risk corrupting the response" - ); - } - - #[test] - fn parse_simple_post_request() { - let body = b"{\"hello\":true}"; - let header = format!( - "POST /v1/chat/completions HTTP/1.1\r\nHost: api.openai.com\r\nContent-Length: {}\r\n\r\n", - body.len() - ); - let mut request = header.into_bytes(); - request.extend_from_slice(body); - let ParseResult::Complete(parsed, consumed) = try_parse_http_request(&request) else { - panic!("expected Complete"); - }; - assert_eq!(parsed.method, "POST"); - assert_eq!(parsed.path, "/v1/chat/completions"); - assert_eq!(parsed.body, body); - assert_eq!(consumed, request.len()); - } - - #[test] - fn parse_incomplete_headers() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: api.openai.com\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Incomplete - )); - } - - #[test] - fn parse_chunked_decodes_body() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: api.openai.com\r\nTransfer-Encoding: chunked\r\n\r\nA\r\n{\"a\":true}\r\n0\r\n\r\n"; - let ParseResult::Complete(parsed, consumed) = try_parse_http_request(request) else { - panic!("expected Complete"); - }; - assert_eq!(parsed.body, br#"{"a":true}"#); - assert_eq!(consumed, request.len()); - } - - #[test] - fn parse_chunked_incomplete() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n4\r\n{\"a\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Incomplete - )); - } - - #[test] - fn format_response_basic() { - let body = b"{\"ok\":true}"; - let response = format_http_response(200, &[], body); - let response_str = String::from_utf8_lossy(&response); - assert!(response_str.starts_with("HTTP/1.1 200 OK\r\n")); - assert!(response_str.contains("content-length: 11\r\n")); - assert!(response_str.ends_with("{\"ok\":true}")); - } - - #[test] - fn format_response_header_chunked() { - let headers = vec![ - ("content-type".to_string(), "text/event-stream".to_string()), - ("x-request-id".to_string(), "abc123".to_string()), - ]; - let header = format_http_response_header(200, &headers); - let header_str = String::from_utf8_lossy(&header); - assert!(header_str.starts_with("HTTP/1.1 200 OK\r\n")); - assert!(header_str.contains("content-type: text/event-stream\r\n")); - assert!(header_str.contains("x-request-id: abc123\r\n")); - assert!(header_str.contains("transfer-encoding: chunked\r\n")); - assert!(header_str.ends_with("\r\n")); - // Must NOT contain content-length - assert!(!header_str.to_lowercase().contains("content-length")); - } - - #[test] - fn format_response_header_strips_upstream_framing() { - let headers = vec![ - ("content-length".to_string(), "9999".to_string()), - ("transfer-encoding".to_string(), "chunked".to_string()), - ("content-type".to_string(), "application/json".to_string()), - ]; - let header = format_http_response_header(200, &headers); - let header_str = String::from_utf8_lossy(&header); - // Should not contain the upstream content-length or transfer-encoding values - assert!(!header_str.contains("content-length: 9999")); - // Should contain exactly one transfer-encoding: chunked (ours) - assert_eq!(header_str.matches("transfer-encoding: chunked").count(), 1); - } - - #[test] - fn format_chunk_basic() { - let data = b"hello"; - let chunk = format_chunk(data); - assert_eq!(chunk, b"5\r\nhello\r\n"); - } - - #[test] - fn format_chunk_empty() { - // Empty chunk is NOT the terminator — it's a zero-length data segment - let chunk = format_chunk(b""); - assert_eq!(chunk, b"0\r\n\r\n"); - } - - #[test] - fn format_chunk_terminator_value() { - assert_eq!(format_chunk_terminator(), b"0\r\n\r\n"); - } - - #[test] - fn format_chunk_large_hex() { - let data = vec![0x41u8; 256]; // 0x100 bytes - let chunk = format_chunk(&data); - assert!(chunk.starts_with(b"100\r\n")); - assert!(chunk.ends_with(b"\r\n")); - assert_eq!(chunk.len(), 3 + 2 + 256 + 2); // "100" + \r\n + data + \r\n - } - - // ---- SEC-010: parse_chunked_body resource limits ---- - - #[test] - fn parse_chunked_multi_chunk_body() { - // Two chunks: 5 bytes + 6 bytes - let request = b"POST /v1/chat HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n6\r\n world\r\n0\r\n\r\n"; - let ParseResult::Complete(parsed, _) = try_parse_http_request(request) else { - panic!("expected Complete"); - }; - assert_eq!(parsed.body, b"hello world"); - } - - #[test] - fn parse_chunked_rejects_too_many_chunks() { - // Build a request with MAX_CHUNK_COUNT + 1 tiny chunks - let mut buf = Vec::new(); - buf.extend_from_slice(b"POST /v1/chat HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n"); - for _ in 0..=MAX_CHUNK_COUNT { - buf.extend_from_slice(b"1\r\nX\r\n"); - } - buf.extend_from_slice(b"0\r\n\r\n"); - assert!(matches!( - try_parse_http_request(&buf), - ParseResult::Incomplete - )); - } - - #[test] - fn parse_chunked_within_chunk_count_limit() { - // MAX_CHUNK_COUNT chunks should succeed - let mut buf = Vec::new(); - buf.extend_from_slice(b"POST /v1/chat HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n"); - for _ in 0..100 { - buf.extend_from_slice(b"1\r\nX\r\n"); - } - buf.extend_from_slice(b"0\r\n\r\n"); - let ParseResult::Complete(parsed, _) = try_parse_http_request(&buf) else { - panic!("expected Complete for 100 chunks"); - }; - assert_eq!(parsed.body.len(), 100); - } - - /// SEC: Transfer-Encoding substring match must not match partial tokens. - #[test] - fn te_substring_not_chunked() { - let body = r#"{"model":"m","messages":[]}"#; - let request = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: x\r\n\ - Transfer-Encoding: chunkedx\r\n\ - Content-Length: {}\r\n\ - \r\n{body}", - body.len(), - ); - let ParseResult::Complete(parsed, _) = try_parse_http_request(request.as_bytes()) else { - panic!("expected Complete for non-matching TE with valid CL"); - }; - assert_eq!(parsed.body.len(), body.len()); - } - - // ---- SEC: Content-Length validation ---- - - #[test] - fn reject_differing_duplicate_content_length() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: 0\r\nContent-Length: 50\r\n\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) if reason.contains("differing values") - )); - } - - #[test] - fn accept_identical_duplicate_content_length() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: 5\r\nContent-Length: 5\r\n\r\nhello"; - let ParseResult::Complete(parsed, _) = try_parse_http_request(request) else { - panic!("expected Complete for identical duplicate CL"); - }; - assert_eq!(parsed.body, b"hello"); - } - - #[test] - fn reject_non_numeric_content_length() { - let request = - b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: abc\r\n\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) if reason.contains("invalid Content-Length") - )); - } - - #[test] - fn reject_two_non_numeric_content_lengths() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: abc\r\nContent-Length: def\r\n\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Invalid(_) - )); - } - - // ---- SEC-009: CL/TE desynchronisation ---- - - /// Reject requests with both Content-Length and Transfer-Encoding to - /// prevent CL/TE request smuggling (RFC 7230 Section 3.3.3). - #[test] - fn reject_dual_content_length_and_transfer_encoding() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\n"; - assert!( - matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) - if reason.contains("Transfer-Encoding") - && reason.contains("Content-Length") - ), - "Must reject request with both CL and TE" - ); - } - - /// Same rejection regardless of header order. - #[test] - fn reject_dual_transfer_encoding_and_content_length() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nTransfer-Encoding: chunked\r\nContent-Length: 5\r\n\r\n"; - assert!( - matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) - if reason.contains("Transfer-Encoding") - && reason.contains("Content-Length") - ), - "Must reject request with both TE and CL" - ); - } - - #[test] - fn format_sse_error_produces_valid_sse_json() { - let output = format_sse_error("chunk idle timeout exceeded"); - let text = std::str::from_utf8(&output).expect("should be valid utf8"); - - // Must start with "data: " (SSE format) - assert!(text.starts_with("data: "), "must be an SSE data line"); - - // Must end with double newline (SSE event boundary) - assert!(text.ends_with("\n\n"), "must end with SSE event boundary"); - - // The JSON payload between "data: " and "\n\n" must parse - let json_str = text.trim_start_matches("data: ").trim_end(); - let parsed: serde_json::Value = serde_json::from_str(json_str).expect("must be valid JSON"); - - assert_eq!(parsed["error"]["type"], "proxy_stream_error"); - assert_eq!(parsed["error"]["message"], "chunk idle timeout exceeded"); - } - - #[test] - fn format_sse_error_escapes_quotes_in_reason() { - let output = format_sse_error("error: \"bad\" response"); - let text = std::str::from_utf8(&output).unwrap(); - let json_str = text.trim_start_matches("data: ").trim_end(); - let parsed: serde_json::Value = - serde_json::from_str(json_str).expect("must produce valid JSON with escaped quotes"); - assert_eq!(parsed["error"]["message"], "error: \"bad\" response"); - } - - #[test] - fn format_sse_error_escapes_control_characters_in_reason() { - // A future caller passing a dynamic upstream error message (containing - // \n, \r, or \t — common in connection-reset errors and tracebacks) - // must still produce parseable SSE JSON. - let output = format_sse_error("upstream error: connection\nreset\tafter 0 bytes"); - let text = std::str::from_utf8(&output).unwrap(); - let json_str = text.trim_start_matches("data: ").trim_end(); - let parsed: serde_json::Value = serde_json::from_str(json_str) - .expect("must produce valid JSON when reason contains control characters"); - assert_eq!( - parsed["error"]["message"], - "upstream error: connection\nreset\tafter 0 bytes" - ); - } - - #[test] - fn format_sse_error_does_not_inject_extra_sse_events() { - // SSE events are separated by `\n\n`. If the reason string contains - // `\n\n`, an unescaped formatter would split the single error event - // into two SSE frames, allowing a malicious upstream to inject a - // forged event into the client's perceived stream - // (e.g. a fake tool_call delta). - let output = format_sse_error( - "safe prefix\n\ndata: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"id\":\"FORGED\"}]}}]}", - ); - let text = std::str::from_utf8(&output).unwrap(); - - // Exactly one SSE event boundary (the trailing one) — the reason - // string must not introduce additional `\n\n` sequences. - let boundary_count = text.matches("\n\n").count(); - assert_eq!( - boundary_count, 1, - "format_sse_error must emit exactly one SSE event boundary; \ - reason string must not be able to inject extra events" - ); - } -} diff --git a/crates/openshell-supervisor-network/src/l7/mod.rs b/crates/openshell-supervisor-network/src/l7/mod.rs index 70a980ba2d..0751289547 100644 --- a/crates/openshell-supervisor-network/src/l7/mod.rs +++ b/crates/openshell-supervisor-network/src/l7/mod.rs @@ -10,7 +10,6 @@ pub mod graphql; pub(crate) mod http; -pub mod inference; pub mod jsonrpc; pub(crate) mod middleware; pub mod path; diff --git a/crates/openshell-supervisor-network/src/lib.rs b/crates/openshell-supervisor-network/src/lib.rs index 4fec48b300..e00a5bf484 100644 --- a/crates/openshell-supervisor-network/src/lib.rs +++ b/crates/openshell-supervisor-network/src/lib.rs @@ -4,12 +4,11 @@ //! Networking component of the `OpenShell` supervisor. //! //! Owns the egress proxy, L7 enforcement, OPA policy engine, identity cache, -//! inference routing, and TLS interception. The denial-event channel is +//! TLS interception, and credential injection. The denial-event channel is //! owned by the orchestrator; this crate produces denials but does not //! aggregate them. pub mod identity; -pub mod inference_routes; pub mod l7; pub mod opa; pub(crate) mod policy_dns; diff --git a/crates/openshell-supervisor-network/src/opa.rs b/crates/openshell-supervisor-network/src/opa.rs index 63aa2c2c70..4d09db8273 100644 --- a/crates/openshell-supervisor-network/src/opa.rs +++ b/crates/openshell-supervisor-network/src/opa.rs @@ -5730,7 +5730,7 @@ process: // network_action tests // ======================================================================== - const INFERENCE_TEST_DATA: &str = r#" + const PROVIDER_ENDPOINT_TEST_DATA: &str = r#" network_policies: claude_code: name: claude_code @@ -5755,7 +5755,7 @@ process: run_as_group: sandbox "#; - const NO_INFERENCE_TEST_DATA: &str = r#" + const OTHER_ENDPOINT_TEST_DATA: &str = r#" network_policies: gitlab: name: gitlab @@ -5774,19 +5774,19 @@ process: run_as_group: sandbox "#; - fn inference_engine() -> OpaEngine { - OpaEngine::from_strings(TEST_POLICY, INFERENCE_TEST_DATA) - .expect("Failed to load inference test data") + fn provider_endpoint_engine() -> OpaEngine { + OpaEngine::from_strings(TEST_POLICY, PROVIDER_ENDPOINT_TEST_DATA) + .expect("Failed to load provider endpoint test data") } - fn no_inference_engine() -> OpaEngine { - OpaEngine::from_strings(TEST_POLICY, NO_INFERENCE_TEST_DATA) - .expect("Failed to load no-inference test data") + fn other_endpoint_engine() -> OpaEngine { + OpaEngine::from_strings(TEST_POLICY, OTHER_ENDPOINT_TEST_DATA) + .expect("Failed to load alternate endpoint test data") } #[test] fn explicitly_allowed_endpoint_binary_returns_allow() { - let engine = inference_engine(); + let engine = provider_endpoint_engine(); let input = NetworkInput { host: "api.anthropic.com".into(), port: 443, @@ -5808,7 +5808,7 @@ process: fn relaxed_binary_identity_allows_declared_endpoint_without_binary_match() { let engine = OpaEngine::from_strings_with_binary_identity_required( TEST_POLICY, - INFERENCE_TEST_DATA, + PROVIDER_ENDPOINT_TEST_DATA, false, ) .expect("Failed to load relaxed binary identity test data"); @@ -5846,7 +5846,7 @@ process: #[test] fn unknown_endpoint_returns_deny() { - let engine = inference_engine(); + let engine = provider_endpoint_engine(); let input = NetworkInput { host: "api.openai.com".into(), port: 443, @@ -5863,8 +5863,8 @@ process: } #[test] - fn unknown_endpoint_without_inference_returns_deny() { - let engine = no_inference_engine(); + fn unknown_endpoint_with_other_policy_returns_deny() { + let engine = other_endpoint_engine(); let input = NetworkInput { host: "api.openai.com".into(), port: 443, @@ -5884,7 +5884,7 @@ process: fn endpoint_in_policy_binary_not_allowed_returns_deny() { // api.anthropic.com is declared but python3 is not in the binary list. // With binary allow/deny, this is denied. - let engine = inference_engine(); + let engine = provider_endpoint_engine(); let input = NetworkInput { host: "api.anthropic.com".into(), port: 443, @@ -5901,8 +5901,8 @@ process: } #[test] - fn endpoint_in_policy_binary_not_allowed_without_inference_returns_deny() { - let engine = no_inference_engine(); + fn endpoint_in_policy_binary_not_allowed_with_other_policy_returns_deny() { + let engine = other_endpoint_engine(); let input = NetworkInput { host: "gitlab.com".into(), port: 443, diff --git a/crates/openshell-supervisor-network/src/proxy.rs b/crates/openshell-supervisor-network/src/proxy.rs index dc2736a4ea..10fce0ea3e 100644 --- a/crates/openshell-supervisor-network/src/proxy.rs +++ b/crates/openshell-supervisor-network/src/proxy.rs @@ -25,9 +25,8 @@ use openshell_core::policy::ProxyPolicy; use openshell_core::provider_credentials::{ProviderCredentialSnapshot, ProviderCredentialState}; use openshell_core::secrets::{self, SecretResolver, rewrite_header_line_checked}; use openshell_ocsf::{ - ActionId, ActivityId, AiModel, ApiActivityBuilder, DispositionId, Endpoint, - HttpActivityBuilder, HttpRequest, NetworkActivityBuilder, Process, SeverityId, StatusId, - Url as OcsfUrl, ocsf_emit, + ActionId, ActivityId, DispositionId, Endpoint, HttpActivityBuilder, HttpRequest, + NetworkActivityBuilder, Process, SeverityId, StatusId, Url as OcsfUrl, ocsf_emit, }; #[cfg(target_os = "linux")] use std::mem::size_of; @@ -62,8 +61,6 @@ const TUNNEL_PROTOCOL_PEEK_TIMEOUT: std::time::Duration = std::time::Duration::f const TUNNEL_PROTOCOL_PEEK_POLL: std::time::Duration = std::time::Duration::from_millis(5); #[cfg(test)] const TUNNEL_PROTOCOL_PEEK_POLL: std::time::Duration = std::time::Duration::from_millis(1); -const INFERENCE_LOCAL_HOST: &str = "inference.local"; -const INFERENCE_LOCAL_PORT: u16 = 443; const FORWARD_ENCODED_SLASH_REJECTION_DETAIL: &str = "request-target contains an encoded '/' (%2F) which is not allowed on this endpoint"; #[cfg(target_os = "linux")] @@ -132,102 +129,6 @@ const CLOUD_METADATA_IPS: &[IpAddr] = &[ IpAddr::V4(std::net::Ipv4Addr::new(169, 254, 169, 254)), ]; -/// Maximum total bytes for a streaming inference response body (32 MiB). -#[cfg(not(test))] -const MAX_STREAMING_BODY: usize = 32 * 1024 * 1024; -// Keep unit tests deterministic without pushing tens of MiB through loopback. -#[cfg(test)] -const MAX_STREAMING_BODY: usize = 1024; - -/// Idle timeout per chunk when relaying streaming inference responses. -/// -/// Reasoning models (e.g. nemotron-3-super, o1, o3) can pause for 60+ seconds -/// between "thinking" and output phases. 120s provides headroom while still -/// catching genuinely stuck streams. -#[cfg(not(test))] -const CHUNK_IDLE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(120); -// Exercise idle-timeout truncation without slowing the full package test suite. -#[cfg(test)] -const CHUNK_IDLE_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(100); - -/// Outcome of an inference interception attempt. -/// -/// Returned by [`handle_inference_interception`] so the call site can emit -/// a structured CONNECT deny log when the connection is not successfully routed. -#[derive(Debug)] -enum InferenceOutcome { - /// At least one request was successfully routed to a local inference backend. - Routed, - /// The connection was denied (TLS failure, non-inference request, etc.). - Denied { reason: String }, -} - -/// Inference routing context for sandbox-local execution. -/// -/// Holds a `Router` (HTTP client) and cached sets of resolved routes. -/// User routes serve `inference.local` traffic; system routes are consumed -/// in-process by the supervisor for platform functions (e.g. agent harness). -pub struct InferenceContext { - pub patterns: Vec, - router: openshell_router::Router, - /// Routes for the user-facing `inference.local` endpoint. - routes: Arc>>, - /// Routes for supervisor-only system inference (`sandbox-system`). - system_routes: Arc>>, -} - -impl InferenceContext { - // `router`/`routes` are intentionally distinct nouns (the router and the - // route list it consumes); both names are clearer than alternatives. - #[allow(clippy::similar_names)] - pub fn new( - patterns: Vec, - router: openshell_router::Router, - routes: Vec, - system_routes: Vec, - ) -> Self { - Self { - patterns, - router, - routes: Arc::new(tokio::sync::RwLock::new(routes)), - system_routes: Arc::new(tokio::sync::RwLock::new(system_routes)), - } - } - - /// Get a handle to the user route cache for background refresh. - pub fn route_cache( - &self, - ) -> Arc>> { - self.routes.clone() - } - - /// Get a handle to the system route cache for background refresh. - pub fn system_route_cache( - &self, - ) -> Arc>> { - self.system_routes.clone() - } - - /// Make an inference call using system routes (supervisor-only). - /// - /// This is the in-process API for platform functions. It bypasses the - /// CONNECT proxy entirely — the supervisor calls the router directly - /// from the host network namespace. - pub async fn system_inference( - &self, - protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, - ) -> Result { - let routes = self.system_routes.read().await; - self.router - .proxy_with_candidates(protocol, method, path, headers, body, &routes) - .await - } -} - pub struct ProxyHandle { #[allow(dead_code)] http_addr: Option, @@ -248,7 +149,6 @@ impl ProxyHandle { identity_cache: Arc, entrypoint_pid: Arc, tls_state: Option>, - inference_ctx: Option>, provider_credentials: Option, policy_local_ctx: Option>, denial_tx: Option>, @@ -380,7 +280,6 @@ impl ProxyHandle { let cache = identity_cache.clone(); let spid = entrypoint_pid.clone(); let tls = tls_state.clone(); - let inf = inference_ctx.clone(); let policy_local = policy_local_ctx.clone(); let proposals = agent_proposals.clone(); let gw = trusted_host_gateway.clone(); @@ -404,7 +303,6 @@ impl ProxyHandle { cache, spid, tls, - inf, policy_local, proposals, gw, @@ -1688,7 +1586,6 @@ async fn handle_tcp_connection( identity_cache: Arc, entrypoint_pid: Arc, tls_state: Option>, - inference_ctx: Option>, policy_local_ctx: Option>, agent_proposals: openshell_core::proposals::AgentProposals, trusted_host_gateway: Arc>, @@ -1776,33 +1673,6 @@ async fn handle_tcp_connection( let host = normalize_host(&raw_host); let (host_lc, raw_host_lc) = (host.to_ascii_lowercase(), raw_host.to_ascii_lowercase()); - if host_lc == INFERENCE_LOCAL_HOST && port == INFERENCE_LOCAL_PORT { - respond(&mut client, b"HTTP/1.1 200 Connection Established\r\n\r\n").await?; - let outcome = handle_inference_interception( - client, - INFERENCE_LOCAL_HOST, - port, - tls_state.as_ref(), - inference_ctx.as_ref(), - ) - .await?; - if let InferenceOutcome::Denied { reason } = outcome { - emit_activity(&activity_tx, true, "forward_policy"); - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, port)) - .message(format!("Inference interception denied: {reason}")) - .status_detail(&reason) - .build(); - ocsf_emit!(event); - } - return Ok(()); - } - let workload_addr = client.peer_addr().into_diagnostic()?; let proxy_addr = client.local_addr().into_diagnostic()?; let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); @@ -2810,600 +2680,6 @@ fn authorize_egress_intent( } } -/// Maximum buffer size for inference request parsing (10 MiB). -const MAX_INFERENCE_BUF: usize = 10 * 1024 * 1024; - -/// Initial buffer size for inference request parsing (64 KiB). -const INITIAL_INFERENCE_BUF: usize = 65536; - -/// Handle an intercepted connection for inference routing. -/// -/// TLS-terminates the client connection, parses HTTP requests, and executes -/// inference API calls locally via `openshell-router`. -/// Non-inference requests are denied with 403. -/// -/// Returns [`InferenceOutcome::Routed`] if at least one request was successfully -/// routed, or [`InferenceOutcome::Denied`] with a reason for all denial cases. -async fn handle_inference_interception( - client: TcpStream, - host: &str, - port: u16, - tls_state: Option<&Arc>, - inference_ctx: Option<&Arc>, -) -> Result { - let Some(ctx) = inference_ctx else { - return Ok(InferenceOutcome::Denied { - reason: "cluster inference context not configured".to_string(), - }); - }; - - let Some(tls) = tls_state else { - return Ok(InferenceOutcome::Denied { - reason: "missing TLS state".to_string(), - }); - }; - - // TLS-terminate the client side (present a cert for the target host) - let mut tls_client = match crate::l7::tls::tls_terminate_client(client, tls, host).await { - Ok(c) => c, - Err(e) => { - return Ok(InferenceOutcome::Denied { - reason: format!("TLS handshake failed: {e}"), - }); - } - }; - - process_inference_keepalive(&mut tls_client, ctx, port).await -} - -/// Read and process HTTP requests from a TLS-terminated inference connection. -/// -/// Each request is matched against inference patterns and routed locally. -/// Any non-inference request is immediately denied and the connection is closed, -/// even if previous requests on the same keep-alive connection were routed -/// successfully. -async fn process_inference_keepalive( - stream: &mut S, - ctx: &InferenceContext, - port: u16, -) -> Result { - use crate::l7::inference::{ParseResult, format_http_response, try_parse_http_request}; - - let mut buf = vec![0u8; INITIAL_INFERENCE_BUF]; - let mut used = 0usize; - let mut routed_any = false; - - loop { - let n = match stream.read(&mut buf[used..]).await { - Ok(n) => n, - Err(e) => { - if routed_any { - break; - } - return Ok(InferenceOutcome::Denied { - reason: format!("I/O error: {e}"), - }); - } - }; - if n == 0 { - if routed_any { - break; - } - return Ok(InferenceOutcome::Denied { - reason: "client closed connection".to_string(), - }); - } - used += n; - - // Try to parse a complete HTTP request - match try_parse_http_request(&buf[..used]) { - ParseResult::Complete(request, consumed) => { - let was_routed = route_inference_request(&request, ctx, stream).await?; - if was_routed { - routed_any = true; - } else { - // Deny and close: a non-inference request must not be silently - // ignored on a keep-alive connection that previously routed - // inference traffic. - return Ok(InferenceOutcome::Denied { - reason: "connection not allowed by policy".to_string(), - }); - } - - // Shift buffer for next request - buf.copy_within(consumed..used, 0); - used -= consumed; - } - ParseResult::Incomplete => { - // Need more data — grow buffer if full - if used == buf.len() { - if buf.len() >= MAX_INFERENCE_BUF { - let response = format_http_response(413, &[], b"Payload Too Large"); - write_all(stream, &response).await?; - if routed_any { - break; - } - return Ok(InferenceOutcome::Denied { - reason: "payload too large".to_string(), - }); - } - buf.resize((buf.len() * 2).min(MAX_INFERENCE_BUF), 0); - } - } - ParseResult::Invalid(reason) => { - { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Refuse) - .action(ActionId::Denied) - .disposition(DispositionId::Rejected) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, port)) - .message(format!("Rejecting malformed inference request: {reason}")) - .status_detail(&reason) - .build(); - ocsf_emit!(event); - } - let response = format_http_response(400, &[], b"Bad Request"); - write_all(stream, &response).await?; - return Ok(InferenceOutcome::Denied { reason }); - } - } - } - - Ok(InferenceOutcome::Routed) -} - -/// Extract the model name from an inference request body. -fn extract_model_from_request(body: &[u8]) -> Option { - #[derive(serde::Deserialize)] - struct Req { - model: Option, - } - serde_json::from_slice::(body) - .ok() - .and_then(|r| r.model) - .filter(|m| !m.is_empty()) -} - -/// Extract token usage from an inference response body. -fn extract_usage_from_response(body: &[u8]) -> (Option, Option) { - #[derive(serde::Deserialize)] - struct Resp { - usage: Option, - } - #[derive(serde::Deserialize)] - struct Usage { - prompt_tokens: Option, - completion_tokens: Option, - } - match serde_json::from_slice::(body) { - Ok(r) => { - let u = r.usage.unwrap_or(Usage { - prompt_tokens: None, - completion_tokens: None, - }); - (u.prompt_tokens, u.completion_tokens) - } - Err(_) => (None, None), - } -} - -/// Emit an OCSF API Activity [6003] event with the `ai_operation` profile after an inference call. -#[allow(clippy::too_many_arguments)] -fn emit_ai_inference( - method: &str, - path: &str, - route_model: Option<&str>, - route_provider: Option<&str>, - status: StatusId, - input_tokens: Option, - output_tokens: Option, - latency: std::time::Duration, -) { - let model_name = route_model.unwrap_or("unknown"); - let provider_name = route_provider.unwrap_or("unknown"); - let latency_ms = u64::try_from(latency.as_millis()).unwrap_or(u64::MAX); - - let operation = format!("{method} {path}"); - - let mut builder = ApiActivityBuilder::new(openshell_ocsf::ctx::ctx(), &operation) - .severity(SeverityId::Informational) - .status(status) - .ai_model(AiModel::new(model_name, provider_name)) - .http_request(HttpRequest::new( - method, - OcsfUrl::new("https", INFERENCE_LOCAL_HOST, path, 443), - )) - .dst_endpoint(Endpoint::from_domain( - INFERENCE_LOCAL_HOST, - INFERENCE_LOCAL_PORT, - )) - .unmapped("latency_ms", latency_ms); - - if let Some(t) = input_tokens { - builder = builder.unmapped("input_tokens", t); - } - if let Some(t) = output_tokens { - builder = builder.unmapped("output_tokens", t); - } - - let msg = format!( - "Model call: {model_name} via {provider_name} ({}in, {}out)", - input_tokens.map_or_else(|| "?".to_string(), |t| t.to_string()), - output_tokens.map_or_else(|| "?".to_string(), |t| t.to_string()), - ); - builder = builder.message(msg); - - ocsf_emit!(builder.build()); -} - -/// Route a parsed inference request locally via the sandbox router, or deny it. -/// -/// Returns `Ok(true)` if the request was routed to an inference backend, -/// `Ok(false)` if it was denied as a non-inference request. -async fn route_inference_request( - request: &crate::l7::inference::ParsedHttpRequest, - ctx: &InferenceContext, - tls_client: &mut (impl tokio::io::AsyncWrite + Unpin), -) -> Result { - use crate::l7::inference::{detect_inference_pattern, format_http_response}; - - let normalized_path = normalize_inference_path(&request.path); - - if let Some(pattern) = - detect_inference_pattern(&request.method, &normalized_path, &ctx.patterns) - { - { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Allowed) - .disposition(DispositionId::Detected) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "Intercepted inference request, routing locally: {} {} (protocol={}, kind={})", - request.method, normalized_path, pattern.protocol, pattern.kind - )) - .build(); - ocsf_emit!(event); - } - - let routes = ctx.routes.read().await; - - if routes.is_empty() { - let body = serde_json::json!({ - "error": "cluster inference is not configured", - "hint": "run: openshell cluster inference set --help" - }); - let body_bytes = body.to_string(); - let response = format_http_response( - 503, - &[("content-type".to_string(), "application/json".to_string())], - body_bytes.as_bytes(), - ); - write_all(tls_client, &response).await?; - return Ok(true); - } - - // Buffered protocols (embeddings, model discovery) return a single JSON - // object, not an SSE token stream. Serve them buffered with an accurate - // Content-Length: the streaming path would append an SSE error frame to - // the body on a size-cap or idle-timeout truncation, corrupting a - // payload the client parses as one JSON object. Framing is declared per - // protocol on the matched pattern. - let _req_model = extract_model_from_request(&request.body); - let normalized_protocol = pattern.protocol.to_ascii_lowercase(); - let selected_route = routes - .iter() - .find(|r| r.protocols.iter().any(|p| p == &normalized_protocol)) - .or_else(|| routes.first()); - let route_model = selected_route.map(|r| r.model.clone()); - let route_endpoint = selected_route.map(|r| r.endpoint.clone()); - let infer_start = std::time::Instant::now(); - - if pattern.is_buffered() { - match ctx - .router - .proxy_with_candidates( - &pattern.protocol, - &request.method, - &normalized_path, - request.headers.clone(), - bytes::Bytes::from(request.body.clone()), - &routes, - ) - .await - { - Ok(resp) => { - let (input_tokens, output_tokens) = extract_usage_from_response(&resp.body); - let resp_status = if (200..300).contains(&resp.status) { - StatusId::Success - } else { - StatusId::Failure - }; - emit_ai_inference( - &request.method, - &normalized_path, - resp.route_model.as_deref(), - resp.route_endpoint.as_deref(), - resp_status, - input_tokens, - output_tokens, - infer_start.elapsed(), - ); - - let resp_headers = sanitize_inference_response_headers(resp.headers); - let response = format_http_response(resp.status, &resp_headers, &resp.body); - write_all(tls_client, &response).await?; - } - Err(e) => { - emit_ai_inference( - &request.method, - &normalized_path, - route_model.as_deref(), - route_endpoint.as_deref(), - StatusId::Failure, - None, - None, - infer_start.elapsed(), - ); - write_inference_router_error(tls_client, &e).await?; - } - } - return Ok(true); - } - - match ctx - .router - .proxy_with_candidates_streaming( - &pattern.protocol, - &request.method, - &normalized_path, - request.headers.clone(), - bytes::Bytes::from(request.body.clone()), - &routes, - ) - .await - { - Ok(mut resp) => { - use crate::l7::inference::{ - format_chunk, format_chunk_terminator, format_http_response_header, - format_sse_error, - }; - - let stream_route_model = resp.route_model.take(); - let stream_route_endpoint = resp.route_endpoint.take(); - - let resp_headers = sanitize_inference_response_headers( - std::mem::take(&mut resp.headers).into_iter().collect(), - ); - - // Write response headers immediately (chunked TE). - let header_bytes = format_http_response_header(resp.status, &resp_headers); - write_all(tls_client, &header_bytes).await?; - - // Stream body chunks with byte cap and idle timeout. - // - // Each upstream chunk is wrapped in HTTP chunked framing and - // flushed immediately so SSE events reach the client without - // delay. Unlike the previous per-byte write_all+flush, we - // coalesce the framing header + data + trailer into a single - // write_all call, reducing the number of TLS records per chunk - // from 3 to 1 while preserving incremental delivery. - let resp_status_ok = (200..300).contains(&resp.status); - let mut total_bytes: usize = 0; - let mut stream_failed = false; - loop { - match tokio::time::timeout(CHUNK_IDLE_TIMEOUT, resp.next_chunk()).await { - Ok(Ok(Some(chunk))) => { - total_bytes += chunk.len(); - if total_bytes > MAX_STREAMING_BODY { - warn!( - total_bytes = total_bytes, - limit = MAX_STREAMING_BODY, - "streaming response exceeded byte limit, truncating" - ); - let err = format_sse_error( - "response truncated: exceeded maximum streaming body size", - ); - let _ = write_all(tls_client, &format_chunk(&err)).await; - stream_failed = true; - break; - } - let encoded = format_chunk(&chunk); - write_all(tls_client, &encoded).await?; - } - Ok(Ok(None)) => break, - Ok(Err(e)) => { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "error reading upstream response chunk after \ - {total_bytes} bytes: {e}" - )) - .build(); - ocsf_emit!(event); - let err = format_sse_error("response truncated: upstream read error"); - let _ = write_all(tls_client, &format_chunk(&err)).await; - stream_failed = true; - break; - } - Err(_) => { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "streaming response chunk idle timeout after \ - {total_bytes} bytes, closing" - )) - .build(); - ocsf_emit!(event); - let err = - format_sse_error("response truncated: chunk idle timeout exceeded"); - let _ = write_all(tls_client, &format_chunk(&err)).await; - stream_failed = true; - break; - } - } - } - - // Terminate the chunked stream. - write_all(tls_client, format_chunk_terminator()).await?; - - // Emit API Activity for the completed streaming call. - let stream_status = if stream_failed || !resp_status_ok { - StatusId::Failure - } else { - StatusId::Success - }; - emit_ai_inference( - &request.method, - &normalized_path, - stream_route_model.as_deref(), - stream_route_endpoint.as_deref(), - stream_status, - None, - None, - infer_start.elapsed(), - ); - } - Err(e) => { - emit_ai_inference( - &request.method, - &normalized_path, - route_model.as_deref(), - route_endpoint.as_deref(), - StatusId::Failure, - None, - None, - infer_start.elapsed(), - ); - write_inference_router_error(tls_client, &e).await?; - } - } - Ok(true) - } else { - // Not an inference request — deny - { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "connection not allowed by policy: {} {}", - request.method, normalized_path - )) - .build(); - ocsf_emit!(event); - } - let body = serde_json::json!({"error": "connection not allowed by policy"}); - let body_bytes = body.to_string(); - let response = format_http_response( - 403, - &[("content-type".to_string(), "application/json".to_string())], - body_bytes.as_bytes(), - ); - write_all(tls_client, &response).await?; - Ok(false) - } -} - -/// Emit an OCSF failure event and write a buffered JSON error response for a -/// router error hit while proxying an inference request. -/// -/// Shared by the streaming and buffered routing paths so both surface upstream -/// failures with the same status mapping and the same audit record. -async fn write_inference_router_error( - tls_client: &mut (impl tokio::io::AsyncWrite + Unpin), - err: &openshell_router::RouterError, -) -> Result<()> { - use crate::l7::inference::format_http_response; - - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Low) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "inference endpoint detected but upstream service failed: {err}" - )) - .build(); - ocsf_emit!(event); - - let (status, msg) = router_error_to_http(err); - let body = serde_json::json!({ "error": msg }).to_string(); - let response = format_http_response( - status, - &[("content-type".to_string(), "application/json".to_string())], - body.as_bytes(), - ); - write_all(tls_client, &response).await -} - -/// Map router errors to HTTP status codes and sanitized messages. -/// -/// Returns generic, client-safe messages instead of verbatim internal details; -/// the full error is recorded in the OCSF failure event by the caller. -fn router_error_to_http(err: &openshell_router::RouterError) -> (u16, String) { - use openshell_router::RouterError; - match err { - RouterError::RouteNotFound(_) => (400, "no inference route configured".to_string()), - RouterError::NoCompatibleRoute(_) => { - (400, "no compatible inference route available".to_string()) - } - RouterError::Unauthorized(_) => (401, "unauthorized".to_string()), - RouterError::UpstreamUnavailable(_) => (503, "inference service unavailable".to_string()), - RouterError::UpstreamProtocol(_) | RouterError::Internal(_) => { - (502, "inference service error".to_string()) - } - } -} - -fn sanitize_inference_response_headers(headers: Vec<(String, String)>) -> Vec<(String, String)> { - headers - .into_iter() - .filter(|(name, _)| !should_strip_response_header(name)) - .collect() -} - -fn should_strip_response_header(name: &str) -> bool { - let name_lc = name.to_ascii_lowercase(); - matches!(name_lc.as_str(), "content-length") || is_hop_by_hop_header(&name_lc) -} - -fn is_hop_by_hop_header(name: &str) -> bool { - matches!( - name, - "connection" - | "keep-alive" - | "proxy-authenticate" - | "proxy-authorization" - | "proxy-connection" - | "te" - | "trailer" - | "transfer-encoding" - | "upgrade" - ) -} - -/// Write all bytes to an async writer. -async fn write_all(writer: &mut (impl tokio::io::AsyncWrite + Unpin), data: &[u8]) -> Result<()> { - use tokio::io::AsyncWriteExt; - writer.write_all(data).await.into_diagnostic()?; - writer.flush().await.into_diagnostic()?; - Ok(()) -} - fn emit_l7_tunnel_close_after_policy_change(host: &str, port: u16, error: miette::Report) { let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Open) @@ -4238,23 +3514,6 @@ fn endpoint_config_string_array(config: ®orus::Value, key: &str) -> Vec String { - match crate::l7::path::canonicalize_request_target( - path, - &crate::l7::path::CanonicalizeOptions::default(), - ) { - Ok((canon, _)) => canon.path, - Err(_) => path.to_string(), - } -} - /// Extract the hostname from an absolute-form URI used in plain HTTP proxy requests. /// /// For example, `"http://example.com/path"` yields `"example.com"` and @@ -6349,10 +5608,9 @@ mod tests { use super::*; use openshell_core::proposals::AgentProposals; use std::collections::HashMap as TestHashMap; - use std::future::Future; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; use std::sync::Arc; - use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; struct DenyWebSocketPreflight; @@ -6526,7 +5784,6 @@ network_policies: {} Arc::new(AtomicU32::new(std::process::id())), None, None, - None, AgentProposals::default(), Arc::new(None), Arc::new(None), @@ -9362,724 +8619,7 @@ network_policies: ); } - #[tokio::test] - async fn inference_interception_applies_router_header_allowlist() { - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - use tokio::net::TcpListener; - - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_task = tokio::spawn(async move { - use crate::l7::inference::{ParseResult, try_parse_http_request}; - - let (mut upstream, _) = listener.accept().await.unwrap(); - let mut buf = Vec::new(); - let mut chunk = [0u8; 4096]; - - loop { - let n = upstream.read(&mut chunk).await.unwrap(); - assert!(n > 0, "upstream request closed before request completed"); - buf.extend_from_slice(&chunk[..n]); - - match try_parse_http_request(&buf) { - ParseResult::Complete(_, consumed) => { - upstream - .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok") - .await - .unwrap(); - return String::from_utf8_lossy(&buf[..consumed]).to_string(); - } - ParseResult::Incomplete => continue, - ParseResult::Invalid(reason) => { - panic!("forwarded request should parse cleanly: {reason}"); - } - } - } - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!("http://{upstream_addr}"), - model: "meta/llama-3.1-8b-instruct".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![ - "openai-organization".to_string(), - "x-model-id".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }], - vec![], - ); - - let body = r#"{"model":"ignored","messages":[{"role":"user","content":"hi"}]}"#; - let request = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - OpenAI-Organization: org_123\r\n\ - Authorization: Bearer client-key\r\n\ - Cookie: session=abc\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response_text = String::from_utf8_lossy(&response); - assert!(response_text.starts_with("HTTP/1.1 200")); - - let outcome = server_task.await.unwrap().unwrap(); - assert!( - matches!(outcome, InferenceOutcome::Routed), - "expected Routed outcome, got: {outcome:?}" - ); - - let forwarded = upstream_task.await.unwrap(); - let forwarded_lc = forwarded.to_ascii_lowercase(); - assert!(forwarded_lc.contains("openai-organization: org_123")); - assert!(forwarded_lc.contains("authorization: bearer test-api-key")); - assert!(!forwarded_lc.contains("authorization: bearer client-key")); - assert!(!forwarded_lc.contains("cookie:")); - } - - fn streaming_inference_route(endpoint: String) -> openshell_router::config::ResolvedRoute { - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint, - model: "meta/llama-3.1-8b-instruct".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - fn embeddings_inference_route(endpoint: String) -> openshell_router::config::ResolvedRoute { - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint, - model: "text-embedding-3-small".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_embeddings".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - /// Embeddings responses are a single buffered JSON object, not an SSE - /// stream. They must be framed with `Content-Length` and must never be sent - /// through the chunked streaming path, whose truncation handlers would - /// append an SSE `proxy_stream_error` frame into the JSON body. - #[tokio::test] - async fn inference_embeddings_served_buffered_with_content_length() { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_body = r#"{"object":"list","data":[{"object":"embedding","index":0,"embedding":[0.1,0.2]}],"model":"text-embedding-3-small"}"#; - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - read_forwarded_inference_request(&mut upstream).await; - // Buffered upstream response with Content-Length (no chunked TE). - let resp = format!( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", - upstream_body.len(), - upstream_body, - ); - upstream.write_all(resp.as_bytes()).await.unwrap(); - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![embeddings_inference_route(format!( - "http://{upstream_addr}" - ))], - vec![], - ); - - let body = r#"{"model":"text-embedding-3-small","input":"hello"}"#; - let request = format!( - "POST /v1/embeddings HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - - server_task.await.unwrap().unwrap(); - upstream_task.await.unwrap(); - - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n"), - "expected buffered 200 response, got: {response}" - ); - let lower = response.to_ascii_lowercase(); - assert!( - lower.contains("content-length:"), - "embeddings response must be Content-Length framed, got: {response}" - ); - assert!( - !lower.contains("transfer-encoding: chunked"), - "embeddings response must NOT be chunked, got: {response}" - ); - assert!( - !response.contains("proxy_stream_error"), - "embeddings response must not carry an SSE error frame, got: {response}" - ); - assert!( - response.contains(r#""object":"list""#), - "embeddings JSON body must be forwarded intact, got: {response}" - ); - } - - fn model_discovery_inference_route( - endpoint: String, - ) -> openshell_router::config::ResolvedRoute { - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint, - model: "text-embedding-3-small".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["model_discovery".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - /// `GET /v1/models` (model discovery) returns one JSON object — a model - /// list — exactly like embeddings. It must be served buffered with - /// `Content-Length`, never through the chunked streaming path whose - /// truncation handlers would append an SSE `proxy_stream_error` frame into - /// the JSON body. This guards the framing classification for the protocol. - #[tokio::test] - async fn inference_model_discovery_served_buffered_with_content_length() { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_body = - r#"{"object":"list","data":[{"id":"text-embedding-3-small","object":"model"}]}"#; - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - read_forwarded_inference_request(&mut upstream).await; - let resp = format!( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", - upstream_body.len(), - upstream_body, - ); - upstream.write_all(resp.as_bytes()).await.unwrap(); - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![model_discovery_inference_route(format!( - "http://{upstream_addr}" - ))], - vec![], - ); - - // GET model discovery carries no request body. - let request = "GET /v1/models HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Length: 0\r\n\r\n" - .to_string(); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - - server_task.await.unwrap().unwrap(); - upstream_task.await.unwrap(); - - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n"), - "expected buffered 200 response, got: {response}" - ); - let lower = response.to_ascii_lowercase(); - assert!( - lower.contains("content-length:"), - "model discovery response must be Content-Length framed, got: {response}" - ); - assert!( - !lower.contains("transfer-encoding: chunked"), - "model discovery response must NOT be chunked, got: {response}" - ); - assert!( - !response.contains("proxy_stream_error"), - "model discovery response must not carry an SSE error frame, got: {response}" - ); - assert!( - response.contains(r#""object":"list""#), - "model discovery JSON body must be forwarded intact, got: {response}" - ); - } - - /// `GET /v1/models/{id}` (model discovery glob) must forward the model id in - /// the path through the buffered path with the id intact, never streamed. - #[tokio::test] - async fn inference_model_discovery_glob_path_served_buffered() { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_body = r#"{"id":"gpt-4.1","object":"model"}"#; - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - let forwarded = read_forwarded_request_line(&mut upstream).await; - let resp = format!( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", - upstream_body.len(), - upstream_body, - ); - upstream.write_all(resp.as_bytes()).await.unwrap(); - forwarded - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![model_discovery_inference_route(format!( - "http://{upstream_addr}" - ))], - vec![], - ); - - let request = "GET /v1/models/gpt-4.1 HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Length: 0\r\n\r\n" - .to_string(); - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - server_task.await.unwrap().unwrap(); - let (method, forwarded_path) = upstream_task.await.unwrap(); - - assert_eq!(method, "GET"); - assert_eq!( - forwarded_path, "/v1/models/gpt-4.1", - "the model id in the glob path must be forwarded intact" - ); - let lower = response.to_ascii_lowercase(); - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n") - && lower.contains("content-length:") - && !lower.contains("transfer-encoding: chunked") - && !response.contains("proxy_stream_error"), - "glob model discovery must be buffered and Content-Length framed, got: {response}" - ); - } - - /// A failed model-discovery upstream must produce a buffered, Content-Length - /// framed JSON error, never a chunked SSE `proxy_stream_error` frame. - #[tokio::test] - async fn inference_model_discovery_error_served_buffered() { - // A port with no listener so the upstream connection is refused. - let dead_addr = { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr = listener.local_addr().unwrap(); - drop(listener); - addr - }; - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![model_discovery_inference_route(format!( - "http://{dead_addr}" - ))], - vec![], - ); - - let request = "GET /v1/models HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Length: 0\r\n\r\n" - .to_string(); - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - server_task.await.unwrap().unwrap(); - - let lower = response.to_ascii_lowercase(); - assert!( - response.starts_with("HTTP/1.1 5"), - "a refused upstream should yield a 5xx, got: {response}" - ); - assert!( - lower.contains("content-length:") - && !lower.contains("transfer-encoding: chunked") - && !response.contains("proxy_stream_error"), - "buffered model-discovery error must be Content-Length framed JSON, got: {response}" - ); - assert!( - response.contains("error"), - "error response should carry a JSON error body, got: {response}" - ); - } - - async fn read_forwarded_inference_request(stream: &mut S) { - use crate::l7::inference::{ParseResult, try_parse_http_request}; - - let mut buf = Vec::new(); - let mut chunk = [0u8; 4096]; - loop { - let n = stream.read(&mut chunk).await.unwrap(); - assert!(n > 0, "upstream request closed before completion"); - buf.extend_from_slice(&chunk[..n]); - - match try_parse_http_request(&buf) { - ParseResult::Complete(_, _) => return, - ParseResult::Incomplete => continue, - ParseResult::Invalid(reason) => { - panic!("forwarded request should parse cleanly: {reason}"); - } - } - } - } - - /// Like [`read_forwarded_inference_request`] but returns the forwarded - /// request line (method, path) so a test can assert the upstream URL path. - async fn read_forwarded_request_line(stream: &mut S) -> (String, String) { - use crate::l7::inference::{ParseResult, try_parse_http_request}; - - let mut buf = Vec::new(); - let mut chunk = [0u8; 4096]; - loop { - let n = stream.read(&mut chunk).await.unwrap(); - assert!(n > 0, "upstream request closed before completion"); - buf.extend_from_slice(&chunk[..n]); - - match try_parse_http_request(&buf) { - ParseResult::Complete(req, _) => return (req.method, req.path), - ParseResult::Incomplete => continue, - ParseResult::Invalid(reason) => { - panic!("forwarded request should parse cleanly: {reason}"); - } - } - } - } - - async fn run_live_streaming_inference(serve_upstream: F) -> String - where - F: FnOnce(TcpStream) -> Fut + Send + 'static, - Fut: Future + Send + 'static, - { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - read_forwarded_inference_request(&mut upstream).await; - serve_upstream(upstream).await; - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![streaming_inference_route(format!("http://{upstream_addr}"))], - vec![], - ); - - let body = r#"{"model":"ignored","messages":[{"role":"user","content":"hi"}]}"#; - let request = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - Accept: text/event-stream\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - - let outcome = server_task.await.unwrap().unwrap(); - assert!( - matches!(outcome, InferenceOutcome::Routed), - "expected Routed outcome, got: {outcome:?}" - ); - upstream_task.await.unwrap(); - - String::from_utf8(response).unwrap() - } - - fn assert_streaming_sse_error(response: &str, message: &str) { - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n"), - "expected successful streaming response, got: {response}" - ); - assert!( - response - .to_ascii_lowercase() - .contains("transfer-encoding: chunked"), - "expected chunked streaming response, got: {response}" - ); - assert!( - response.contains("\"type\":\"proxy_stream_error\""), - "expected proxy_stream_error SSE event, got: {response}" - ); - assert!( - response.contains(&format!("\"message\":\"{message}\"")), - "expected SSE message {message:?}, got: {response}" - ); - assert!( - response.ends_with("0\r\n\r\n"), - "streaming response must end with chunked terminator, got: {response}" - ); - } - - #[tokio::test] - async fn inference_stream_byte_limit_injects_sse_error() { - let response = run_live_streaming_inference(|mut upstream| async move { - use crate::l7::inference::{format_chunk, format_chunk_terminator}; - - upstream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: text/event-stream\r\n\ - Transfer-Encoding: chunked\r\n\r\n", - ) - .await - .unwrap(); - let body = vec![b'a'; MAX_STREAMING_BODY + 1]; - let _ = upstream.write_all(&format_chunk(&body)).await; - let _ = upstream.write_all(format_chunk_terminator()).await; - }) - .await; - - assert_streaming_sse_error( - &response, - "response truncated: exceeded maximum streaming body size", - ); - } - - #[tokio::test] - async fn inference_stream_upstream_read_error_injects_sse_error() { - let response = run_live_streaming_inference(|mut upstream| async move { - upstream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: text/event-stream\r\n\ - Content-Length: 64\r\n\r\n\ - partial", - ) - .await - .unwrap(); - }) - .await; - - assert!( - response.contains("partial"), - "expected initial upstream bytes before truncation, got: {response}" - ); - assert_streaming_sse_error(&response, "response truncated: upstream read error"); - } - - #[tokio::test] - async fn inference_stream_idle_timeout_injects_sse_error() { - let response = run_live_streaming_inference(|mut upstream| async move { - upstream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: text/event-stream\r\n\ - Transfer-Encoding: chunked\r\n\r\n", - ) - .await - .unwrap(); - tokio::time::sleep(CHUNK_IDLE_TIMEOUT + std::time::Duration::from_millis(50)).await; - }) - .await; - - assert_streaming_sse_error(&response, "response truncated: chunk idle timeout exceeded"); - } - - // -- router_error_to_http -- - - #[test] - fn router_error_route_not_found_maps_to_400() { - let err = openshell_router::RouterError::RouteNotFound("local".into()); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 400); - assert_eq!(msg, "no inference route configured"); - // SEC-008: must NOT leak the route hint to sandboxed code - assert!(!msg.contains("local")); - } - - #[test] - fn router_error_no_compatible_route_maps_to_400() { - let err = openshell_router::RouterError::NoCompatibleRoute("anthropic_messages".into()); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 400); - assert_eq!(msg, "no compatible inference route available"); - // SEC-008: must NOT leak the protocol name to sandboxed code - assert!(!msg.contains("anthropic_messages")); - } - - #[test] - fn router_error_unauthorized_maps_to_401() { - let err = - openshell_router::RouterError::Unauthorized("bad token from 10.0.0.5:8080".into()); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 401); - assert_eq!(msg, "unauthorized"); - // SEC-008: must NOT leak upstream details to sandboxed code - assert!(!msg.contains("10.0.0.5")); - } - - #[test] - fn router_error_upstream_unavailable_maps_to_503() { - let err = openshell_router::RouterError::UpstreamUnavailable( - "connection refused to 10.0.0.5:8080".into(), - ); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 503); - assert_eq!(msg, "inference service unavailable"); - // SEC-008: must NOT leak upstream address to sandboxed code - assert!(!msg.contains("10.0.0.5")); - } - - #[test] - fn router_error_upstream_protocol_maps_to_502() { - let err = openshell_router::RouterError::UpstreamProtocol( - "TLS handshake failed for nim.internal.svc:443".into(), - ); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 502); - assert_eq!(msg, "inference service error"); - // SEC-008: must NOT leak internal hostnames to sandboxed code - assert!(!msg.contains("nim.internal")); - } - - #[test] - fn router_error_internal_maps_to_502() { - let err = openshell_router::RouterError::Internal( - "failed to read /etc/openshell/routes.json".into(), - ); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 502); - assert_eq!(msg, "inference service error"); - // SEC-008: must NOT leak file paths to sandboxed code - assert!(!msg.contains("/etc/openshell")); - } - - #[test] - fn sanitize_response_headers_strips_hop_by_hop() { - let headers = vec![ - ("transfer-encoding".to_string(), "chunked".to_string()), - ("content-length".to_string(), "128".to_string()), - ("connection".to_string(), "keep-alive".to_string()), - ("content-type".to_string(), "text/event-stream".to_string()), - ("cache-control".to_string(), "no-cache".to_string()), - ]; - - let kept = sanitize_inference_response_headers(headers); - - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("transfer-encoding")), - "transfer-encoding should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("content-length")), - "content-length should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("connection")), - "connection should be stripped" - ); - assert!( - kept.iter() - .any(|(k, _)| k.eq_ignore_ascii_case("content-type")), - "content-type should be preserved" - ); - assert!( - kept.iter() - .any(|(k, _)| k.eq_ignore_ascii_case("cache-control")), - "cache-control should be preserved" - ); - } - - // -- is_always_blocked_ip -- + // -- is_always_blocked_ip -- #[test] fn test_always_blocked_loopback_v4() { @@ -11782,87 +10322,6 @@ network_policies: assert!(result.is_empty()); } - /// Regression test: exercises the actual keep-alive interception loop to - /// verify that a non-inference request is denied even after a previous - /// inference request was successfully routed on the same connection. - /// - /// Before the fix, `handle_inference_interception` used - /// `else if !routed_any` which silently dropped denials once `routed_any` - /// was true, allowing non-inference HTTP requests to piggyback on a - /// keep-alive connection that had previously handled inference traffic. - /// Regression test: exercises the actual keep-alive interception loop to - /// verify that a non-inference request is denied even after a previous - /// inference request was successfully routed on the same connection. - /// - /// The server runs in a spawned task with empty routes (the inference - /// request gets a 503 "not configured" but is still recognized as - /// inference and returns Ok(true)). The client sends the inference - /// request, reads the 503 response, then sends a non-inference request - /// on the same connection. The server must return Denied. - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn test_keepalive_denies_non_inference_after_routed() { - use openshell_router::Router; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - - let router = Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - // Empty routes: inference request gets 503 but returns Ok(true). - let ctx = InferenceContext::new(patterns, router, vec![], vec![]); - - let body = r#"{"model":"test","messages":[{"role":"user","content":"hi"}]}"#; - let inference_req = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - let non_inference_req = "GET /admin/config HTTP/1.1\r\nHost: inference.local\r\n\r\n"; - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - - // Spawn the server task so it runs concurrently. - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - // Client: send inference request, read response, send non-inference. - client_write - .write_all(inference_req.as_bytes()) - .await - .unwrap(); - - // Read the 503 response so the server loops back to read. - let mut buf = vec![0u8; 4096]; - let _ = client_read.read(&mut buf).await.unwrap(); - - // Send non-inference request on the same keep-alive connection. - client_write - .write_all(non_inference_req.as_bytes()) - .await - .unwrap(); - drop(client_write); - - // Drain remaining response bytes. - tokio::spawn(async move { - let mut buf = vec![0u8; 4096]; - loop { - match client_read.read(&mut buf).await { - Ok(0) | Err(_) => break, - Ok(_) => continue, - } - } - }); - - let outcome = server_task.await.unwrap().unwrap(); - - assert!( - matches!(outcome, InferenceOutcome::Denied { .. }), - "expected Denied after non-inference request on keep-alive, got: {outcome:?}" - ); - } - // -- build_json_error_response -- #[test] @@ -12095,7 +10554,6 @@ network_policies: cache, entrypoint_pid, None, // tls_state — ephemeral CA unavailable - None, // inference_ctx None, // policy_local_ctx AgentProposals::default(), // agent_proposals Arc::new(None), // trusted_host_gateway @@ -12163,7 +10621,6 @@ network_policies: Arc::new(AtomicU32::new(std::process::id())), None, None, - None, AgentProposals::default(), Arc::new(None), Arc::new(None), diff --git a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs index 186d156086..30ee3d701a 100644 --- a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs +++ b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs @@ -545,7 +545,6 @@ network_policies: Arc::new(AtomicU32::new(0)), None, None, - None, AgentProposals::default(), Arc::new(None), Arc::new(None), diff --git a/crates/openshell-supervisor-network/src/run.rs b/crates/openshell-supervisor-network/src/run.rs index 2a71702b4b..af9ffb8407 100644 --- a/crates/openshell-supervisor-network/src/run.rs +++ b/crates/openshell-supervisor-network/src/run.rs @@ -4,7 +4,7 @@ //! Networking stack startup for the sandbox. //! //! Builds the network namespace (Linux), the CONNECT proxy with TLS L7 -//! interception, the inference context, and wires the proxy to the +//! interception and wires the proxy to the //! caller-supplied denial-event channel. Returns a [`Networking`] handle //! whose RAII fields keep the proxy task alive for the lifetime of the //! sandbox supervisor. @@ -176,8 +176,8 @@ pub struct Networking { /// # Errors /// /// Returns an error if proxy mode is requested but the proxy configuration, -/// OPA engine, or identity cache is missing, if inference route resolution -/// fails, or if the proxy server fails to start. +/// OPA engine, or identity cache is missing, or if the proxy server fails to +/// start. #[allow(clippy::too_many_arguments)] pub async fn run_networking( policy: &SandboxPolicy, @@ -190,7 +190,6 @@ pub async fn run_networking( sandbox_id: Option<&str>, sandbox_name: Option<&str>, openshell_endpoint: Option<&str>, - #[allow(unused_variables)] inference_routes: Option<&str>, denial_tx: Option>, activity_tx: Option, agent_proposals: AgentProposals, @@ -425,14 +424,6 @@ pub async fn run_networking( SocketAddr::new(ip, port) }); - // Build inference context for local routing of intercepted inference calls. - let inference_ctx = crate::inference_routes::build_inference_context( - sandbox_id, - openshell_endpoint, - inference_routes, - ) - .await?; - let proxy_handle = ProxyHandle::start_with_bind_addr( proxy_policy, bind_addr, @@ -440,7 +431,6 @@ pub async fn run_networking( cache, entrypoint_pid.clone(), tls_state, - inference_ctx, Some(provider_credentials.clone()), Some(policy_local_ctx.clone()), denial_tx.clone(), diff --git a/crates/openshell-supervisor-network/tests/system_inference.rs b/crates/openshell-supervisor-network/tests/system_inference.rs deleted file mode 100644 index 3c8e6ee8fc..0000000000 --- a/crates/openshell-supervisor-network/tests/system_inference.rs +++ /dev/null @@ -1,167 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Integration test for the in-process system inference API. -//! -//! Uses the router's built-in `mock://` route support to verify the full -//! in-process path: route selection → `proxy_with_candidates()` → -//! mock backend → response. - -use openshell_router::Router; -use openshell_router::config::{AuthHeader, ResolvedRoute}; -use openshell_supervisor_network::proxy::InferenceContext; - -fn make_system_route() -> ResolvedRoute { - ResolvedRoute { - name: "sandbox-system".to_string(), - endpoint: "mock://system-test".to_string(), - model: "system/policy-analyzer".to_string(), - api_key: "system-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } -} - -fn make_user_route() -> ResolvedRoute { - ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "mock://user-test".to_string(), - model: "user/gpt-4o".to_string(), - api_key: "user-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } -} - -#[tokio::test] -async fn system_inference_routes_to_mock_backend() { - let router = Router::new().unwrap(); - let patterns = openshell_supervisor_network::l7::inference::default_patterns(); - - let ctx = InferenceContext::new( - patterns, - router, - vec![make_user_route()], - vec![make_system_route()], - ); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "anything", - "messages": [{"role": "user", "content": "analyze this policy"}] - })) - .unwrap(); - - let response = ctx - .system_inference( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - ) - .await - .expect("system_inference should succeed"); - - assert_eq!(response.status, 200); - - // Verify the mock response came back with the system model - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["model"], "system/policy-analyzer"); - - // Verify the mock header is present - assert!( - response - .headers - .iter() - .any(|(k, v)| k == "x-openshell-mock" && v == "true") - ); -} - -#[tokio::test] -async fn system_inference_uses_system_routes_not_user_routes() { - let router = Router::new().unwrap(); - let patterns = openshell_supervisor_network::l7::inference::default_patterns(); - - // Only user routes configured — no system routes - let ctx = InferenceContext::new(patterns, router, vec![make_user_route()], vec![]); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "gpt-4o", - "messages": [{"role": "user", "content": "hello"}] - })) - .unwrap(); - - let result = ctx - .system_inference( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![], - bytes::Bytes::from(body), - ) - .await; - - // Should fail because the system route cache is empty — user routes - // are not accessible through system_inference(). - assert!( - result.is_err(), - "system_inference should fail when no system routes are configured" - ); -} - -#[tokio::test] -async fn system_inference_with_anthropic_protocol() { - let router = Router::new().unwrap(); - let patterns = openshell_supervisor_network::l7::inference::default_patterns(); - - let system_route = ResolvedRoute { - name: "sandbox-system".to_string(), - endpoint: "mock://anthropic-system".to_string(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "ant-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let ctx = InferenceContext::new(patterns, router, vec![], vec![system_route]); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-20250514", - "max_tokens": 1, - "messages": [{"role": "user", "content": "analyze policy"}] - })) - .unwrap(); - - let response = ctx - .system_inference( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - ) - .await - .expect("anthropic system_inference should succeed"); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["type"], "message"); -} diff --git a/deploy/docker/Dockerfile.gateway-macos b/deploy/docker/Dockerfile.gateway-macos index c7d526a039..4ecbac0397 100644 --- a/deploy/docker/Dockerfile.gateway-macos +++ b/deploy/docker/Dockerfile.gateway-macos @@ -57,19 +57,17 @@ COPY crates/openshell-gateway/Cargo.toml crates/openshell-gateway/Cargo.toml COPY crates/openshell-driver-kubernetes/Cargo.toml crates/openshell-driver-kubernetes/Cargo.toml COPY crates/openshell-policy/Cargo.toml crates/openshell-policy/Cargo.toml COPY crates/openshell-prover/Cargo.toml crates/openshell-prover/Cargo.toml -COPY crates/openshell-router/Cargo.toml crates/openshell-router/Cargo.toml COPY crates/openshell-server/Cargo.toml crates/openshell-server/Cargo.toml COPY crates/openshell-core/build.rs crates/openshell-core/build.rs COPY proto/ proto/ -RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-gateway", "crates/openshell-server", "crates/openshell-core", "crates/openshell-driver-kubernetes", "crates/openshell-policy", "crates/openshell-prover", "crates/openshell-router"]|' Cargo.toml +RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-gateway", "crates/openshell-server", "crates/openshell-core", "crates/openshell-driver-kubernetes", "crates/openshell-policy", "crates/openshell-prover"]|' Cargo.toml RUN mkdir -p crates/openshell-core/src \ crates/openshell-gateway/src \ crates/openshell-driver-kubernetes/src \ crates/openshell-policy/src \ crates/openshell-prover/src \ - crates/openshell-router/src \ crates/openshell-server/src && \ touch crates/openshell-core/src/lib.rs && \ touch crates/openshell-gateway/src/lib.rs && \ @@ -78,7 +76,6 @@ RUN mkdir -p crates/openshell-core/src \ printf 'fn main() {}\n' > crates/openshell-driver-kubernetes/src/main.rs && \ touch crates/openshell-policy/src/lib.rs && \ touch crates/openshell-prover/src/lib.rs && \ - touch crates/openshell-router/src/lib.rs && \ touch crates/openshell-server/src/lib.rs RUN --mount=type=cache,id=cargo-registry-gateway-macos,sharing=locked,target=/root/.cargo/registry \ @@ -96,7 +93,6 @@ RUN touch crates/openshell-core/src/lib.rs \ crates/openshell-driver-kubernetes/src/main.rs \ crates/openshell-policy/src/lib.rs \ crates/openshell-prover/src/lib.rs \ - crates/openshell-router/src/lib.rs \ crates/openshell-server/src/lib.rs \ crates/openshell-core/build.rs \ proto/*.proto diff --git a/deploy/man/openshell-gateway.8.md b/deploy/man/openshell-gateway.8.md index 2d584c4ba1..ee26a4477f 100644 --- a/deploy/man/openshell-gateway.8.md +++ b/deploy/man/openshell-gateway.8.md @@ -18,7 +18,7 @@ openshell-gateway - OpenShell gateway server daemon **openshell-gateway** is the control-plane server for OpenShell. It manages sandbox lifecycle, stores provider credentials, delivers -network and filesystem policies to sandboxes, routes inference +network and filesystem policies to sandboxes, manages provider access requests, and provides the SSH tunnel endpoint for CLI-to-sandbox connections. diff --git a/deploy/man/openshell.1.md b/deploy/man/openshell.1.md index 849d299cbb..7dabf558ab 100644 --- a/deploy/man/openshell.1.md +++ b/deploy/man/openshell.1.md @@ -19,7 +19,7 @@ openshell - CLI for managing OpenShell sandboxes, gateway registrations, and pro **openshell** is the command-line interface for OpenShell, a platform providing safe, sandboxed runtimes for autonomous AI agents. It manages gateway registrations, sandbox lifecycle, credential providers, -network policies, and inference routing. +network policies, and provider attachments. The CLI communicates with a gateway server over gRPC. The gateway can run as a package-managed systemd user service, a Helm deployment, a @@ -114,16 +114,12 @@ development task, or behind a cloud reverse proxy. **provider delete** *NAME* : Delete a provider. -## Inference Routing +**sandbox provider attach** *SANDBOX* *PROVIDER* +: Attach a provider to a running sandbox. New processes receive its + endpoint-bound credential placeholders and provider-derived policy. -**inference set** **--provider** *NAME* **--model** *MODEL* -: Configure inference routing. - -**inference get** -: Show current inference configuration. - -**inference update** \[**--model** *MODEL*\] -: Update inference configuration. +**sandbox provider detach** *SANDBOX* *PROVIDER* +: Revoke a provider attachment from a running sandbox. ## Other diff --git a/deploy/rpm/QUICKSTART.md b/deploy/rpm/QUICKSTART.md index 442458d09d..f37f611351 100644 --- a/deploy/rpm/QUICKSTART.md +++ b/deploy/rpm/QUICKSTART.md @@ -144,13 +144,15 @@ openshell provider create --name openai --type openai \ openshell provider list ``` -## Configure inference routing (optional) +## Configure provider-backed inference (optional) -To route inference requests through a specific provider and model: +To grant a sandbox access to a model provider: ```shell -openshell inference set --provider openai --model gpt-4 -openshell inference get +openshell sandbox provider attach openai + +# Configure the workload to call the provider's native endpoint and select +# the model in the client. ``` ## Next steps diff --git a/deploy/rpm/TROUBLESHOOTING.md b/deploy/rpm/TROUBLESHOOTING.md index 103ce3bf9d..fe6da795ab 100644 --- a/deploy/rpm/TROUBLESHOOTING.md +++ b/deploy/rpm/TROUBLESHOOTING.md @@ -13,7 +13,7 @@ and what to use instead. ### Commands that work normally -All sandbox, provider, policy, inference, and settings commands +All sandbox, provider, policy, and settings commands communicate with the gateway over gRPC and work identically regardless of deployment mode: @@ -23,7 +23,8 @@ openshell sandbox create|list|get|delete|connect|exec openshell logs openshell provider create|list|get|update|delete openshell policy get|set|update|list|prove -openshell inference set|get|update +openshell provider list +openshell sandbox provider list openshell settings get|set openshell forward start|stop|list openshell term diff --git a/docs/about/how-it-works.mdx b/docs/about/how-it-works.mdx index 5223072a28..9878fceb68 100644 --- a/docs/about/how-it-works.mdx +++ b/docs/about/how-it-works.mdx @@ -4,14 +4,14 @@ title: "How OpenShell Works" sidebar-title: "How It Works" description: "Understand the OpenShell architecture, runtime boundaries, gateways, sandboxes, and ecosystem integration points." -keywords: "Generative AI, Cybersecurity, AI Agents, Architecture, Gateway, Sandbox, Inference Routing" +keywords: "Generative AI, Cybersecurity, AI Agents, Architecture, Gateway, Sandbox, Providers" position: 2 --- OpenShell is built around three stable runtime components: the **CLI**, the **Gateway**, and the **Supervisor**. The CLI, SDK, and TUI provide user-facing access. The gateway is the -control plane: it owns API access, state, policy and settings delivery, provider and inference configuration, and relay coordination. The supervisor runs inside every sandbox workload and is the local security boundary. It launches the agent as a restricted child process and enforces policy where process identity, filesystem access, network egress, and +control plane: it owns API access, state, policy and settings delivery, provider configuration, and relay coordination. The supervisor runs inside every sandbox workload and is the local security boundary. It launches the agent as a restricted child process and enforces policy where process identity, filesystem access, network egress, and runtime credentials are visible. Infrastructure-specific work sits behind integration boundaries. Compute, @@ -46,7 +46,6 @@ flowchart TB AGENT["Restricted agent process"] PROXY["Policy proxy"] POLICY["OPA policy engine"] - ROUTER["Inference router"] end CLI -->|"gRPC / HTTP"| GW @@ -64,8 +63,7 @@ flowchart TB AGENT -->|"ordinary egress"| PROXY PROXY -->|"evaluate"| POLICY PROXY -->|"allowed traffic"| EXT["External services"] - PROXY -->|"inference.local"| ROUTER - ROUTER -->|"managed inference"| MODEL["Inference backends"] + PROXY -->|"profile-authorized traffic"| MODEL["Model providers"] ``` ## Deployment Models @@ -92,11 +90,11 @@ device plugins without changing the gateway and sandbox contract. | [Gateways](/sandboxes/manage-gateways) | Authenticated control plane that owns API access, durable state, sandbox lifecycle, settings delivery, authorization, and relay coordination. | | [Providers](/sandboxes/manage-providers) | Credential and provider records that map logical agent needs to platform or user-managed secrets without exposing raw credentials to the agent process. | | [Policies](/sandboxes/policies) | Declarative controls for filesystem access, process identity, network egress, L7 rules, credential injection, and runtime policy updates. | -| [Inference Routing](/sandboxes/inference-routing) | Managed `https://inference.local` path that routes model traffic to configured backends while keeping provider credentials outside the sandbox. | +| [Provider-backed Inference](/sandboxes/inference-routing) | Per-sandbox provider attachment, native model endpoints, and endpoint-bound credential injection. | ## Gateways and Sandboxes -The gateway and sandbox split control-plane authority from runtime enforcement. The gateway owns durable platform state: sandboxes, policy revisions, runtime settings, provider records, inference configuration, session records, and authorization decisions. A sandbox owns the local execution boundary: process identity, filesystem access, network egress, credential injection, local logs, and the agent child process. +The gateway and sandbox split control-plane authority from runtime enforcement. The gateway owns durable platform state: sandboxes, policy revisions, runtime settings, provider records, session records, and authorization decisions. A sandbox owns the local execution boundary: process identity, filesystem access, network egress, credential injection, local logs, and the agent child process. The relationship is supervisor initiated. Each sandbox supervisor connects outbound to a known gateway endpoint, authenticates as a sandbox workload, and keeps a live session open for control traffic and relays. This avoids requiring every compute driver to solve gateway-to-sandbox reachability through pod IPs, bridge networks, port mappings, NAT traversal, or custom tunnels. @@ -114,18 +112,18 @@ then launches the agent under the active policy. | Filesystem | Applies filesystem policy before the agent starts so undeclared paths are inaccessible and declared paths are read-only or read-write as configured. | | Network | Routes ordinary egress through the policy proxy so destination, port, binary identity, and L7 request rules can be evaluated before traffic leaves the sandbox. | | Credentials | Receives credential material from the gateway and injects it only through configured policy paths or request-time proxy rules. | -| Inference | Intercepts `https://inference.local` and forwards model traffic through the configured inference route instead of exposing provider credentials to the agent. | +| Provider access | Enforces profile-derived policy and substitutes provider credential placeholders only at profile-authorized endpoints. | | Observability | Emits local security and lifecycle logs, pushes sandbox logs to the gateway, and keeps relay endpoints available for connect, exec, and file transfer operations. | Static controls such as filesystem and process isolation are established at sandbox start and require sandbox recreation to change. Dynamic controls such as -network policy, credential delivery, and inference routing can refresh over the +network policy and credential delivery can refresh over the live gateway-supervisor session. ## Ecosystem Integration OpenShell integrates with infrastructure ecosystems instead of replacing them. Runtimes, schedulers, secret stores, identity providers, workload identity systems, image pipelines, storage, and GPU or device exposure remain owned by the platforms that provide them. -The gateway owns OpenShell control-plane semantics: sandbox state, lifecycle ordering, policy and settings resolution, credential mapping, authorization, inference configuration, and relay coordination. Drivers translate those semantics into platform-native operations. +The gateway owns OpenShell control-plane semantics: sandbox state, lifecycle ordering, policy and settings resolution, credential mapping, authorization, and relay coordination. Drivers translate those semantics into platform-native operations. -The supervisor owns OpenShell sandbox semantics. Filesystem policy, process privilege reduction, network proxying, inference interception, credential injection, security logging, and gateway relay behavior stay consistent across Docker, Podman, Kubernetes, VM-backed sandboxes, and future integrations. +The supervisor owns OpenShell sandbox semantics. Filesystem policy, process privilege reduction, network proxying, provider credential injection, security logging, and gateway relay behavior stay consistent across Docker, Podman, Kubernetes, VM-backed sandboxes, and future integrations. diff --git a/docs/about/overview.mdx b/docs/about/overview.mdx index 6ef41b34a1..90b565ae38 100644 --- a/docs/about/overview.mdx +++ b/docs/about/overview.mdx @@ -4,7 +4,7 @@ title: "Overview of NVIDIA OpenShell" sidebar-title: "Overview" description: "OpenShell is the safe, private runtime for autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Inference Routing" +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Provider Credentials" position: 1 --- @@ -22,7 +22,7 @@ The table below summarizes common failure modes and how OpenShell mitigates them |---|---|---| | Data exfiltration | Agent uploads source code or internal files to unauthorized endpoints. | Network policies allow only approved destinations; other outbound traffic is denied. | | Credential theft | Agent reads local secrets such as SSH keys or cloud credentials. | Filesystem restrictions (Landlock) confine access to declared paths only. | -| Unauthorized API usage | Agent sends prompts or data to unapproved model providers. | Privacy routing and network policies control where inference traffic can go. | +| Unauthorized API usage | Agent sends prompts or data to unapproved model providers. | Provider profiles and network policies restrict model traffic to approved endpoints and binaries. | | Privilege escalation | Agent attempts `sudo`, setuid paths, or dangerous syscall behavior. | Unprivileged process identity and seccomp restrictions block escalation paths. | ## Protection Layers at a Glance @@ -34,7 +34,7 @@ OpenShell applies defense in depth across the following policy domains. | Filesystem | Prevents reads/writes outside allowed paths. | Locked at sandbox creation. | | Network | Blocks unauthorized outbound connections. | Hot-reloadable at runtime. | | Process | Blocks privilege escalation and dangerous syscalls. | Locked at sandbox creation. | -| Inference | Reroutes model API calls to controlled backends. | Hot-reloadable at runtime. | +| Provider credentials | Resolves opaque credential placeholders only at profile-authorized endpoints. | Attachments, rotation, and revocation update at runtime; new environment variables require a new process. | For details, refer to [Customize Sandbox Policies](/sandboxes/policies) and [Default Policy](/reference/default-policy). @@ -45,7 +45,7 @@ OpenShell supports a range of agent deployment patterns. | Use Case | Description | |-----------------------------|----------------------------------------------------------------------------------------------------------| | Secure coding agents | Run Claude Code, OpenCode, Codex, or GitHub Copilot CLI with constrained file and network access. | -| Private enterprise development | Route inference to self-hosted or private backends while keeping sensitive context under your control. | +| Private enterprise development | Grant selected sandboxes access to self-hosted or private model endpoints while keeping sensitive context under your control. | | Compliance and audit | Treat policy YAML as version-controlled security controls that can be reviewed and audited. | | Reusable environments | Use community sandbox images or bring your own containerized runtime. | diff --git a/docs/about/supported-agents.mdx b/docs/about/supported-agents.mdx index 9d38541dd4..13704aa94b 100644 --- a/docs/about/supported-agents.mdx +++ b/docs/about/supported-agents.mdx @@ -10,12 +10,12 @@ The following table summarizes the agents that run in OpenShell sandboxes. Most | Agent | Source | Default Policy | Notes | |---|---|---|---| -| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Full coverage | Works out of the box. Requires `ANTHROPIC_API_KEY` for direct Anthropic access, or use `inference.local` with a configured provider (e.g. Vertex AI). | -| [OpenCode](https://opencode.ai/) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Partial coverage | Pre-installed. Use `ANTHROPIC_BASE_URL="https://inference.local/v1"` with a configured provider. Add `opencode.ai` endpoint and OpenCode binary paths to the policy for full functionality. | +| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Full coverage | Works out of the box with an attached `claude-code` provider. For another model platform, attach an endpoint-bearing profile and configure Claude Code for that platform's native API. | +| [OpenCode](https://opencode.ai/) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Partial coverage | Pre-installed. Attach the selected model provider, configure its native base URL and model, and ensure the imported profile names OpenCode's service endpoint and binary paths. | | [Codex](https://developers.openai.com/codex) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | No coverage | Pre-installed. Requires a custom policy with OpenAI endpoints and Codex binary paths. Requires `OPENAI_API_KEY`. | | [GitHub Copilot CLI](https://docs.github.com/en/copilot/github-copilot-in-the-cli) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Full coverage | Pre-installed. Works out of the box. Requires `GITHUB_TOKEN` or `COPILOT_GITHUB_TOKEN`. | -| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run OpenClaw more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | -| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run Hermes Agent more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | +| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run OpenClaw more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | +| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run Hermes Agent more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | | [Ollama](https://ollama.com/) | [`ollama`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/ollama) | Bundled | Run cloud and local models. Includes Claude Code, Codex, and OpenCode. Launch with `openshell sandbox create --from ollama`. | | [Pi](https://pi.dev/) | [`pi`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/pi) | Bundled | Comes with Pi pre-installed. Launch with `openshell sandbox create --from pi`. | diff --git a/docs/get-started/tutorials/docker-compose.mdx b/docs/get-started/tutorials/docker-compose.mdx index 677f34595b..fd8c13f059 100644 --- a/docs/get-started/tutorials/docker-compose.mdx +++ b/docs/get-started/tutorials/docker-compose.mdx @@ -137,9 +137,9 @@ docker pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest -OpenClaw runs inside OpenShell through [NemoClaw](https://github.com/NVIDIA/NemoClaw), which manages the sandbox image, inference routing, and security policies. +OpenClaw runs inside OpenShell through [NemoClaw](https://github.com/NVIDIA/NemoClaw), which manages the sandbox image, model-provider setup, and security policies. -Follow the [NemoClaw Quickstart](https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart/) to set up an OpenClaw sandbox with managed inference. +Follow the [NemoClaw Quickstart](https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart/) to set up an OpenClaw sandbox. diff --git a/docs/get-started/tutorials/index.mdx b/docs/get-started/tutorials/index.mdx index bba5996277..2050480b90 100644 --- a/docs/get-started/tutorials/index.mdx +++ b/docs/get-started/tutorials/index.mdx @@ -29,12 +29,12 @@ Configure a Microsoft Graph provider profile with gateway-managed OAuth2 refresh -Route inference through Ollama using cloud-hosted or local models, and verify it from a sandbox. +Run Ollama models in a community sandbox or attach a profile for a host-level service. -Route inference to a local LM Studio server using the OpenAI-compatible or Anthropic-compatible APIs. +Attach an endpoint-bearing profile and call a local LM Studio server directly. diff --git a/docs/get-started/tutorials/inference-ollama.mdx b/docs/get-started/tutorials/inference-ollama.mdx index 4f46b847e4..c23e6eaf6c 100644 --- a/docs/get-started/tutorials/inference-ollama.mdx +++ b/docs/get-started/tutorials/inference-ollama.mdx @@ -4,54 +4,35 @@ title: "Run Local Inference with Ollama" sidebar-title: "Inference with Ollama" slug: "get-started/tutorials/inference-ollama" -description: "Run local and cloud models inside an OpenShell sandbox using the Ollama community sandbox, or route sandbox requests to a host-level Ollama server." -keywords: "Generative AI, Cybersecurity, Tutorial, Inference Routing, Ollama, Local Inference, Sandbox" +description: "Run Ollama inside a community sandbox or attach a profile for a host-level Ollama server." +keywords: "Generative AI, Cybersecurity, Tutorial, Ollama, Local Inference, Sandbox, Provider Profile" --- -This tutorial covers two ways of running Ollama with OpenShell: +This tutorial covers two Ollama deployments: -1. Ollama sandbox. This is the recommended way to run Ollama. A self-contained sandbox with Ollama, Claude Code, and Codex pre-installed. One command starts it. -2. Host-level Ollama. This is an alternative way to run Ollama. Run Ollama on the gateway host and route sandbox inference to it. Use this option when you want a single Ollama instance shared across multiple sandboxes. - -After completing this tutorial, you know how to: - -- Launch the Ollama community sandbox for a batteries-included experience. -- Use `ollama launch` to start coding agents inside a sandbox. -- Expose a host-level Ollama server to sandboxes through `inference.local`. +1. The self-contained Ollama community sandbox. This is the recommended path. +2. A host-level Ollama service shared by explicitly authorized sandboxes. ## Prerequisites -- A working OpenShell installation. Complete the [Quickstart](/get-started/quickstart) before proceeding. - -## Option A: Ollama Community Sandbox (Recommended) - -The Ollama community sandbox bundles Ollama, Claude Code, OpenCode, and Codex into a single image. Ollama starts automatically when the sandbox launches. +Complete the [Quickstart](/get-started/quickstart) before proceeding. - +## Option A: Ollama Community Sandbox -### Create the Sandbox +The community image bundles Ollama and supported coding agents: ```shell openshell sandbox create --from ollama ``` -This pulls the community sandbox image, applies the bundled policy, and drops you into a shell with Ollama running. - -### Chat with a Model - -Chat with a local model +Inside the sandbox, run a local or cloud model: ```shell ollama run qwen3.5 -``` - -Or a cloud model - -```shell ollama run kimi-k2.5:cloud ``` -Or use `ollama launch` to start a coding agent with Ollama as the model backend: +Or launch a coding agent against Ollama: ```shell ollama launch claude @@ -59,153 +40,107 @@ ollama launch codex ollama launch opencode ``` -For CI/CD and automated workflows, `ollama launch` supports a headless mode: +For an automated workflow: ```shell ollama launch claude --yes --model qwen3.5 ``` - - -### Model Recommendations +Useful starter models include `qwen3.5:0.8b` for smoke tests and `qwen3.5` +for coding and tool use. Check the +[Ollama model library](https://ollama.com/library) for current model details. -| Use case | Model | Notes | -|---|---|---| -| Smoke test | `qwen3.5:0.8b` | Fast, lightweight, good for verifying setup | -| Coding and reasoning | `qwen3.5` | Strong tool calling support for agentic workflows | -| Complex tasks | `nemotron-3-super` | 122B parameter model, needs 48GB+ VRAM | -| No local GPU | `qwen3.5:cloud` | Runs on Ollama's cloud infrastructure, no `ollama pull` required | - - -Cloud models use the `:cloud` tag suffix and do not require local hardware. - -```shell -openshell sandbox create --from ollama -``` - - - -### Tool Calling - -Agentic workflows (Claude Code, Codex, OpenCode) rely on tool calling. The following models have reliable tool calling support: Qwen 3.5, Nemotron-3-Super, GLM-5, and Kimi-K2.5. Check the [Ollama model library](https://ollama.com/library) for the latest models. - -### Updating Ollama - -To update Ollama inside a running sandbox: - -```shell -update-ollama -``` - -Or auto-update on every sandbox start: +Update the bundled Ollama installation with `update-ollama`, or request an +update at sandbox start: ```shell openshell sandbox create --from ollama -e OLLAMA_UPDATE=1 ``` -## Option B: Host-Level Ollama - -Use this approach when you want a single Ollama instance on the gateway host, shared across multiple sandboxes through `inference.local`. - - -This approach uses Ollama because it is easy to install and run locally, but you can substitute other inference engines such as vLLM, SGLang, TRT-LLM, and NVIDIA NIM by changing the startup command, base URL, and model name. +## Option B: Host-level Ollama - +Use this path when Ollama runs on the same machine as the OpenShell gateway and +selected sandboxes should share it. - - -### Install and Start Ollama - -Install [Ollama](https://ollama.com/) on the gateway host: +### Start Ollama on a Reachable Address ```shell curl -fsSL https://ollama.com/install.sh | sh -``` - -Start Ollama on all interfaces so it is reachable from sandboxes: - -```shell OLLAMA_HOST=0.0.0.0:11434 ollama serve ``` - -If you see `Error: listen tcp 0.0.0.0:11434: bind: address already in use`, Ollama is already running as a system service. Stop it first: - -```shell -systemctl stop ollama -OLLAMA_HOST=0.0.0.0:11434 ollama serve -``` - - - -### Pull a Model - -In a second terminal, pull a model: +In another terminal, pull a model: ```shell -ollama run qwen3.5:0.8b +ollama pull qwen3.5:0.8b ``` -Type `/bye` to exit the interactive session. The model stays loaded. +### Import an Endpoint-bearing Profile -### Create a Provider +Save this as `ollama-openai.yaml`: -Create an OpenAI-compatible provider pointing at the host Ollama: - -```shell -openshell provider create \ - --name ollama \ - --type openai \ - --credential OPENAI_API_KEY=empty \ - --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 +```yaml +id: ollama-openai +display_name: Ollama +description: Host-level Ollama OpenAI-compatible API +category: inference +inference_capable: true +credentials: [] +endpoints: + - host: host.openshell.internal + port: 11434 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/curl + - /usr/local/bin/curl + - /usr/bin/python3 + - /usr/local/bin/python + - /sandbox/.uv/python/** + - /sandbox/.venv/** ``` -OpenShell injects `host.openshell.internal` so sandboxes and the gateway can reach the host machine. You can also use the host's LAN IP. - -### Set Inference Routing - ```shell -openshell inference set --provider ollama --model qwen3.5:0.8b +openshell provider profile lint -f ollama-openai.yaml +openshell provider profile import -f ollama-openai.yaml +openshell provider create --name ollama --type ollama-openai ``` -Confirm: - -```shell -openshell inference get -``` +The provider has no secret because this Ollama server does not authenticate. +Its attachment still carries the endpoint and binary policy. -### Verify from a Sandbox +### Attach and Verify ```shell -openshell sandbox create -- \ - curl https://inference.local/v1/chat/completions \ - --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' +openshell sandbox create \ + --name ollama-client \ + --provider ollama \ + -- \ + curl http://host.openshell.internal:11434/v1/chat/completions \ + --json '{ + "model":"qwen3.5:0.8b", + "messages":[{"role":"user","content":"hello"}], + "max_tokens":10 + }' ``` -The response should be JSON from the model. - - +For an OpenAI SDK client, set its base URL to +`http://host.openshell.internal:11434/v1`, use any non-empty API key value the +SDK accepts, and select the real Ollama model in the request. ## Troubleshooting -Common issues and fixes: - -- **Ollama not reachable from sandbox:** Ollama must be bound to `0.0.0.0`, not `127.0.0.1`. This applies to host-level Ollama only; the community sandbox handles this automatically. -- **`OPENAI_BASE_URL` wrong:** Use `http://host.openshell.internal:11434/v1`, not `localhost` or `127.0.0.1`. -- **Model not found:** Run `ollama ps` to confirm the model is loaded. Run `ollama pull ` if needed. -- **HTTPS instead of HTTP:** Code inside sandboxes must call `https://inference.local`, not `http://`. -- **AMD GPU driver issues:** Ollama v0.18+ requires ROCm 7 drivers for AMD GPUs. Update your drivers if you see GPU detection failures. - -Useful commands: - -```shell -openshell status -openshell inference get -openshell provider get ollama -``` +- Bind host-level Ollama to `0.0.0.0`, not `127.0.0.1`. +- Use `host.openshell.internal`, not `localhost`, from a sandbox. +- Confirm attachment with `openshell sandbox provider list ollama-client`. +- Inspect the effective policy with `openshell policy get ollama-client --full`. +- Run `ollama ps` and `ollama pull ` when the model is unavailable. +- If the gateway is remote, the hostname refers to the remote gateway host, + not your laptop. Use a shared service address or tunnel instead. ## Next Steps -- To learn more about managed inference, refer to [Inference Routing](/sandboxes/inference-routing). -- To configure a different self-hosted backend, refer to [Inference Routing](/sandboxes/inference-routing#configure-inference-routing). -- To learn how sandbox containers are selected, refer to [Sandboxes](/sandboxes/manage-sandboxes#custom-containers). +- [Provider-backed Inference](/sandboxes/inference-routing) +- [Profiles](/providers/profiles) +- [Customize Sandbox Policies](/sandboxes/policies) diff --git a/docs/get-started/tutorials/local-inference-lmstudio.mdx b/docs/get-started/tutorials/local-inference-lmstudio.mdx index 7ce6040091..4f7ffc7aed 100644 --- a/docs/get-started/tutorials/local-inference-lmstudio.mdx +++ b/docs/get-started/tutorials/local-inference-lmstudio.mdx @@ -1,214 +1,139 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Route Local Inference Requests to LM Studio" +title: "Use Local Inference with LM Studio" sidebar-title: "Local Inference with LM Studio" slug: "get-started/tutorials/local-inference-lmstudio" -description: "Configure inference.local to route sandbox requests to a local LM Studio server running on the gateway host." -keywords: "Generative AI, Cybersecurity, Tutorial, Inference Routing, LM Studio, Local Inference, Sandbox" +description: "Attach a provider profile for a local LM Studio server and call its native compatible endpoints." +keywords: "Generative AI, Cybersecurity, Tutorial, LM Studio, Local Inference, Sandbox, Provider Profile" --- -This tutorial describes how to configure OpenShell to route inference requests to a local LM Studio server. - - -The LM Studio server provides easy setup with both OpenAI and Anthropic compatible endpoints. - - - -This tutorial covers: - -- Expose a local inference server to OpenShell sandboxes. -- Verify end-to-end inference from inside a sandbox. +This tutorial exposes an LM Studio server on the gateway host to selected +sandboxes. The provider attachment grants access; the workload calls LM +Studio's OpenAI- or Anthropic-compatible endpoint directly. ## Prerequisites -First, complete OpenShell installation and follow the [Quickstart](/get-started/quickstart). - -[Install the LM Studio app](https://lmstudio.ai/download). Make sure that your LM Studio is running in the same environment as your gateway. +- Complete the [Quickstart](/get-started/quickstart). +- Install [LM Studio](https://lmstudio.ai/download) on the gateway host. -If you prefer to work without having to keep the LM Studio app open, download llmster (headless LM Studio) with the following command: +For a headless installation: - + + ```shell curl -fsSL https://lmstudio.ai/install.sh | bash ``` - + + ```shell irm https://lmstudio.ai/install.ps1 | iex ``` + -And start llmster: +Start the daemon and server: ```shell lms daemon up +lms server start --bind 0.0.0.0 ``` - - -## Start LM Studio Local Server - -Start the LM Studio local server from the Developer tab, and verify the OpenAI-compatible endpoint is enabled. - -LM Studio listens to `127.0.0.1:1234` by default. For use with OpenShell, configure LM Studio to listen on all interfaces (`0.0.0.0`). - -If you use the GUI, go to the Developer Tab, select Server Settings, then enable Serve on Local Network. - -If you use llmster in headless mode, run `lms server start --bind 0.0.0.0`. - -## Test with a small model - -In the LM Studio app, head to the Model Search tab to download a small model like Qwen3.5 2B. - -In the terminal, use the following command to download and load the model: +Load a model: ```shell lms get qwen/qwen3.5-2b lms load qwen/qwen3.5-2b ``` -## Add LM Studio as a provider - -Choose the provider type that matches the client protocol you want to route through `inference.local`. - - - - -Add LM Studio as an OpenAI-compatible provider through `host.openshell.internal`: - -```shell -openshell provider create \ - --name lmstudio \ - --type openai \ - --credential OPENAI_API_KEY=lmstudio \ - --config OPENAI_BASE_URL=http://host.openshell.internal:1234/v1 +## Create the Provider Profile + +Save this as `lmstudio.yaml`: + +```yaml +id: lmstudio +display_name: LM Studio +description: Host-level LM Studio compatible APIs +category: inference +inference_capable: true +credentials: [] +endpoints: + - host: host.openshell.internal + port: 1234 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/curl + - /usr/local/bin/curl + - /usr/bin/python3 + - /usr/local/bin/python + - /sandbox/.uv/python/** + - /sandbox/.venv/** ``` -Use this provider for clients that send OpenAI-compatible requests such as `POST /v1/chat/completions` or `POST /v1/responses`. - - - - - -Add a provider that points to LM Studio's Anthropic-compatible `POST /v1/messages` endpoint: - ```shell -openshell provider create \ - --name lmstudio-anthropic \ - --type anthropic \ - --credential ANTHROPIC_API_KEY=lmstudio \ - --config ANTHROPIC_BASE_URL=http://host.openshell.internal:1234 +openshell provider profile lint -f lmstudio.yaml +openshell provider profile import -f lmstudio.yaml +openshell provider create --name lmstudio --type lmstudio ``` -Use this provider for Anthropic-compatible `POST /v1/messages` requests. - - - - +The profile intentionally has no credential. LM Studio does not require one by +default, but the provider attachment still supplies its policy boundary. -## Configure LM Studio as the local inference provider +## Attach and Call the Native Endpoint -Set the managed inference route for the active gateway: - -

- - - +OpenAI-compatible request: ```shell -openshell inference set --provider lmstudio --model qwen/qwen3.5-2b -``` - -If the command succeeds, OpenShell has verified that the upstream is reachable and accepts the expected OpenAI-compatible request shape. - - - - - -```shell -openshell inference set --provider lmstudio-anthropic --model qwen/qwen3.5-2b -``` - -If the command succeeds, OpenShell has verified that the upstream is reachable and accepts the expected Anthropic-compatible request shape. - - - - - -
- -The active `inference.local` route is gateway-scoped, so only one provider and model pair is active at a time. Re-run `openshell inference set` whenever you want to switch between OpenAI-compatible and Anthropic-compatible clients. - -Confirm the saved config: - -```shell -openshell inference get -``` - -You should see either `Provider: lmstudio` or `Provider: lmstudio-anthropic`, along with `Model: qwen/qwen3.5-2b`. - -## Verify from Inside a Sandbox - -Run a simple request through `https://inference.local`: - - - - -```shell showLineNumbers={true} -openshell sandbox create -- \ - curl https://inference.local/v1/chat/completions \ - --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' - -openshell sandbox create -- \ - curl https://inference.local/v1/responses \ +openshell sandbox create \ + --name lmstudio-client \ + --provider lmstudio \ + -- \ + curl http://host.openshell.internal:1234/v1/chat/completions \ --json '{ - "instructions": "You are a helpful assistant.", - "input": "hello", - "max_output_tokens": 10 + "model":"qwen/qwen3.5-2b", + "messages":[{"role":"user","content":"hello"}], + "max_tokens":10 }' ``` - - - +Anthropic-compatible request: ```shell -openshell sandbox create -- \ - curl https://inference.local/v1/messages \ - --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' +openshell sandbox exec lmstudio-client -- \ + curl http://host.openshell.internal:1234/v1/messages \ + --json '{ + "model":"qwen/qwen3.5-2b", + "messages":[{"role":"user","content":"hello"}], + "max_tokens":10 + }' ``` - - - - - +Configure SDKs with the same native base URL. Some SDKs require a non-empty +API key even when LM Studio ignores it; use a literal non-secret value such as +`unused`. Keep the actual model ID in the client request. ## Troubleshooting -If setup fails, check these first: - -- LM Studio local server is running and reachable from the gateway host -- `OPENAI_BASE_URL` uses `http://host.openshell.internal:1234/v1` when you use an `openai` provider -- `ANTHROPIC_BASE_URL` uses `http://host.openshell.internal:1234` when you use an `anthropic` provider -- The gateway and LM Studio run on the same machine or a reachable network path -- The configured model name matches the model exposed by LM Studio - -Useful commands: - -```shell -openshell status -openshell inference get -openshell provider get lmstudio -openshell provider get lmstudio-anthropic -``` +- Enable **Serve on Local Network** in the LM Studio Developer tab, or use + `lms server start --bind 0.0.0.0`. +- Use `host.openshell.internal`, not `127.0.0.1` or `localhost`, in the + sandbox. +- Confirm the model is loaded with `lms ps`. +- Confirm the attachment with + `openshell sandbox provider list lmstudio-client`. +- Inspect endpoint and binary policy with + `openshell policy get lmstudio-client --full`. +- A remote gateway cannot reach an LM Studio process on your laptop without a + tunnel or shared network path. ## Next Steps -- To learn more about using the LM Studio CLI, refer to [LM Studio docs](https://lmstudio.ai/docs/cli) -- To learn more about managed inference, refer to [Inference Routing](/sandboxes/inference-routing). -- To configure a different self-hosted backend, refer to [Inference Routing](/sandboxes/inference-routing#configure-inference-routing). +- [Provider-backed Inference](/sandboxes/inference-routing) +- [Profiles](/providers/profiles) +- [LM Studio CLI](https://lmstudio.ai/docs/cli) diff --git a/docs/index.mdx b/docs/index.mdx index 3d774b1b40..b7623332ec 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "NVIDIA OpenShell Developer Guide" description: "OpenShell is the safe, private runtime for autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Inference Routing" +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Provider Credentials" position: 1 --- @@ -91,9 +91,9 @@ Deploy gateways, create sandboxes, configure policies, providers, and community Concept - + -Keep inference traffic private by routing API calls to local or self-hosted backends. +Attach model providers to selected sandboxes and call their native endpoints without exposing credentials. Concept diff --git a/docs/observability/logging.mdx b/docs/observability/logging.mdx index bc8d543246..4bc4aad6de 100644 --- a/docs/observability/logging.mdx +++ b/docs/observability/logging.mdx @@ -24,7 +24,7 @@ These events cover startup plumbing, gRPC communication, and internal state tran ### OCSF structured events -Network, process, filesystem, configuration, and API activity events use the [Open Cybersecurity Schema Framework (OCSF)](https://ocsf.io) format. OCSF is an open standard for normalizing security telemetry across tools and platforms. OpenShell maps sandbox events to OCSF v1.8.0 event classes, including API Activity [6003] with the `ai_operation` profile for inference observability. +Network, process, filesystem, configuration, and API activity events use the [Open Cybersecurity Schema Framework (OCSF)](https://ocsf.io) format. OCSF is an open standard for normalizing security telemetry across tools and platforms. OpenShell maps sandbox events to OCSF v1.8.0 event classes. In the log file, OCSF events appear in a shorthand format with an `OCSF` level label, designed for quick human and agent scanning: @@ -59,9 +59,8 @@ OpenShell maps sandbox events to these OCSF classes: | `SSH:` | SSH Activity | 4007 | SSH handshakes, authentication, channel operations | | `PROC:` | Process Activity | 1007 | Process start, exit, timeout, signal failures | | `FINDING:` | Detection Finding | 2004 | Security findings (nonce replay, proxy bypass, unsafe policy) | -| `CONFIG:` | Device Config State Change | 5019 | Policy load/reload, Landlock, TLS setup, inference routes | +| `CONFIG:` | Device Config State Change | 5019 | Policy load/reload, Landlock, TLS setup, provider and settings changes | | `LIFECYCLE:` | Application Lifecycle | 6002 | Sandbox supervisor start, SSH server ready | -| `API:INFERENCE` | API Activity | 6003 | AI model inference calls through `inference.local` (model, provider, latency, tokens) | ## Reading the Shorthand Format diff --git a/docs/observability/ocsf-json-export.mdx b/docs/observability/ocsf-json-export.mdx index ab76ead8a7..4fa7e45b4c 100644 --- a/docs/observability/ocsf-json-export.mdx +++ b/docs/observability/ocsf-json-export.mdx @@ -140,33 +140,6 @@ The `class_uid` field identifies the event type: | 2004 | Detection Finding | `FINDING:` | | 5019 | Device Config State Change | `CONFIG:` | | 6002 | Application Lifecycle | `LIFECYCLE:` | -| 6003 | API Activity | `API:INFERENCE` | - -### API Activity [6003] — AI Inference - -When the inference proxy routes a model call through `inference.local`, an API Activity event is emitted with the `ai_operation` profile: - -```json -{ - "class_uid": 6003, - "class_name": "API Activity", - "activity_id": 99, - "activity_name": "Other", - "api": { "operation": "POST /v1/messages" }, - "actor": { "process": { "name": "openshell-supervisor", "pid": 1 } }, - "src_endpoint": { "ip": "127.0.0.1", "port": 3128 }, - "ai_model": { "name": "claude-haiku-4-5-20251001", "ai_provider": "https://api.anthropic.com/v1" }, - "metadata": { "version": "1.8.0", "profiles": ["container", "host", "ai_operation"] }, - "status": "Success", - "unmapped": { "latency_ms": 701 } -} -``` - -The shorthand log renders this as: - -```text -OCSF API:INFERENCE [INFO] Success claude-haiku-4-5-20251001 via https://api.anthropic.com/v1 701ms [POST /v1/messages] -``` ## SIEM Schema Version Compatibility diff --git a/docs/providers/google-vertex-ai.mdx b/docs/providers/google-vertex-ai.mdx index 5c577efd5a..781768b7ed 100644 --- a/docs/providers/google-vertex-ai.mdx +++ b/docs/providers/google-vertex-ai.mdx @@ -3,28 +3,30 @@ # SPDX-License-Identifier: Apache-2.0 title: "Google Vertex AI" sidebar-title: "Google Vertex AI" -description: "Configure OpenShell to route inference traffic through Google Vertex AI, including Anthropic Claude and Gemini models." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Google Vertex AI, Anthropic Claude, Inference Routing" +description: "Attach a Google Vertex AI provider and call native Vertex endpoints with gateway-refreshed credentials." +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Google Vertex AI, Anthropic Claude" --- -Google Vertex AI is a managed machine learning platform that hosts Anthropic Claude, Gemini, and third-party models through Google Cloud. OpenShell can route `inference.local` traffic to Vertex AI using gateway-managed credential refresh, so sandbox agents do not handle GCP credentials directly. +The `google-vertex-ai` provider gives selected sandboxes access to native +Google Vertex AI endpoints. OpenShell keeps refresh bootstrap material at the +gateway, rotates short-lived access tokens, and resolves token placeholders +only at endpoints authorized by the provider profile. -## Prerequisites - -Before creating a Vertex AI provider, ensure you have: +OpenShell does not choose a model or transform a request. The workload uses the +native Vertex endpoint and request format for its selected model. -- A GCP project with the [Vertex AI API](https://console.cloud.google.com/apis/library/aiplatform.googleapis.com) enabled. -- One of the following: - - A GCP service account with the **Vertex AI User** role and a downloaded JSON key file, for production use. - - The `gcloud` CLI with Application Default Credentials configured, for local development. +## Prerequisites -## Authentication +- A GCP project with the Vertex AI API enabled. +- A service account with the Vertex AI User role and a downloaded JSON key for + production, or gcloud Application Default Credentials for local development. +- Access to the selected model in the intended Vertex region. -The `google-vertex-ai` provider supports two credential sources. +## Create a Provider ### Service Account Key -Supply the JSON key file content as the `GOOGLE_SERVICE_ACCOUNT_KEY` credential. OpenShell persists that value only as gateway-side refresh bootstrap material until you update or delete it. The raw service-account JSON and private key are not sandbox runtime credentials and are not exposed to sandboxes. Runtime inference requests use short-lived access tokens minted by the gateway and stored under a separate credential key. +Create the provider with the JSON key as gateway-only bootstrap material: ```shell openshell provider create \ @@ -35,7 +37,7 @@ openshell provider create \ --config VERTEX_AI_REGION=us-central1 ``` -Then configure gateway-managed refresh so the gateway uses the private key as refresh bootstrap material and rotates access tokens: +Configure gateway-managed refresh: ```shell openshell provider refresh configure vertex-prod \ @@ -46,15 +48,16 @@ openshell provider refresh configure vertex-prod \ --secret-material-key private_key ``` +The private key remains in the gateway credential store. Sandboxes receive +only an opaque placeholder for the short-lived access token. + ### gcloud Application Default Credentials -For local development, configure ADC first, then pass `--from-gcloud-adc`: +For local development: ```shell gcloud auth application-default login -``` -```shell openshell provider create \ --name vertex-local \ --type google-vertex-ai \ @@ -63,154 +66,115 @@ openshell provider create \ --config VERTEX_AI_REGION=us-central1 ``` -`--from-gcloud-adc` reads `GOOGLE_APPLICATION_CREDENTIALS` first, then falls back to `$CLOUDSDK_CONFIG/application_default_credentials.json` when that environment variable is set, then to `~/.config/gcloud/application_default_credentials.json`. It configures an OAuth2 refresh token flow on the gateway and immediately mints the first access token before the command returns. If the command succeeds, the provider is ready for inference right away. It only works with user credentials generated by `gcloud auth application-default login`. If your ADC file is a service account key, the CLI returns an error and directs you to use the service account key method above. - -ADC-backed providers mint and rotate access tokens into `GOOGLE_VERTEX_AI_TOKEN`. - - -`--from-gcloud-adc` is valid for `google-vertex-ai` and `google-cloud` providers. - +`--from-gcloud-adc` reads authorized-user ADC, configures an OAuth2 refresh +grant at the gateway, and immediately mints `GOOGLE_VERTEX_AI_TOKEN`. The ADC +file and refresh token do not enter the sandbox. ## Configuration Keys -Pass these as `--config KEY=VALUE` when creating the provider, or set them as environment variables and use `--from-existing`. - | Key | Required | Default | Description | |---|---|---|---| -| `VERTEX_AI_PROJECT_ID` | Yes (unless `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` is set) | — | GCP project ID. | -| `VERTEX_AI_REGION` | No | `us-central1` | Vertex location selector. Use a regional location such as `us-central1`, or `global`, `us`, or `eu` for the supported global and multi-region endpoints. | -| `GOOGLE_VERTEX_AI_BASE_URL` | No | — | Full base URL override for non-Anthropic routes. Must be an official Vertex AI HTTPS endpoint root. | -| `VERTEX_AI_BASE_URL` | No | — | Backward-compatible alias for `GOOGLE_VERTEX_AI_BASE_URL`. | -| `VERTEX_AI_PUBLISHER` | No | Inferred from model name | Set to `anthropic` to force Anthropic Messages API routing, or any other value for OpenAI-compatible routing. | - -When `VERTEX_AI_PROJECT_ID` is set and no base URL override is present, the gateway maps `VERTEX_AI_REGION` to the Vertex host automatically: +| `VERTEX_AI_PROJECT_ID` | Yes | — | GCP project ID exposed as non-secret workload configuration. | +| `VERTEX_AI_REGION` | No | `us-central1` | Vertex location exposed as non-secret workload configuration. | -- Regional locations such as `us-central1` use `https://-aiplatform.googleapis.com`. -- `global` uses `https://aiplatform.googleapis.com`. -- `us` and `eu` use `https://aiplatform..rep.googleapis.com`. +When the provider is attached, OpenShell also projects standard project and +location aliases such as `GOOGLE_CLOUD_PROJECT`, `ANTHROPIC_VERTEX_PROJECT_ID`, +`CLOUD_ML_REGION`, and `VERTEX_LOCATION`. -For Anthropic models, OpenShell builds the publisher-model Vertex path automatically and injects `anthropic_version` into the request body. Vertex rawPredict does not receive `anthropic-version` as a header, and OpenShell strips `anthropic-beta` for Vertex Claude routes. For non-Anthropic models, OpenShell uses Vertex's OpenAI-compatible Chat Completions route under `.../endpoints/openapi/chat/completions`. +## Attach the Provider - -Use `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` only for non-Anthropic Vertex routes. OpenShell rejects Anthropic models when a base URL override is set because Anthropic routes require model-path shaping and `anthropic_version` body injection. Overrides must use `https://` and an official Vertex AI hostname such as `aiplatform.googleapis.com`, `aiplatform.us.rep.googleapis.com`, `aiplatform.eu.rep.googleapis.com`, or `-aiplatform.googleapis.com`. - - -## Supported Models - -Vertex AI hosts Anthropic Claude models (claude-3-5-sonnet, claude-3-opus, and others) through a native Messages API integration, and Gemini and other third-party models through Vertex's OpenAI-compatible Chat Completions endpoint. OpenShell infers the routing path from the model name. For the full list of available models and regions, refer to the [Google Cloud model garden documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/overview). - -Model names that match the `claude-*` pattern route through the Anthropic Messages API on Vertex. All other model names route through Vertex Chat Completions. Set `VERTEX_AI_PUBLISHER=anthropic` to force Anthropic routing when the model name does not follow the standard pattern. - -OpenShell exposes Anthropic Vertex routes for inference only. It does not advertise OpenAI-style model discovery for those routes, so use the Google Cloud docs or Model Garden to discover supported Anthropic model IDs. - -## Configure Inference Routing - -Point `inference.local` at the provider. The Vertex AI profile contributes its -network endpoints to sandbox policies automatically: +Attach it while creating a sandbox: ```shell -openshell inference set \ - --provider vertex-prod \ - --model claude-sonnet-4-6 +openshell sandbox create \ + --name vertex-agent \ + --provider vertex-local ``` -Use `--no-verify` if the endpoint verification fails. This is common with the `global` region, where the validation probe may not match the actual rawPredict path: +Or attach it to an existing sandbox: ```shell -openshell inference set \ - --provider vertex-prod \ - --model claude-sonnet-4-6 \ - --no-verify +openshell sandbox provider attach vertex-agent vertex-local ``` -Sandboxes on that gateway reach the model at `https://inference.local`. For full details on inference routing, refer to [Inference Routing](/sandboxes/inference-routing). - -## Use from a Sandbox +Launch a new process after runtime attachment so it receives the provider +environment. Existing processes do not gain newly attached environment +variables. -Agents inside sandboxes should reach Vertex AI through `inference.local`, not by connecting to Vertex AI directly. The gateway manages GCP credential refresh and request translation; the agent only needs to point its SDK at the local endpoint. +## Call the Native Vertex API -The complete setup from scratch: +Claude models use Vertex's publisher-model endpoint. Run a request from a new +sandbox process: ```shell -# 1. Create the provider -openshell provider create \ - --name vertex-local \ - --type google-vertex-ai \ - --from-gcloud-adc \ - --config VERTEX_AI_PROJECT_ID=my-gcp-project \ - --config VERTEX_AI_REGION=us-central1 - -# 2. Configure inference routing -openshell inference set --provider vertex-local --model claude-sonnet-4-6 --no-verify - -# 3. Create a sandbox with the provider attached -openshell sandbox create --name my-sandbox --provider vertex-local +openshell sandbox exec vertex-agent -- sh -lc ' + token=${GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN:-$GOOGLE_VERTEX_AI_TOKEN} + curl -X POST \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d '\''{ + "anthropic_version":"vertex-2023-10-16", + "max_tokens":1024, + "messages":[{"role":"user","content":"Hello"}] + }'\'' \ + "https://${CLOUD_ML_REGION}-aiplatform.googleapis.com/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${CLOUD_ML_REGION}/publishers/anthropic/models/claude-sonnet-4-6:rawPredict" +' ``` -Then inside the sandbox, launch the agent as shown below. +Use the model ID and location supported by your GCP project. For `global`, `us`, +or `eu`, use the corresponding Google-documented hostname instead of the +regional `-aiplatform.googleapis.com` form. - - +Gemini and third-party models use their documented native or +OpenAI-compatible Vertex endpoints. Configure the model, URL, streaming mode, +and timeout in the client. OpenShell does not rewrite them. -```shell -ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare -``` - -`--bare` skips the OAuth login flow and uses `ANTHROPIC_API_KEY` directly for authentication. The key value does not reach Vertex AI — `inference.local` strips it and injects the real GCP access token before forwarding. +## Verify and Troubleshoot - -Do not set `CLAUDE_CODE_USE_VERTEX=1` inside the sandbox. That flag makes Claude Code connect directly to Vertex AI and attempt GCP credential discovery (ADC file, metadata service), which fails because the sandbox does not expose GCP credentials. Use `inference.local` instead. - - - - +Inspect the attachment and effective policy: ```shell -ANTHROPIC_BASE_URL="https://inference.local/v1" ANTHROPIC_API_KEY=unused opencode +openshell sandbox provider list vertex-agent +openshell policy get vertex-agent --full +openshell provider refresh status vertex-local ``` -OpenCode requires `/v1` in the base URL. Without it, OpenCode sends `POST /messages` instead of `POST /v1/messages`, which does not match the inference pattern and is denied. - - - +Common failures: -### Policy Proposals +- A missing token variable usually means the process started before provider + attachment. Launch a new process. +- `connection not allowed by policy` means the provider endpoint or caller + binary is absent from the effective policy. A gateway global policy override + suppresses provider-derived entries. +- `credential_endpoint_mismatch` means the request destination is outside the + provider profile's endpoint binding. +- A Vertex 400 or 404 usually means the model, location, publisher path, or + request body does not match the native API. +- A Vertex 401 or 403 can indicate an expired refresh grant or missing GCP IAM + permission. Check `provider refresh status` and the Vertex AI User role. -After running an agent, the TUI (`openshell term`) may show policy proposals for denied endpoints. Common ones for Vertex AI sandboxes: +Provider creation does not verify model access. The native request is the +end-to-end check. -| Endpoint | Action | Reason | -|---|---|---| -| `metadata.google.internal:80` | **Reject** | Resolves to `169.254.169.254` (GCE metadata service). Always blocked regardless of policy — the proxy blocks the resolved IP unconditionally to prevent credential exfiltration. | -| `downloads.claude.ai:443` | Approve if desired | Claude Code update checking and asset loading. Not required for inference. | -| `storage.googleapis.com:443` | Approve if desired | Google Cloud Storage. Used by some Claude Code features. Not required for inference. | +## Migrate an Existing Vertex Route -## From Existing Environment +An earlier managed route stored the provider and model separately and rewrote +requests for the workload. After upgrading, the provider and its refresh state +remain, but the route does not. -If one of these token env vars is already set in your shell, create the provider with `--from-existing`: - -- `GOOGLE_VERTEX_AI_TOKEN` or `VERTEX_AI_TOKEN` -- `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` or `VERTEX_AI_SERVICE_ACCOUNT_TOKEN` - -OpenShell also reads these config env vars during `--from-existing`: - -- `VERTEX_AI_PROJECT_ID` -- `VERTEX_AI_REGION` -- `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` -- `VERTEX_AI_PUBLISHER` - -Then create the provider: - -```shell -openshell provider create \ - --name vertex-env \ - --type google-vertex-ai \ - --from-existing -``` +1. Attach the preserved Vertex provider to each intended sandbox. +2. Launch new workload processes. +3. Move the route's model and timeout into the client configuration. +4. Change the client to the native Vertex endpoint and request format. +5. Verify one non-streaming and one streaming native request before production + rollout. -This reads credentials and config from the environment variables listed in the configuration keys table above. +Do not attach the provider to every sandbox automatically. The old route was +workspace-global; the replacement intentionally grants access per sandbox. ## Next Steps -- To configure `inference.local` routing, refer to [Inference Routing](/sandboxes/inference-routing). -- To manage provider credentials and refresh, refer to [Providers](/sandboxes/manage-providers). -- To apply network policies to sandboxes using this provider, refer to [Policies](/sandboxes/policies). +- [Provider-backed Inference](/sandboxes/inference-routing) +- [Profiles](/providers/profiles) +- [Providers](/sandboxes/manage-providers) +- [Customize Sandbox Policies](/sandboxes/policies) diff --git a/docs/providers/profiles.mdx b/docs/providers/profiles.mdx index c310c02b15..e8f6ecc45a 100644 --- a/docs/providers/profiles.mdx +++ b/docs/providers/profiles.mdx @@ -55,11 +55,11 @@ supply that boundary by default. For an endpointless profile, a sandbox policy endpoint can name the attached provider instance explicitly. The proxy checks the resulting association before it substitutes the real value. -The built-in `openai` and `anthropic` profiles use their public vendor -endpoints. Configuring an alternate base URL makes those providers route-only: -OpenShell withholds their static credentials and fixed-vendor policy from -direct sandbox traffic while gateway inference routing continues to use the -configured upstream. +The built-in `openai` and `anthropic` profiles bind credentials to their public +vendor endpoints. To use a proxy or compatible API at another host, import a +custom profile that declares that host and attach a provider created from that +profile. OpenShell never broadens a built-in profile's credential boundary from +a base URL environment variable alone. A request can use a static credential only when all of these checks pass: @@ -204,16 +204,15 @@ The following provider profile design items are not part of the current behavior | Binary-scoped credential injection | Provider profile binaries affect policy composition but do not yet restrict placeholder resolution by calling binary. Static and dynamic credentials are endpoint-scoped. | | Credential verification on create | `openshell provider create` does not yet probe provider verification endpoints or expose `--no-verify`. | | Automatic credential scope extraction | OpenShell does not yet inspect upstream provider responses to discover credential scopes. | -| Inference mounting from attached providers | `inference_capable` is profile metadata. Attaching an inference-capable provider does not yet create `inference.local` routes. | -| Multi-provider inference routing | Path-based routing such as `inference.local/openai/...` and `inference.local/anthropic/...` is not yet wired to provider profiles. | | Policy prover integration | OpenShell does not yet run the policy prover automatically on sandbox startup or block startup based on prover findings. | | Refresh telemetry as OCSF events | Credential refresh logs are secret-safe gateway logs. OCSF refresh events and metrics are future work. | -Use [Inference Routing](/sandboxes/inference-routing) for the current `inference.local` model. +Use [Provider-backed Inference](/sandboxes/inference-routing) to attach an +inference provider and call its native endpoint. ## Provider Profiles -A provider profile defines a provider type. It contains metadata, credential declarations, endpoint policy, binary policy, inference metadata, and optional credential refresh metadata. +A provider profile defines a provider type. It contains metadata, credential declarations, endpoint policy, binary policy, an informational provider category, and optional credential refresh metadata. List available profiles: @@ -439,7 +438,10 @@ environment value under the actual environment variable key. `binaries` contains the executable paths allowed to reach the profile endpoints when the profile contributes policy to a sandbox. -`inference_capable` marks profiles that are intended to participate in inference workflows. It does not currently mount or configure `inference.local`. +`inference_capable` is informational metadata that marks profiles intended for +model and inference APIs. It does not grant access, select a model, configure a +client, or change routing. The sandbox must attach a provider instance, and the +workload calls the profile-authorized native endpoint. ### Refresh Metadata diff --git a/docs/reference/gateway-auth.mdx b/docs/reference/gateway-auth.mdx index ebfd5c7cbc..34a97b8fa1 100644 --- a/docs/reference/gateway-auth.mdx +++ b/docs/reference/gateway-auth.mdx @@ -210,7 +210,7 @@ Common identity providers such as Keycloak (RS256), Microsoft Entra ID (RSA), an If `OPENSHELL_OIDC_SCOPES_CLAIM` is set, the gateway also enforces scopes. It accepts space-delimited scope strings such as `scope: "openid sandbox:read"` and JSON arrays such as `scp: ["sandbox:read"]`. Standard OIDC scopes such as `openid`, `profile`, `email`, and `offline_access` are ignored for authorization. `openshell:all` grants access to all scoped methods. -Supervisor-to-gateway RPCs do not use user OIDC tokens or mTLS user identity. Each sandbox supervisor presents a gateway-minted `Authorization: Bearer` token scoped to its sandbox ID. On Kubernetes, the Kubernetes compute driver validates the projected ServiceAccount token with TokenReview, verifies the live pod UID and controlling `Sandbox` ownerReference, and returns the authenticated sandbox ID to the gateway. The gateway verifies that sandbox still exists before minting its JWT. Log upload, policy status, credential environment lookup, inference bundle lookup, and sandbox config sync run with sandbox-restricted scope, while CLI users authenticate with OIDC, edge auth, local mTLS user authentication, or an explicitly enabled unauthenticated local developer mode. `GetInferenceBundle` returns route material that includes provider credentials, so it requires a sandbox principal; user callers manage inference configuration through the user-facing inference APIs instead. +Supervisor-to-gateway RPCs do not use user OIDC tokens or mTLS user identity. Each sandbox supervisor presents a gateway-minted `Authorization: Bearer` token scoped to its sandbox ID. On Kubernetes, the Kubernetes compute driver validates the projected ServiceAccount token with TokenReview, verifies the live pod UID and controlling `Sandbox` ownerReference, and returns the authenticated sandbox ID to the gateway. The gateway verifies that sandbox still exists before minting its JWT. Log upload, policy status, provider environment lookup, and sandbox config sync run with sandbox-restricted scope, while CLI users authenticate with OIDC, edge auth, local mTLS user authentication, or an explicitly enabled unauthenticated local developer mode. Provider environment responses expose only the credentials and configuration attached to that sandbox, subject to endpoint binding and credential expiry checks. Re-authenticate an OIDC gateway with: diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 987e66b0d9..51978bbb2f 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -143,7 +143,7 @@ the gateway. If the primary listener covers that address, the gateway reuses it and sandbox JWT authentication restricts the supervisor to its callback RPC allowlist. If the primary listener is not reachable through that address, the gateway creates an additional callback-only listener. Use the primary endpoint -for CLI, administrator, health, reflection, inference-route management, and +for CLI, administrator, health, reflection, provider management, and HTTP requests. A `PermissionDenied` response from an additional callback-only listener is expected for those requests. Do not broaden the primary listener to `0.0.0.0` solely to make sandbox callbacks reachable. diff --git a/docs/sandboxes/inference-routing.mdx b/docs/sandboxes/inference-routing.mdx index 1b47bfdf95..902c09eea0 100644 --- a/docs/sandboxes/inference-routing.mdx +++ b/docs/sandboxes/inference-routing.mdx @@ -1,348 +1,311 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Inference Routing" -sidebar-title: "Inference Routing" -description: "Understand and configure OpenShell inference routing through inference.local and external endpoints." -keywords: "Generative AI, Cybersecurity, Inference Routing, Configuration, Privacy, LLM, Provider" +title: "Provider-backed Inference" +sidebar-title: "Provider-backed Inference" +description: "Grant a sandbox access to model providers through provider profiles, attachments, and native endpoints." +keywords: "Generative AI, Cybersecurity, Inference, Provider Profiles, Credentials, LLM" position: 8 --- -OpenShell handles inference traffic through two paths: external endpoints and `inference.local`. +OpenShell grants model-provider access through provider profiles and sandbox +attachments. The workload calls the provider's native API. OpenShell evaluates +the request against the profile-derived network policy and substitutes the real +credential only at an endpoint authorized by that profile. -| Path | How it works | -|---|---| -| External endpoints | Traffic to hosts like `api.openai.com` or `api.anthropic.com` is treated like any other outbound request, allowed or denied by `network_policies`. Refer to [Policies](/sandboxes/policies). | -| `inference.local` | A sandbox-local HTTPS endpoint that routes model requests through the gateway. The privacy router strips sandbox-supplied credentials, forwards only approved inference headers, injects the configured backend credentials, and forwards to the managed model endpoint. | - -## How `inference.local` Works - -When code inside a sandbox calls `https://inference.local`, the privacy router routes the request to the configured backend for that gateway. The configured model is applied to generation requests, provider credentials come from OpenShell rather than from code inside the sandbox, and only approved inference headers are forwarded upstream. - -If code calls an external inference host directly, OpenShell evaluates that traffic only through `network_policies`. +This keeps the complete access contract in one place: -| Property | Detail | +| Concern | Owner | |---|---| -| Credentials | No sandbox API keys needed. Credentials come from the configured provider record. The router strips caller-supplied `Authorization` before forwarding the request. | -| Header forwarding | `inference.local` forwards only a per-provider header allowlist. OpenAI routes allow `openai-organization` and `x-model-id`. Anthropic routes allow `anthropic-version` and `anthropic-beta`. Vertex Claude rawPredict routes strip `anthropic-beta` and do not forward `anthropic-version` as a header because the router injects `anthropic_version` into the Vertex request body. NVIDIA routes allow `x-model-id`. AWS Bedrock routes have no passthrough headers today. All other caller headers are stripped. | -| Configuration | One provider and one model define sandbox inference for the active gateway. Every sandbox on that gateway sees the same `inference.local` backend. | -| Provider support | NVIDIA, Anthropic, Google Vertex AI, AWS Bedrock (via a translating bridge — direct AWS with SigV4 signing is a separate follow-up), and any OpenAI-compatible provider all work through the same endpoint. Vertex routes Claude models through `/v1/messages` and non-Anthropic models through `/v1/chat/completions`. The gateway resolves the upstream Vertex host from the provider config, including regional, global, and supported multi-region endpoints. | -| Streaming reliability | The router tolerates idle gaps of up to 120 seconds between streamed chunks so long reasoning responses are not cut off mid-stream. | -| Hot refresh | OpenShell picks up provider credential changes and inference updates without recreating sandboxes. Changes propagate within about 5 seconds by default. | - -## Supported API Patterns - -Supported request patterns depend on the provider configured for `inference.local`. - - - - -| Pattern | Method | Path | -|---|---|---| -| Chat Completions | `POST` | `/v1/chat/completions` | -| Completions | `POST` | `/v1/completions` | -| Responses | `POST` | `/v1/responses` | -| Embeddings | `POST` | `/v1/embeddings` | -| Model Discovery | `GET` | `/v1/models` | -| Model Discovery | `GET` | `/v1/models/*` | - - - - - -| Pattern | Method | Path | -|---|---|---| -| Messages | `POST` | `/v1/messages` | - - - - - -| Pattern | Method | Path | -|---|---|---| -| InvokeModel | `POST` | `/model/{modelId}/invoke` | - -The `{modelId}` segment is constrained to a single non-empty path segment to avoid path-traversal liabilities. `/model//invoke` and `/model/a/b/invoke` both no-match. - - -Today the `aws-bedrock` provider type is bridge-fronted only. The router does not inject any auth header on outbound requests; the configured `BEDROCK_BASE_URL` is expected to point at a translating bridge or Bedrock-compatible proxy whose own pod holds operator-side credentials. SigV4 signing for direct AWS Bedrock is deferred to a follow-up release. - -`InvokeModelWithResponseStream` is intentionally not advertised yet. The streaming path emits AWS event-stream framing, which our protocol-aware error path does not yet model; surfacing it without that work risks shipping responses the sandbox cannot interpret on failure. It will land alongside the streaming-error work in a follow-up. - - - - - -Requests to `inference.local` that do not match the configured provider's supported patterns are denied. - -Google Vertex AI does not expose every OpenAI-compatible path through `inference.local`. Vertex routes for Gemini and other non-Anthropic models currently support Chat Completions. Vertex routes for Claude models use the Anthropic Messages pattern. Base URL overrides are only supported for non-Anthropic Vertex routes. - -## Configure Inference Routing - -The managed local inference endpoint uses three values: - -| Value | Description | -|---|---| -| Provider record | The credential backend OpenShell uses to authenticate with the upstream model host. | -| Model ID | The model to use for generation requests. | -| Timeout | Per-request timeout in seconds for upstream inference calls. Defaults to 60 seconds. | - -For tested providers and base URLs, refer to [Supported Inference Providers](/sandboxes/manage-providers#supported-inference-providers). +| Credential and refresh lifecycle | Provider instance | +| Authorized hosts, ports, paths, and binaries | Provider profile | +| Which workload receives access | Sandbox provider attachment | +| Base URL, model, request shape, and timeout | Native client or workload configuration | -## Create a Provider +## Define and Attach a Hosted Provider -Create a provider that holds the backend credentials you want OpenShell to use. - - - +Start from an existing profile, review its access, and import it under a new +ID. For example, export the NVIDIA profile: ```shell -openshell provider create --name nvidia-prod --type nvidia --from-existing +openshell provider profile export nvidia -o yaml > nvidia-native.yaml ``` -This reads `NVIDIA_API_KEY` from your environment. - - - - - -Any cloud provider that exposes an OpenAI-compatible API works with the `openai` provider type. You need three values from the provider: the base URL, an API key, and a model name. - -```shell -openshell provider create \ - --name my-cloud-provider \ - --type openai \ - --credential OPENAI_API_KEY= \ - --config OPENAI_BASE_URL=https://api.example.com/v1 +In `nvidia-native.yaml`, change `id` to `nvidia-native`, give the profile a +distinct `display_name`, and set `binaries` to the paths that may call the API. +Keep the credential and endpoint definitions you intend to grant. For a Python +workload, the edited fields can look like: + +```yaml +id: nvidia-native +display_name: NVIDIA Native API +binaries: + - /usr/bin/python3 + - /usr/bin/python3.13 + - /usr/local/bin/python + - /sandbox/.venv/** ``` -Replace the base URL and API key with the values from your provider. For supported providers out of the box, refer to [Supported Inference Providers](/sandboxes/manage-providers#supported-inference-providers). For other providers, refer to your provider's documentation for the correct base URL, available models, and API key setup. - -This override is used by gateway inference routing. It disables the built-in -public OpenAI endpoint policy and credential binding for direct sandbox -traffic, so the alternate-upstream key cannot be substituted at -`api.openai.com`. Import an endpoint-bearing custom profile if a sandbox must -contact the alternate upstream directly. - - - - +Lint and import the complete edited profile, then create a provider from its +new ID: ```shell -openshell provider create \ - --name vertex-local \ - --type google-vertex-ai \ - --from-gcloud-adc \ - --config VERTEX_AI_PROJECT_ID=my-gcp-project \ - --config VERTEX_AI_REGION=us-central1 -``` +openshell provider profile lint -f nvidia-native.yaml +openshell provider profile import -f nvidia-native.yaml -Use [Google Vertex AI](/providers/google-vertex-ai) for the full auth flows, including the production service-account refresh path, ADC-backed providers that mint `GOOGLE_VERTEX_AI_TOKEN`, and `--from-existing` support. - - - - - -```shell openshell provider create \ - --name my-local-model \ - --type openai \ - --credential OPENAI_API_KEY=empty-if-not-required \ - --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 + --name nvidia-prod \ + --type nvidia-native \ + --from-existing + +openshell sandbox create \ + --name inference-demo \ + --provider nvidia-prod \ + -- python app.py ``` -Use `--config OPENAI_BASE_URL` to point to any OpenAI-compatible server running where the gateway runs. For host-backed local inference, use `host.openshell.internal` or the host's LAN IP. Avoid `127.0.0.1` and `localhost`. Set `OPENAI_API_KEY` to a dummy value if the server does not require authentication. - -The override is route-only and does not add the local endpoint to sandbox -network policy. - - -For a self-contained setup, the Ollama sandbox bundles Ollama inside the sandbox itself, so no host-level provider is needed. Refer to [Inference Ollama](/get-started/tutorials/inference-ollama) for details. +The imported provider profile supplies `NVIDIA_API_KEY` as an opaque placeholder and +allows the profile's native endpoint. Configure the client with the real model +identifier: - - -Ollama also supports cloud-hosted models using the `:cloud` tag suffix, for example `qwen3.5:cloud`. - - +```python +import os +from openai import OpenAI - +client = OpenAI( + base_url="https://integrate.api.nvidia.com/v1", + api_key=os.environ["NVIDIA_API_KEY"], + timeout=300, +) -```shell -openshell provider create --name anthropic-prod --type anthropic --from-existing +response = client.responses.create( + model="nvidia/nemotron-3-nano-30b-a3b", + input="Hello!", +) +print(response.output_text) ``` -This reads `ANTHROPIC_API_KEY` from your environment. +Use the same export, edit, and import flow for OpenAI, Anthropic, or another +hosted service. Set a new profile ID, retain only the endpoints and credentials +the workload needs, and name the actual client binaries. Provider attachment +does not select or rewrite a model. - +This release still loads built-in profiles for compatibility, so existing +providers continue to resolve their profiles. Treat those built-ins as starting +templates for new provider definitions rather than the primary setup workflow. - +## Attach a Provider to a Running Sandbox ```shell -openshell provider create \ - --name bedrock-bridge \ - --type aws-bedrock \ - --credential AWS_ACCESS_KEY_ID=unused-bridge-fronted-shape \ - --config BEDROCK_BASE_URL=http://your-bedrock-bridge.your-ns.svc.cluster.local:8080 +openshell sandbox provider attach inference-demo nvidia-prod +openshell sandbox provider list inference-demo ``` -Then set the inference route, passing `--no-verify` because the validation probe does not yet support Bedrock protocols: +Running sandboxes poll for provider and effective-policy changes. Launch a new +process after attachment so it receives the new credential placeholder: ```shell -openshell inference set \ - --provider bedrock-bridge \ - --model anthropic.claude-3-5-sonnet-20241022-v2:0 \ - --no-verify +openshell sandbox exec inference-demo -- python app.py ``` -**Why a placeholder credential?** `provider create` requires a non-empty `credentials` map even when the upstream auth scheme is `AuthHeader::None` — `aws-bedrock` falls into that bucket today because the router never injects a credential header on outbound requests; the bridge holds operator-side auth in its own pod. Any non-empty string value satisfies the structural requirement; `unused-bridge-fronted-shape` makes the intent obvious in `openshell provider get` output. The same pattern applies to any standalone-router profile that registers `AuthHeader::None`. When the SigV4 follow-up lands and the router begins signing requests itself, this becomes a real key. - -**About the bridge-fronted shape.** The router does not inject any auth header on outbound requests. Point `BEDROCK_BASE_URL` at a translating bridge or Bedrock-compatible proxy that handles authentication in its own pod. The bridge is expected to accept Bedrock InvokeModel requests on the patterns listed above and forward to the operator's real upstream. - -**About `--no-verify`.** The default validation probe does not yet recognize the `aws_bedrock_invoke` protocol, so without `--no-verify` the `inference set` call would fail before it could mint a route. The first sandbox round-trip is the real verification today. - -**For direct AWS Bedrock**, refer to a future release that adds the SigV4 router-side signer. Until then, a `BEDROCK_BASE_URL` is required at provider-create time — the core profile sets `default_base_url: ""`, so route resolution rejects providers without it rather than silently forwarding prompts to AWS with no usable auth. - - - +An already-running process does not gain new environment variables. Credential +rotation and detach still take effect immediately at placeholder resolution. -## Set Inference Routing - -Point `inference.local` at that provider and choose the model to use: +Detach the provider to revoke its policy and credential access: ```shell -openshell inference set \ - --provider nvidia-prod \ - --model nvidia/nemotron-3-nano-30b-a3b +openshell sandbox provider detach inference-demo nvidia-prod ``` -To override the default 60-second per-request timeout, add `--timeout`: - -```shell -openshell inference set \ - --provider nvidia-prod \ - --model nvidia/nemotron-3-nano-30b-a3b \ - --timeout 300 +## Use a Custom or Self-hosted Endpoint + +Do not reuse the built-in `openai` or `anthropic` profile for an alternate +host. Those profiles bind credentials to the public vendor endpoints. Import a +profile that names the intended endpoint and the binaries that may call it. + +For a credentialless Ollama server on the gateway host, save this profile as +`ollama-openai.yaml`: + +```yaml +id: ollama-openai +display_name: Ollama +description: Host-local Ollama OpenAI-compatible API +category: inference +inference_capable: true +credentials: [] +endpoints: + - host: host.openshell.internal + port: 11434 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/curl + - /usr/local/bin/curl + - /usr/bin/python3 + - /usr/local/bin/python + - /sandbox/.uv/python/** + - /sandbox/.venv/** ``` -The value is in seconds. When `--timeout` is omitted or set to `0`, the default of 60 seconds applies. Increase `--timeout` when you expect extended thinking phases so the full response completes before the request deadline. - -## Inspect and Update the Config - -Confirm that the provider and model are set correctly: - -```shell -openshell inference get -Gateway inference: - - Provider: nvidia-prod - Model: nvidia/nemotron-3-nano-30b-a3b - Timeout: 300s - Version: 1 -``` - -Use `update` when you want to change only one field: +Import the profile, create an instance, and attach it: ```shell -openshell inference update --model nvidia/nemotron-3-nano-30b-a3b -openshell inference update --provider openai-prod -openshell inference update --timeout 120 +openshell provider profile lint -f ollama-openai.yaml +openshell provider profile import -f ollama-openai.yaml +openshell provider create --name ollama --type ollama-openai + +openshell sandbox create \ + --name ollama-client \ + --provider ollama \ + --env OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 \ + -- python app.py ``` -## Use the Local Endpoint from a Sandbox +Use any non-empty placeholder value if the client library requires an API key +for a server that does not authenticate requests: -After inference is configured, code inside any sandbox can call `https://inference.local` directly. The client-supplied `model` and `api_key` values are not sent upstream — the privacy router injects the real credentials from the configured provider and rewrites the model before forwarding. Some SDKs require a non-empty API key even though `inference.local` does not use the sandbox-provided value; pass any placeholder such as `unused`. - - - +```python +import os +from openai import OpenAI -```shell -ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare +client = OpenAI( + base_url=os.environ["OPENAI_BASE_URL"], + api_key="unused", +) +response = client.chat.completions.create( + model="qwen3.5:0.8b", + messages=[{"role": "user", "content": "Hello"}], +) ``` -`--bare` skips the OAuth login flow and uses `ANTHROPIC_API_KEY` directly. The key is stripped by the proxy and never reaches the upstream provider. +For an authenticated alternate endpoint, declare a credential in the custom +profile, bind it to that endpoint, and create the provider from the original +credential source. OpenShell never exports stored credential values. - -Claude Code appends `/v1/messages` to `ANTHROPIC_BASE_URL`, so omit the `/v1` suffix from the base URL. - +## Verify Access - - +Inspect the effective policy, including provider-derived entries: ```shell -ANTHROPIC_BASE_URL="https://inference.local/v1" ANTHROPIC_API_KEY=unused opencode +openshell policy get inference-demo --full ``` - -OpenCode appends `/messages` directly to `ANTHROPIC_BASE_URL`. Include the `/v1` suffix so the full path becomes `/v1/messages`, which matches the inference pattern. - +Then run a native request from a new sandbox process. A successful request +confirms endpoint policy, binary attribution, credential substitution, DNS, +and upstream service behavior. - - +If the request is denied: -```python -from openai import OpenAI +- Confirm the provider is attached with `openshell sandbox provider list`. +- Confirm the caller binary and native endpoint appear in + `openshell policy get --full`. +- If the provider was attached after the process started, launch a new process. +- If the sandbox uses a gateway global policy override, add the native endpoint + there because a global override suppresses provider-derived policy layers. +- Inspect logs for `credential_endpoint_mismatch`. That error means policy + admitted the request but the provider profile did not authorize its + credential at the requested endpoint. -client = OpenAI(base_url="https://inference.local/v1", api_key="unused") +## Migrate from Managed Inference Routes -response = client.chat.completions.create( - model="anything", - messages=[{"role": "user", "content": "Hello"}], -) -``` +OpenShell removed the workspace-global managed inference route and the +`openshell inference` commands. Upgrades remove stored route records. Provider +records, provider refresh configuration, and existing sandbox attachments are +preserved. Built-in profiles remain available during the transition, but new +setups should import an explicit profile under a new ID. - - +The old route cannot be converted automatically. It applied one provider and +model to every sandbox in a workspace, while provider attachments intentionally +grant access to selected sandboxes. OpenShell cannot infer which sandboxes +should receive that authority. -```python -import anthropic +### Now -client = anthropic.Anthropic( - base_url="https://inference.local", - api_key="unused", -) +Earlier releases configured one shared route: -message = client.messages.create( - model="anything", - max_tokens=1024, - messages=[{"role": "user", "content": "Hello"}], -) +```shell +openshell provider create --name nvidia-prod --type nvidia --from-existing +openshell inference set \ + --provider nvidia-prod \ + --model nvidia/nemotron-3-nano-30b-a3b \ + --timeout 300 ``` - - - -Use `inference.local` when inference should stay private and credentials should not be exposed inside the sandbox. External providers reached directly belong in `network_policies` instead. +Clients called `https://inference.local`, supplied a placeholder key and model, +and relied on OpenShell to rewrite the request. -When the upstream runs on the same machine as the gateway, bind it to `0.0.0.0` and point the provider at `host.openshell.internal` or the host's LAN IP. `127.0.0.1` and `localhost` usually fail because the request originates from the gateway or sandbox runtime, not from your shell. +### After -If the gateway runs on a remote host or behind a cloud deployment, `host.openshell.internal` points to that remote machine, not to your laptop. A locally running Ollama or vLLM process is not reachable from a remote gateway unless you add your own tunnel or shared network path. - -## Verify from a Sandbox - -`openshell inference set` and `openshell inference update` verify the resolved upstream endpoint by default before saving the configuration. If the endpoint is not live yet, retry with `--no-verify` to persist the route without the probe. - -To confirm end-to-end connectivity from a sandbox, run: +Export the old provider type's profile, edit its ID and access contract, import +it, and create a replacement provider from the original credential source: ```shell -curl https://inference.local/v1/responses \ - -H "Content-Type: application/json" \ - -d '{ - "instructions": "You are a helpful assistant.", - "input": "Hello!" - }' +openshell provider profile export nvidia -o yaml > nvidia-native.yaml +# Edit id, display_name, endpoints, and binaries in nvidia-native.yaml. +openshell provider profile lint -f nvidia-native.yaml +openshell provider profile import -f nvidia-native.yaml +openshell provider create \ + --name nvidia-native-prod \ + --type nvidia-native \ + --from-existing +openshell sandbox provider attach inference-demo nvidia-native-prod ``` -A successful response confirms the privacy router can reach the configured backend and the model is serving requests. - -- Gateway-scoped: Every sandbox using the active gateway sees the same `inference.local` backend. -- HTTPS only: `inference.local` is intercepted only for HTTPS traffic. -- Hot reload: Provider, model, and timeout changes are picked up by running sandboxes within about 5 seconds by default. No sandbox recreation is required. +OpenShell does not export stored credential values or change a provider's +profile type in place. If the original credential source is unavailable, the +compatibility built-in lets you attach the preserved provider while you arrange +credential rotation into the replacement provider. + +Update each workload to: + +1. Call the provider's native endpoint. +2. Read the credential variable declared by its profile. +3. Send the real provider model identifier. +4. Configure request timeouts in the client. +5. Use the provider's native request format. + +Launch a new process after attachment and verify a native request before +upgrading production workloads. Code that still calls `inference.local` fails +DNS resolution because OpenShell no longer resolves or trusts that virtual +host. + +### Migration Checklist + +Before upgrading, record the old provider, model, and timeout with the previous +release's `openshell inference get`. Identify actual consumers, export and edit +the source profile, import it under a new ID, and create the replacement +provider from the original credential source. Attach it only to those +sandboxes, migrate their clients, and test the native path while the old route +is still available. Delete the old route on the previous release to expose +missed consumers. After upgrading the gateway, delete and recreate every +pre-upgrade sandbox so no old supervisor, DNS entry, trust material, or cached +route survives. Reapply only the provider attachments each replacement +sandbox needs. + +Special cases require additional work: + +- A provider with an alternate `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` needs + an endpoint-bearing custom profile for that host. +- Host-local services need `host.openshell.internal` or a reachable LAN/service + hostname, not `127.0.0.1` or `localhost`. +- Google Vertex AI clients must use the native Vertex endpoint and + authentication behavior. See [Google Vertex AI](/providers/google-vertex-ai). +- A bridge-fronted AWS Bedrock deployment needs a custom profile that declares + the bridge endpoint and allowed client binaries. + +## Security Differences + +Provider attachment preserves credential non-disclosure: workloads receive +opaque placeholders, and the proxy substitutes a real credential only after +network policy and endpoint binding both pass. Native requests are no longer +filtered or rewritten by a model-specific router. The provider profile's L7 +rules therefore define the allowed API surface, and the workload controls +headers, model selection, request shape, streaming, and timeout behavior. ## Next Steps -Explore related topics: - -- To follow a complete Ollama-based local setup, refer to [Inference Ollama](/get-started/tutorials/inference-ollama). -- To follow a complete LM Studio-based local setup, refer to [Local Inference LM Studio](/get-started/tutorials/local-inference-lmstudio). -- To control external endpoints, refer to [Policies](/sandboxes/policies). -- To manage provider records, refer to [Providers](/sandboxes/manage-providers). +- [Profiles](/providers/profiles) +- [Providers](/sandboxes/manage-providers) +- [Customize Sandbox Policies](/sandboxes/policies) +- [Google Vertex AI](/providers/google-vertex-ai) diff --git a/docs/sandboxes/manage-gateways.mdx b/docs/sandboxes/manage-gateways.mdx index 03e4bdfaa1..731fd52c12 100644 --- a/docs/sandboxes/manage-gateways.mdx +++ b/docs/sandboxes/manage-gateways.mdx @@ -15,7 +15,7 @@ The gateway is responsible for: - Provisioning and managing sandboxes, including creation, deletion, and status monitoring. - Storing provider credentials and delivering them to sandboxes at startup. - Delivering network and filesystem policies to sandboxes. Policy enforcement itself happens inside each sandbox through the proxy, OPA, Landlock, and seccomp. -- Managing inference configuration and serving inference bundles so sandboxes can route requests to the correct backend. +- Resolving endpoint-bound provider environments for the providers attached to each sandbox. - Providing the SSH tunnel endpoint so you can connect to sandboxes without exposing them directly. OpenShell separates gateway access from the compute driver that runs sandboxes. Use [Installation](/about/installation) to install OpenShell, choose a compute driver, and start a gateway. This page covers working with gateway entries after a gateway exists. diff --git a/docs/sandboxes/manage-providers.mdx b/docs/sandboxes/manage-providers.mdx index 6146ab8d39..895d6761ca 100644 --- a/docs/sandboxes/manage-providers.mdx +++ b/docs/sandboxes/manage-providers.mdx @@ -422,14 +422,14 @@ The following provider types are supported. | Type | Environment Variables Injected | Typical Use | |---|---|---| | `anthropic` | `ANTHROPIC_API_KEY` | Anthropic API | -| `aws-bedrock` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION` | AWS Bedrock InvokeModel via a translating bridge. Today the router does not inject any auth header; the configured `BEDROCK_BASE_URL` upstream is expected to handle auth itself. Refer to [Inference Routing](/sandboxes/inference-routing). | +| `aws-bedrock` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION` | Declarative Bedrock credential shape. The built-in profile does not grant a bridge or AWS endpoint; use an endpoint-bearing profile for direct access. | | `claude` | `ANTHROPIC_API_KEY`, `CLAUDE_API_KEY` | Claude Code, Anthropic API | | `codex` | `OPENAI_API_KEY` | OpenAI Codex | | `copilot` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, `GITHUB_TOKEN` | GitHub Copilot CLI | | `deepinfra` | `DEEPINFRA_API_KEY` | DeepInfra inference API | | `github` | `GITHUB_TOKEN`, `GH_TOKEN` | GitHub API and `gh` CLI. Refer to [GitHub Sandbox](/get-started/tutorials/github-sandbox). | | `nvidia` | `NVIDIA_API_KEY` | NVIDIA API Catalog | -| `openai` | `OPENAI_API_KEY` | OpenAI API. An `OPENAI_BASE_URL` override remains available for gateway inference routing but does not authorize the fixed public OpenAI endpoint for sandbox traffic. Refer to [Inference Routing](/sandboxes/inference-routing). | +| `openai` | `OPENAI_API_KEY` | Public OpenAI API. Use a custom endpoint-bearing profile for another OpenAI-compatible host. | `ANTHROPIC_API_KEY` is an API key from [console.anthropic.com](https://console.anthropic.com), not a subscription token. Subscription users must generate a separate API key from the Anthropic Console. @@ -442,24 +442,26 @@ ID as the provider `--type`. -## Supported Inference Providers - -The following providers have been tested with `inference.local`. Any provider that exposes an OpenAI-compatible API works with the `openai` routing type. Set `--config OPENAI_BASE_URL` to the provider's base URL and `--credential OPENAI_API_KEY` to your API key. A base URL override makes the provider route-only: it does not contribute the built-in public OpenAI endpoint to sandbox policy or make the key substitutable there. Import an endpoint-bearing custom profile when a sandbox needs direct access to the alternate upstream. - -| Provider | Name | Type | Base URL | API Key Variable | -|---|---|---|---|---| -| AWS Bedrock (via bridge) | `bedrock-bridge` | `aws-bedrock` | Operator-supplied `BEDROCK_BASE_URL` | None at router level (bridge holds creds) | -| NVIDIA API Catalog | `nvidia-prod` | `nvidia` | `https://integrate.api.nvidia.com/v1` | `NVIDIA_API_KEY` | -| Anthropic | `anthropic-prod` | `anthropic` | `https://api.anthropic.com` | `ANTHROPIC_API_KEY` | -| Google Vertex AI | `vertex-prod` | `google-vertex-ai` | Regional, global, or multi-region Vertex endpoint | `GOOGLE_VERTEX_AI_TOKEN` or `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` | -| Baseten | `baseten` | `openai` | `https://inference.baseten.co/v1` | `OPENAI_API_KEY` | -| Bitdeer AI | `bitdeer` | `openai` | `https://api-inference.bitdeer.ai/v1` | `OPENAI_API_KEY` | -| DeepInfra | `deepinfra` | `deepinfra` | `https://api.deepinfra.com/v1/openai` | `DEEPINFRA_API_KEY` | -| Groq | `groq` | `openai` | `https://api.groq.com/openai/v1` | `OPENAI_API_KEY` | -| Ollama (local) | `ollama` | `openai` | `http://host.openshell.internal:11434/v1` | `OPENAI_API_KEY` | -| LM Studio (local) | `lmstudio` | `openai` | `http://host.openshell.internal:1234/v1` | `OPENAI_API_KEY` | - -Refer to your provider's documentation for the correct base URL, available models, and API key setup. For the Vertex-specific auth flows and config keys, refer to [Google Vertex AI](/providers/google-vertex-ai). To configure inference routing, refer to [Inference Routing](/sandboxes/inference-routing). +## Inference Provider Access + +Attach an inference provider to each sandbox that needs it and configure the +workload to call the provider's native endpoint. The workload, not OpenShell, +selects the model and request timeout. + +| Provider | Built-in profile | Native base URL | Credential variable | +|---|---|---|---| +| OpenAI | `openai` | `https://api.openai.com/v1` | `OPENAI_API_KEY` | +| Anthropic | `anthropic` | `https://api.anthropic.com` | `ANTHROPIC_API_KEY` | +| NVIDIA API Catalog | `nvidia` | `https://integrate.api.nvidia.com/v1` | `NVIDIA_API_KEY` | +| DeepInfra | `deepinfra` | `https://api.deepinfra.com/v1/openai` | `DEEPINFRA_API_KEY` | +| Google Vertex AI | `google-vertex-ai` | Region and model dependent | `GOOGLE_VERTEX_AI_TOKEN` or `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` | + +An OpenAI-compatible protocol does not make the built-in `openai` profile safe +for an arbitrary host. Baseten, Bitdeer, Groq, Ollama, LM Studio, self-hosted +NIM, and other alternate endpoints need a custom profile that declares the +actual host, port, credential, and allowed binaries. See +[Provider-backed Inference](/sandboxes/inference-routing) for complete examples +and migration guidance. ## Next Steps diff --git a/docs/sandboxes/manage-sandboxes.mdx b/docs/sandboxes/manage-sandboxes.mdx index 31c7182e5f..b729ccfafa 100644 --- a/docs/sandboxes/manage-sandboxes.mdx +++ b/docs/sandboxes/manage-sandboxes.mdx @@ -495,7 +495,7 @@ OpenShell Terminal combines sandbox status and live logs in a single real-time d openshell term ``` -Use the terminal to spot blocked connections marked `action=deny` and inference-related proxy activity. If a connection is blocked unexpectedly, add the host to your network policy. Refer to [Policies](/sandboxes/policies) for the workflow. +Use the terminal to spot blocked connections marked `action=deny` and provider-related proxy activity. If a connection is blocked unexpectedly, add the host to your network policy or update the attached provider profile. Refer to [Policies](/sandboxes/policies) for the workflow. The dashboard has three panels stacked vertically: Gateways, Providers (or Global Settings), and Sandboxes. Navigate within a panel with `Up`/`Down` or `j`/`k`. At a list boundary the cursor overflows into the adjacent panel, skipping empty panels. Use `Tab`/`Shift+Tab` to cycle panels directly. Press `h`/`l` or `Left`/`Right` in the middle panel to switch between the Providers and Global Settings tabs. diff --git a/docs/sandboxes/manage-workspaces.mdx b/docs/sandboxes/manage-workspaces.mdx index 86da25fbb9..2212eb93e7 100644 --- a/docs/sandboxes/manage-workspaces.mdx +++ b/docs/sandboxes/manage-workspaces.mdx @@ -9,8 +9,8 @@ position: 3 --- An OpenShell workspace is an access and resource isolation boundary. Sandboxes, -sandbox workload templates, providers, services, policies, settings, and -inference routes belong to a workspace and are not visible to members of other +sandbox workload templates, providers, provider profiles, services, policies, +and settings belong to a workspace and are not visible to members of other workspaces. The CLI targets the `default` workspace unless you set `--workspace` or @@ -193,8 +193,8 @@ openshell workspace delete team-ml A custom workspace must not contain sandboxes, sandbox workload templates, providers, provider profiles, services, SSH sessions, settings, policies, draft policy chunks, or credential refresh state. Remove those resources before -retrying deletion. OpenShell removes membership records and inference routes as -part of successful workspace deletion. +retrying deletion. OpenShell removes membership records as part of successful +workspace deletion. ## Next Steps diff --git a/docs/sandboxes/policies.mdx b/docs/sandboxes/policies.mdx index c5a34db14f..451de9f0d7 100644 --- a/docs/sandboxes/policies.mdx +++ b/docs/sandboxes/policies.mdx @@ -69,7 +69,7 @@ When a hot reload changes rules, the supervisor publishes a new policy generatio | `filesystem_policy` | Static | Controls which directories the agent can access on disk. Paths are split into `read_only` and `read_write` lists. Any path not listed in either list is inaccessible. Set `include_workdir: true` to automatically add the agent's working directory to `read_write`. [Landlock LSM](https://docs.kernel.org/security/landlock.html) enforces these restrictions at the kernel level. | | `landlock` | Static | Configures Landlock LSM enforcement behavior. Set `compatibility` to `best_effort` (skip individual inaccessible paths while applying remaining rules) or `hard_requirement` (fail if any path is inaccessible or the required kernel ABI is unavailable). Refer to the [Policy Schema Reference](/reference/policy-schema#landlock) for the full behavior table. | | `process` | Static | Optionally overrides the OS-level identity for the agent process. Explicit values must be `sandbox` or numeric UID/GID values from `1` through `4294967294`; root and the invalid identity sentinel are rejected. Docker and Podman may use named identities through per-field OCI `USER` fallback; Kubernetes uses its platform-selected numeric identity. The agent also runs with seccomp filters that block dangerous system calls. | -| `network_policies` | Dynamic | Controls network access for ordinary outbound traffic from the sandbox. Each block has a name, a list of endpoints (host, port, protocol, and optional rules), and a list of binaries allowed to use those endpoints.
Every outbound connection except `https://inference.local` passes through the network supervisor, which queries the [policy engine](/about/how-it-works#core-components) with the destination and calling binary. A connection is allowed only when both match an entry in the same policy block.
For endpoints with `protocol: rest`, the proxy auto-detects TLS and terminates it so each HTTP request can be checked against that endpoint's `rules` (method and path). For endpoints with `protocol: websocket`, the proxy validates the RFC 6455 upgrade and evaluates `GET` rules for the handshake plus either `WEBSOCKET_TEXT` rules for raw client text messages or GraphQL operation rules for GraphQL-over-WebSocket messages. Set `websocket_credential_rewrite: true` only when a WebSocket or REST compatibility endpoint must keep placeholder credentials in sandbox-owned text frames and resolve them at the OpenShell relay boundary.
Endpoints with `protocol: tcp` allow ordinary DNS resolution and native TCP connections without inspecting payloads. Endpoints without `protocol` retain L4 passthrough through an explicit proxy.
If no endpoint matches, the connection is denied. Configure managed inference separately through [Inference Routing](/sandboxes/inference-routing). | +| `network_policies` | Dynamic | Controls outbound traffic from the sandbox, including native model-provider endpoints. Each block has a name, a list of endpoints (host, port, protocol, and optional rules), and a list of binaries allowed to use those endpoints.
Every outbound connection passes through the network supervisor, which queries the [policy engine](/about/how-it-works#core-components) with the destination and calling binary. A connection is allowed only when both match an entry in the same policy block. Attached provider profiles can contribute endpoint and binary entries to the effective policy.
For endpoints with `protocol: rest`, the proxy auto-detects TLS and terminates it so each HTTP request can be checked against that endpoint's `rules` (method and path). For endpoints with `protocol: websocket`, the proxy validates the RFC 6455 upgrade and evaluates `GET` rules for the handshake plus either `WEBSOCKET_TEXT` rules for raw client text messages or GraphQL operation rules for GraphQL-over-WebSocket messages. Set `websocket_credential_rewrite: true` only when a WebSocket or REST compatibility endpoint must keep placeholder credentials in sandbox-owned text frames and resolve them at the OpenShell relay boundary.
Endpoints with `protocol: tcp` allow ordinary DNS resolution and native TCP connections without inspecting payloads. Endpoints without `protocol` retain L4 passthrough through an explicit proxy.
If no endpoint matches, the connection is denied. | | `network_middlewares` | Dynamic | Declares keyed HTTP and WebSocket middleware configs. After network and L7 policy admit a request or upgrade, OpenShell matches each config's host selectors independently and runs matching entries by their unique ascending `order` before credential injection. WebSocket-capable entries continue on complete client text messages. | ## Supervisor Middleware diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index 1c541f8f8e..63aaf184a4 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -9,7 +9,7 @@ keywords: "Generative AI, Cybersecurity, Security, Policy, Sandbox, Landlock, Se position: 1 --- -OpenShell enforces sandbox security across four layers: network, filesystem, process, and inference. +OpenShell enforces sandbox security across four layers: network, filesystem, process, and provider credentials. This page documents every configurable control, its default, what it protects, and the risk of relaxing it. For the full policy YAML schema, refer to the [Policy Schema](/reference/policy-schema). @@ -31,7 +31,7 @@ You can update dynamic controls on a running sandbox with `openshell policy upda | Network | Unauthorized outbound connections and data exfiltration. | CONNECT proxy + OPA policy engine | Yes. Use `openshell policy update`, `openshell policy set`, or operator approval in the TUI. | | Filesystem | System binary tampering, credential theft, config manipulation. | Landlock LSM (kernel level) | No. Requires sandbox re-creation. | | Process | Privilege escalation, fork bombs, dangerous syscalls. | Seccomp BPF + privilege drop (`setuid`/`setgid`) | No. Requires sandbox re-creation. | -| Inference | Credential exposure, unauthorized model access. | Proxy intercept of `inference.local` | Yes. Use `openshell inference set`. | +| Provider credentials | Credential exposure and unauthorized service access. | Network policy plus profile endpoint binding and proxy substitution | Yes. Attach or detach providers and update dynamic policy. | ## Network Controls @@ -233,22 +233,19 @@ This ordering is intentional: named network-namespace setup still relies on priv 6. Landlock filesystem restrictions. 7. Runtime seccomp socket domain and syscall filters. -## Inference Controls +## Provider-backed Inference Controls -OpenShell routes all inference traffic through the gateway to isolate provider credentials from the sandbox. - -### Routed Inference through `inference.local` - -The proxy intercepts HTTPS CONNECT requests to `inference.local` and routes matching inference API requests through the sandbox-local router. -The agent never receives the provider API key. +Inference providers use the same endpoint-bound credential mechanism as other +providers. Attach a profile-backed provider only to sandboxes that need it. +The agent receives an opaque placeholder, not the real API key or access token. | Aspect | Detail | |---|---| -| Default | Always active. The proxy handles `inference.local` before OPA policy evaluation. The gateway injects credentials on the host side. | -| Keep-alive isolation | If a sandbox reuses a keep-alive connection that previously carried a routed inference request for a subsequent non-inference request, the proxy denies the non-inference request with `connection not allowed by policy` and closes the connection. This prevents agents from reusing an inference-authorized connection for other destinations. | -| What you can change | Configure inference routes with `openshell inference set`. | -| Risk if bypassed | If an inference provider's host is added directly to `network_policies`, the agent could reach it with a stolen or hardcoded key, bypassing credential isolation. | -| Recommendation | Do not add inference provider hosts to `network_policies`. Use OpenShell inference routing instead. | +| Default | No model-provider access. A sandbox must attach a provider and policy must admit the native endpoint and calling binary. | +| Credential boundary | The placeholder resolves only for hosts, ports, and paths authorized by the provider profile. | +| What you can change | Attach or detach providers and adjust the profile or sandbox policy. Model and timeout configuration remain in the native client. | +| Risk if relaxed | Broad endpoint paths or binary globs can expose more of a provider's API than the workload needs. `allow_uninspected_credentials` removes L7 visibility and should be exceptional. | +| Recommendation | Use narrow endpoint paths and binaries, inspect the effective policy, and keep credentials in provider records rather than workload environment configuration. | ## Gateway Security diff --git a/e2e/python/conftest.py b/e2e/python/conftest.py index 9ae1e62919..502f2540e1 100644 --- a/e2e/python/conftest.py +++ b/e2e/python/conftest.py @@ -11,7 +11,7 @@ import grpc import pytest -from openshell import InferenceRouteClient, Sandbox, SandboxClient, WorkspaceClient +from openshell import Sandbox, SandboxClient, WorkspaceClient if TYPE_CHECKING: from collections.abc import Callable, Iterator @@ -101,11 +101,6 @@ def _create(*, spec: object | None = None, delete_on_exit: bool = True) -> Sandb return _create -@pytest.fixture(scope="session") -def inference_client(sandbox_client: SandboxClient) -> InferenceRouteClient: - return InferenceRouteClient.from_sandbox_client(sandbox_client) - - @pytest.fixture(scope="session") def workspace_client(sandbox_client: SandboxClient) -> WorkspaceClient: return WorkspaceClient.from_sandbox_client(sandbox_client) diff --git a/e2e/python/oidc/workspace_authz_test.py b/e2e/python/oidc/workspace_authz_test.py index 29bc03bbb5..9a904810cb 100644 --- a/e2e/python/oidc/workspace_authz_test.py +++ b/e2e/python/oidc/workspace_authz_test.py @@ -26,13 +26,11 @@ from openshell._proto import ( datamodel_pb2, - inference_pb2, - inference_pb2_grpc, openshell_pb2, openshell_pb2_grpc, ) -from .helpers import extract_sub, get_token, grpc_channel, stub_with_token +from .helpers import extract_sub, get_token, stub_with_token WS = "e2e-authz-test" @@ -500,28 +498,6 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: metadata=m, ), ), - # ── Inference domain ── - ( - "SetInferenceRoute", - lambda _s, m: _inference_stub().SetInferenceRoute( - inference_pb2.SetInferenceRouteRequest( - provider_name="nonexistent", workspace=WS - ), - metadata=m, - ), - ), - ( - "GetInferenceRoute", - lambda _s, m: _inference_stub().GetInferenceRoute( - inference_pb2.GetInferenceRouteRequest(workspace=WS), metadata=m - ), - ), - ( - "DeleteInferenceRoute", - lambda _s, m: _inference_stub().DeleteInferenceRoute( - inference_pb2.DeleteInferenceRouteRequest(workspace=WS), metadata=m - ), - ), ] @@ -600,16 +576,6 @@ def _global_policy_read_rpcs() -> list[tuple[str, Callable]]: ] -_cached_inference_stub: inference_pb2_grpc.InferenceStub | None = None - - -def _inference_stub() -> inference_pb2_grpc.InferenceStub: - global _cached_inference_stub - if _cached_inference_stub is None: - _cached_inference_stub = inference_pb2_grpc.InferenceStub(grpc_channel()) - return _cached_inference_stub - - # ── Test class ─────────────────────────────────────────────────────────── diff --git a/e2e/python/test_inference_routing.py b/e2e/python/test_inference_routing.py deleted file mode 100644 index 0e971f5b89..0000000000 --- a/e2e/python/test_inference_routing.py +++ /dev/null @@ -1,375 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""E2E tests for explicit inference routing via `inference.local`. - -In the new model, sandbox traffic is routed only when the request targets -`inference.local`. There is no implicit catch-all interception for arbitrary -hosts like `api.openai.com`. -""" - -from __future__ import annotations - -import fcntl -from contextlib import contextmanager -from typing import TYPE_CHECKING - -import grpc -import pytest - -from openshell._proto import datamodel_pb2, openshell_pb2, sandbox_pb2 - -if TYPE_CHECKING: - from collections.abc import Callable, Iterator - - from openshell import ( - InferenceRouteClient, - InferenceRouteConfig, - Sandbox, - SandboxClient, - ) - - -_BASE_FILESYSTEM = sandbox_pb2.FilesystemPolicy( - include_workdir=True, - read_only=["/usr", "/lib", "/etc", "/app", "/var/log", "/proc", "/dev/urandom"], - read_write=["/sandbox", "/tmp"], -) -_BASE_LANDLOCK = sandbox_pb2.LandlockPolicy(compatibility="best_effort") -_BASE_PROCESS = sandbox_pb2.ProcessPolicy(run_as_user="sandbox", run_as_group="sandbox") - -pytestmark = pytest.mark.xdist_group("inference-routing") - -_MANAGED_OPENAI_MODEL_ID = "mock/e2e-openai-model" -_MANAGED_OPENAI_PROVIDER_NAME = "e2e-managed-openai" -_INFERENCE_CONFIG_LOCK = "/tmp/openshell-e2e-inference-config.lock" - - -def _baseline_policy() -> sandbox_pb2.SandboxPolicy: - return sandbox_pb2.SandboxPolicy( - version=1, - filesystem=_BASE_FILESYSTEM, - landlock=_BASE_LANDLOCK, - process=_BASE_PROCESS, - ) - - -def _upsert_managed_inference( - inference_client: InferenceRouteClient, - sandbox_client: SandboxClient, - *, - provider_name: str, - provider_type: str, - credential_key: str, - base_url_key: str, - model_id: str, - base_url: str, -) -> None: - provider = datamodel_pb2.Provider( - metadata=datamodel_pb2.ObjectMeta(name=provider_name), - type=provider_type, - credentials={credential_key: "mock"}, - config={ - base_url_key: base_url, - }, - ) - timeout = sandbox_client._timeout - - for _ in range(5): - try: - sandbox_client._stub.UpdateProvider( - openshell_pb2.UpdateProviderRequest( - provider=provider, workspace="default" - ), - timeout=timeout, - ) - break - except grpc.RpcError as exc: - if exc.code() != grpc.StatusCode.NOT_FOUND: - raise - - try: - sandbox_client._stub.CreateProvider( - openshell_pb2.CreateProviderRequest( - provider=provider, workspace="default" - ), - timeout=timeout, - ) - break - except grpc.RpcError as create_exc: - if create_exc.code() == grpc.StatusCode.ALREADY_EXISTS: - continue - raise - else: - raise RuntimeError("failed to upsert managed e2e provider after retries") - - inference_client.set_route( - workspace="default", - provider_name=provider_name, - model_id=model_id, - ) - - -def _current_route( - inference_client: InferenceRouteClient, -) -> InferenceRouteConfig | None: - try: - return inference_client.get_route(workspace="default") - except grpc.RpcError as exc: - if exc.code() == grpc.StatusCode.NOT_FOUND: - return None - raise - - -def _restore_route( - inference_client: InferenceRouteClient, - previous: InferenceRouteConfig | None, -) -> None: - if previous is None: - return - - inference_client.set_route( - workspace="default", - provider_name=previous.provider_name, - model_id=previous.model_id, - # Teardown restores prior shared state as-is, even if the previous - # route is intentionally unreachable or no longer verifiable. - no_verify=True, - ) - - -@contextmanager -def _cluster_config_lock() -> Iterator[None]: - with open(_INFERENCE_CONFIG_LOCK, "a+", encoding="utf-8") as lock_file: - fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX) - try: - yield - finally: - fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN) - - -@pytest.fixture -def managed_openai_route( - inference_client: InferenceRouteClient, - sandbox_client: SandboxClient, -) -> Iterator[str]: - with _cluster_config_lock(): - previous = _current_route(inference_client) - _upsert_managed_inference( - inference_client, - sandbox_client, - provider_name=_MANAGED_OPENAI_PROVIDER_NAME, - provider_type="openai", - credential_key="OPENAI_API_KEY", - base_url_key="OPENAI_BASE_URL", - model_id=_MANAGED_OPENAI_MODEL_ID, - base_url="mock://e2e-managed-openai", - ) - try: - yield _MANAGED_OPENAI_MODEL_ID - finally: - _restore_route(inference_client, previous) - - -def test_model_discovery_call_routed_to_backend( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Model discovery endpoint is treated as an inference protocol.""" - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_models() -> str: - import ssl - import urllib.request - - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - req = urllib.request.Request("https://inference.local/v1/models", method="GET") - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_models, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert "Hello from openshell mock backend" in output - assert managed_openai_route in output - - -def test_inference_call_routed_to_backend( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """OpenAI chat request to `inference.local` is intercepted and routed.""" - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_chat_completions() -> str: - import json - import ssl - import urllib.request - - body = json.dumps( - { - "model": "test-model", - "messages": [{"role": "user", "content": "hello"}], - } - ).encode() - - req = urllib.request.Request( - "https://inference.local/v1/chat/completions", - data=body, - headers={ - "Content-Type": "application/json", - "Authorization": "Bearer dummy-key", - }, - method="POST", - ) - # The proxy will TLS-terminate, so we need to accept its cert. - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_chat_completions, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert "Hello from openshell mock backend" in output - assert managed_openai_route in output - - -def test_non_inference_request_denied( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Non-inference path on `inference.local` is denied with 403.""" - _ = managed_openai_route - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def make_non_inference_request() -> str: - import ssl - import urllib.error - import urllib.request - - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - try: - req = urllib.request.Request("https://inference.local/v1/not-inference") - urllib.request.urlopen(req, timeout=10, context=ctx) - return "unexpected_success" - except urllib.error.HTTPError as e: - return f"http_error_{e.code}" - except Exception as e: - return f"error: {e}" - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(make_non_inference_request, timeout_seconds=30) - assert result.exit_code == 0, f"stderr: {result.stderr}" - assert result.stdout.strip() == "http_error_403" - - -def test_unsupported_protocol_returns_400( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Protocol mismatch returns 400 when no compatible route exists.""" - _ = managed_openai_route - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_anthropic_messages() -> str: - import json - import ssl - import urllib.error - import urllib.request - - body = json.dumps( - { - "model": "claude-test", - "max_tokens": 64, - "messages": [{"role": "user", "content": "hello"}], - } - ).encode() - - req = urllib.request.Request( - "https://inference.local/v1/messages", - data=body, - headers={ - "Content-Type": "application/json", - "x-api-key": "dummy-key", - "anthropic-version": "2023-06-01", - }, - method="POST", - ) - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - try: - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - except urllib.error.HTTPError as exc: - return ( - f"http_error_{exc.code}:{exc.read().decode('utf-8', errors='replace')}" - ) - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_anthropic_messages, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert output.startswith("http_error_400"), output - assert "no compatible inference route" in output - - -def test_non_inference_host_is_not_intercepted( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Requests to non-`inference.local` hosts do not get inference routing.""" - _ = managed_openai_route - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_external_openai_endpoint() -> str: - import json - import ssl - import urllib.error - import urllib.request - - body = json.dumps( - { - "model": "test-model", - "messages": [{"role": "user", "content": "hello"}], - } - ).encode() - - req = urllib.request.Request( - "https://api.openai.com/v1/chat/completions", - data=body, - headers={ - "Content-Type": "application/json", - "Authorization": "Bearer dummy", - }, - method="POST", - ) - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - try: - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - except urllib.error.URLError as exc: - return f"url_error:{exc}" - except Exception as exc: - return f"error:{type(exc).__name__}:{exc}" - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_external_openai_endpoint, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert "Tunnel connection failed: 403 Forbidden" in output diff --git a/e2e/python/test_sandbox_providers.py b/e2e/python/test_sandbox_providers.py index a924168a1b..3ef4276f13 100644 --- a/e2e/python/test_sandbox_providers.py +++ b/e2e/python/test_sandbox_providers.py @@ -11,6 +11,11 @@ from __future__ import annotations +import json +import socket +import subprocess +import sys +import textwrap import time from contextlib import contextmanager from typing import TYPE_CHECKING @@ -70,6 +75,7 @@ def provider( name: str, provider_type: str, credentials: dict[str, str], + profile_workspace: str = "", ) -> Iterator[str]: """Create a provider for the duration of the block, then delete it.""" _delete_provider(stub, name) @@ -79,6 +85,7 @@ def provider( metadata=datamodel_pb2.ObjectMeta(name=name), type=provider_type, credentials=credentials, + profile_workspace=profile_workspace, ) ) ) @@ -99,6 +106,196 @@ def _delete_provider(stub: object, name: str) -> None: raise +def _delete_provider_profile(stub: object, profile_id: str) -> None: + """Delete a provider profile, ignoring not-found errors.""" + try: + stub.DeleteProviderProfile( + openshell_pb2.DeleteProviderProfileRequest( + id=profile_id, + workspace="default", + ) + ) + except grpc.RpcError as exc: + if hasattr(exc, "code") and exc.code() == grpc.StatusCode.NOT_FOUND: + pass + else: + raise + + +@contextmanager +def imported_provider_profile( + stub: object, + *, + profile: openshell_pb2.ProviderProfile, + source: str, +) -> Iterator[str]: + """Import a workspace-scoped provider profile for the duration of the block.""" + _delete_provider_profile(stub, profile.id) + response = stub.ImportProviderProfiles( + openshell_pb2.ImportProviderProfilesRequest( + profiles=[ + openshell_pb2.ProviderProfileImportItem( + profile=profile, + source=source, + ) + ], + workspace="default", + ) + ) + assert response.imported, f"profile import failed: {response.diagnostics!r}" + try: + yield profile.id + finally: + _delete_provider_profile(stub, profile.id) + + +def _native_inference_profile( + *, + profile_id: str, + env_var: str, + port: int, + rules: list[sandbox_pb2.L7Rule], + auth_style: str = "bearer", + header_name: str = "authorization", +) -> openshell_pb2.ProviderProfile: + return openshell_pb2.ProviderProfile( + id=profile_id, + display_name=f"{profile_id} display", + description="E2E imported inference profile fixture", + category=openshell_pb2.PROVIDER_PROFILE_CATEGORY_INFERENCE, + inference_capable=True, + credentials=[ + openshell_pb2.ProviderProfileCredential( + name="api_key", + description="API key", + env_vars=[env_var], + required=True, + auth_style=auth_style, + header_name=header_name, + ) + ], + endpoints=[ + sandbox_pb2.NetworkEndpoint( + host="host.openshell.internal", + port=port, + protocol="rest", + tls="none", + enforcement="enforce", + rules=rules, + allowed_ips=[ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16", + "fc00::/7", + ], + ) + ], + binaries=[ + sandbox_pb2.NetworkBinary(path="/usr/bin/python*"), + sandbox_pb2.NetworkBinary(path="/usr/local/bin/python*"), + sandbox_pb2.NetworkBinary(path="/sandbox/.uv/python/**/python*"), + ], + ) + + +@contextmanager +def native_endpoint_server() -> Iterator[int]: + """Start a small host-side HTTP fixture that echoes auth and request data.""" + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener: + listener.bind(("127.0.0.1", 0)) + port = listener.getsockname()[1] + + script = textwrap.dedent( + """ + import json + import sys + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + PORT = int(sys.argv[1]) + + class Handler(BaseHTTPRequestHandler): + def _reply(self): + content_length = int(self.headers.get("Content-Length", "0")) + body = self.rfile.read(content_length) if content_length else b"" + payload = { + "method": self.command, + "path": self.path, + "authorization": self.headers.get("Authorization"), + "x_api_key": self.headers.get("x-api-key"), + "body": body.decode("utf-8"), + } + encoded = json.dumps(payload).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + def do_GET(self): + self._reply() + + def do_POST(self): + self._reply() + + def log_message(self, fmt, *args): + pass + + ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever() + """ + ) + proc = subprocess.Popen( + [sys.executable, "-c", script, str(port)], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + if proc.poll() is not None: + stdout, stderr = proc.communicate(timeout=5) + raise RuntimeError( + "native endpoint fixture exited early: " + f"stdout={stdout!r} stderr={stderr!r}" + ) + try: + with socket.create_connection(("127.0.0.1", port), timeout=1): + break + except OSError: + time.sleep(0.2) + else: + proc.kill() + stdout, stderr = proc.communicate(timeout=5) + raise RuntimeError( + "native endpoint fixture did not become ready: " + f"stdout={stdout!r} stderr={stderr!r}" + ) + + try: + yield port + finally: + proc.kill() + proc.communicate(timeout=5) + + +def _proxy_connect(): + """Return a closure that sends a raw CONNECT and returns the status line.""" + + def fn(host, port): + import socket + + conn = socket.create_connection(("10.200.0.1", 3128), timeout=10) + try: + conn.sendall( + f"CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}\r\n\r\n".encode() + ) + return conn.recv(256).decode("latin1") + finally: + conn.close() + + return fn + + # =========================================================================== # Tests: placeholder visibility # =========================================================================== @@ -328,6 +525,193 @@ def wait_for_token(sb: Sandbox, expected: str) -> None: raise +def test_imported_openai_profile_allows_native_endpoint_with_attached_provider( + sandbox: Callable[..., Sandbox], + sandbox_client: SandboxClient, +) -> None: + """Imported fixture profiles should support native OpenAI-style access.""" + stub = sandbox_client._stub + profile_id = f"e2e-native-openai-{int(time.time() * 1000)}" + provider_name = f"{profile_id}-provider" + secret = "sk-native-openai-secret" + + profile = _native_inference_profile( + profile_id=profile_id, + env_var="OPENAI_API_KEY", + port=0, + rules=[ + sandbox_pb2.L7Rule( + allow=sandbox_pb2.L7Allow( + method="POST", + path="/v1/chat/completions", + ) + ) + ], + ) + + def call_native_openai(host: str, port: int) -> str: + import json + import os + import urllib.error + import urllib.request + + body = json.dumps( + { + "model": "fixture-openai-model", + "messages": [{"role": "user", "content": "hello"}], + } + ).encode() + request = urllib.request.Request( + f"http://{host}:{port}/v1/chat/completions", + data=body, + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return response.read().decode() + except urllib.error.HTTPError as exc: + raise RuntimeError( + f"native OpenAI request failed with {exc.code}: " + f"{exc.read().decode(errors='replace')}" + ) from exc + + with native_endpoint_server() as port: + profile.endpoints[0].port = port + with imported_provider_profile( + stub, + profile=profile, + source=f"{profile_id}.yaml", + ): + with provider( + stub, + name=provider_name, + provider_type=profile_id, + credentials={"OPENAI_API_KEY": secret}, + profile_workspace="default", + ) as attached_provider: + spec = datamodel_pb2.SandboxSpec( + policy=_default_policy(), + providers=[attached_provider], + ) + with sandbox(spec=spec, delete_on_exit=True) as sb: + result = sb.exec_python( + call_native_openai, + args=("host.openshell.internal", port), + timeout_seconds=60, + ) + assert result.exit_code == 0, result.stderr + payload = json.loads(result.stdout) + body = json.loads(payload["body"]) + assert payload["method"] == "POST" + assert payload["path"] == "/v1/chat/completions" + assert payload["authorization"] == f"Bearer {secret}" + assert body["model"] == "fixture-openai-model" + + +def test_imported_anthropic_profile_uses_native_endpoint_and_inference_local_is_not_privileged( + sandbox: Callable[..., Sandbox], + sandbox_client: SandboxClient, +) -> None: + """Attached imported profiles should not resurrect `inference.local` routing.""" + stub = sandbox_client._stub + profile_id = f"e2e-native-anthropic-{int(time.time() * 1000)}" + provider_name = f"{profile_id}-provider" + secret = "sk-native-anthropic-secret" + + profile = _native_inference_profile( + profile_id=profile_id, + env_var="ANTHROPIC_API_KEY", + port=0, + rules=[ + sandbox_pb2.L7Rule( + allow=sandbox_pb2.L7Allow( + method="POST", + path="/v1/messages", + ) + ) + ], + auth_style="header", + header_name="x-api-key", + ) + + def call_native_anthropic(host: str, port: int) -> str: + import json + import os + import urllib.error + import urllib.request + + body = json.dumps( + { + "model": "fixture-anthropic-model", + "messages": [{"role": "user", "content": "hello"}], + } + ).encode() + request = urllib.request.Request( + f"http://{host}:{port}/v1/messages", + data=body, + headers={ + "Content-Type": "application/json", + "x-api-key": os.environ["ANTHROPIC_API_KEY"], + "anthropic-version": "2023-06-01", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return response.read().decode() + except urllib.error.HTTPError as exc: + raise RuntimeError( + f"native Anthropic request failed with {exc.code}: " + f"{exc.read().decode(errors='replace')}" + ) from exc + + with native_endpoint_server() as port: + profile.endpoints[0].port = port + with imported_provider_profile( + stub, + profile=profile, + source=f"{profile_id}.yaml", + ): + with provider( + stub, + name=provider_name, + provider_type=profile_id, + credentials={"ANTHROPIC_API_KEY": secret}, + profile_workspace="default", + ) as attached_provider: + spec = datamodel_pb2.SandboxSpec( + policy=_default_policy(), + providers=[attached_provider], + ) + with sandbox(spec=spec, delete_on_exit=True) as sb: + native_result = sb.exec_python( + call_native_anthropic, + args=("host.openshell.internal", port), + timeout_seconds=60, + ) + assert native_result.exit_code == 0, native_result.stderr + payload = json.loads(native_result.stdout) + body = json.loads(payload["body"]) + assert payload["method"] == "POST" + assert payload["path"] == "/v1/messages" + assert payload["x_api_key"] == secret + assert body["model"] == "fixture-anthropic-model" + + denied = sb.exec_python( + _proxy_connect(), + args=("inference.local", 443), + timeout_seconds=30, + ) + assert denied.exit_code == 0, denied.stderr + status = denied.stdout.strip() + assert status.startswith("HTTP/1.1 "), status + assert " 200 " not in status, status + + # =========================================================================== # Tests: security & edge cases # =========================================================================== diff --git a/e2e/rust/tests/host_gateway_alias.rs b/e2e/rust/tests/host_gateway_alias.rs index 962e4e0f94..cf7b9e657e 100644 --- a/e2e/rust/tests/host_gateway_alias.rs +++ b/e2e/rust/tests/host_gateway_alias.rs @@ -5,7 +5,6 @@ use std::io::Write; use std::process::Stdio; -use std::sync::Mutex; use openshell_e2e::harness::binary::openshell_cmd; use openshell_e2e::harness::sandbox::SandboxGuard; @@ -15,13 +14,10 @@ use tokio::io::AsyncWriteExt; use tokio::net::TcpListener; use tokio::task::JoinHandle; -const INFERENCE_PROVIDER_NAME: &str = "e2e-host-inference"; -const INFERENCE_PROVIDER_UNREACHABLE_NAME: &str = "e2e-host-inference-unreachable"; const BINDING_PROVIDER_A_NAME: &str = "e2e-static-endpoint-binding-provider-a"; const BINDING_PROVIDER_B_NAME: &str = "e2e-static-endpoint-binding-provider-b"; const BINDING_PROFILE_A_ID: &str = "e2e-static-endpoint-binding-a"; const BINDING_PROFILE_B_ID: &str = "e2e-static-endpoint-binding-b"; -static INFERENCE_ROUTE_LOCK: Mutex<()> = Mutex::new(()); async fn run_cli(args: &[&str]) -> Result { let mut cmd = openshell_cmd(); @@ -238,16 +234,6 @@ impl Drop for HostServer { } } -async fn provider_exists(name: &str) -> bool { - let mut cmd = openshell_cmd(); - cmd.arg("provider") - .arg("get") - .arg(name) - .stdout(Stdio::null()) - .stderr(Stdio::null()); - cmd.status().await.is_ok_and(|status| status.success()) -} - async fn delete_provider(name: &str) { let mut cmd = openshell_cmd(); cmd.arg("provider") @@ -269,22 +255,6 @@ async fn delete_provider_profile(id: &str) { let _ = cmd.status().await; } -async fn create_openai_provider(name: &str, base_url: &str) -> Result { - run_cli(&[ - "provider", - "create", - "--name", - name, - "--type", - "openai", - "--credential", - "OPENAI_API_KEY=dummy", - "--config", - &format!("OPENAI_BASE_URL={base_url}"), - ]) - .await -} - fn write_policy(port: u16) -> Result { let mut file = NamedTempFile::new().map_err(|e| format!("create temp policy file: {e}"))?; let policy = format!( @@ -493,146 +463,3 @@ async fn static_provider_credentials_are_bound_to_profile_endpoints() { delete_provider_profile(BINDING_PROFILE_A_ID).await; delete_provider_profile(BINDING_PROFILE_B_ID).await; } - -#[tokio::test] -async fn sandbox_inference_local_routes_to_host_openshell_internal() { - let _inference_lock = INFERENCE_ROUTE_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - - let current_inference = run_cli(&["inference", "get"]) - .await - .expect("read current inference config"); - if !current_inference.contains("Not configured") { - eprintln!("Skipping test: existing inference config would make shared state unsafe"); - return; - } - - let server = HostServer::start( - r#"{"id":"chatcmpl-test","object":"chat.completion","created":1,"model":"host-echo","choices":[{"index":0,"message":{"role":"assistant","content":"hello-from-host"},"finish_reason":"stop"}]}"#, - ) - .await - .expect("start host inference echo server"); - - if provider_exists(INFERENCE_PROVIDER_NAME).await { - delete_provider(INFERENCE_PROVIDER_NAME).await; - } - - create_openai_provider( - INFERENCE_PROVIDER_NAME, - &format!("http://host.openshell.internal:{}/v1", server.port), - ) - .await - .expect("create host-backed OpenAI provider"); - - let inference_output = run_cli(&[ - "inference", - "set", - "--provider", - INFERENCE_PROVIDER_NAME, - "--model", - "host-echo-model", - "--no-verify", - ]) - .await - .expect("point inference.local at host-backed provider"); - - assert!( - !inference_output.contains("Validated Endpoints:"), - "did not expect local CLI verification for host-only alias:\n{inference_output}" - ); - - let guard = SandboxGuard::create(&[ - "--", - "curl", - "--silent", - "--show-error", - "--max-time", - "15", - "https://inference.local/v1/chat/completions", - "--json", - r#"{"messages":[{"role":"user","content":"hello"}]}"#, - ]) - .await - .expect("sandbox create with inference.local request"); - - assert!( - guard - .create_output - .contains("\"object\":\"chat.completion\""), - "expected sandbox to receive inference response:\n{}", - guard.create_output - ); - assert!( - guard.create_output.contains("hello-from-host"), - "expected sandbox to receive echoed inference content:\n{}", - guard.create_output - ); -} - -#[tokio::test] -async fn inference_set_supports_no_verify_for_unreachable_endpoint() { - let _inference_lock = INFERENCE_ROUTE_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - - let current_inference = run_cli(&["inference", "get"]) - .await - .expect("read current inference config"); - if !current_inference.contains("Not configured") { - eprintln!("Skipping test: existing inference config would make shared state unsafe"); - return; - } - - if provider_exists(INFERENCE_PROVIDER_UNREACHABLE_NAME).await { - delete_provider(INFERENCE_PROVIDER_UNREACHABLE_NAME).await; - } - - create_openai_provider( - INFERENCE_PROVIDER_UNREACHABLE_NAME, - "http://host.openshell.internal:9/v1", - ) - .await - .expect("create unreachable OpenAI provider"); - - let verify_err = run_cli(&[ - "inference", - "set", - "--provider", - INFERENCE_PROVIDER_UNREACHABLE_NAME, - "--model", - "host-echo-model", - ]) - .await - .expect_err("default verification should fail for unreachable endpoint"); - - assert!( - verify_err.contains("failed to verify inference endpoint"), - "expected verification failure output:\n{verify_err}" - ); - let normalized_verify_err: String = verify_err - .chars() - .filter(|c| !c.is_whitespace() && *c != '│') - .collect(); - assert!( - normalized_verify_err.contains("--no-verify"), - "expected retry hint in failure output:\n{verify_err}" - ); - - let no_verify_output = run_cli(&[ - "inference", - "set", - "--provider", - INFERENCE_PROVIDER_UNREACHABLE_NAME, - "--model", - "host-echo-model", - "--no-verify", - ]) - .await - .expect("no-verify should bypass validation"); - - assert!( - !no_verify_output.contains("Validated Endpoints:"), - "did not expect validation output when bypassing verification:\n{no_verify_output}" - ); -} diff --git a/examples/local-inference/README.md b/examples/local-inference/README.md index e1d8699b3e..9f5d8bc7cd 100644 --- a/examples/local-inference/README.md +++ b/examples/local-inference/README.md @@ -1,166 +1,69 @@ -# Inference Routing Example +# Provider-backed Inference Example -This example demonstrates OpenShell's inference interception and routing. -A sandbox process sends inference traffic to `inference.local`, and -OpenShell intercepts and reroutes it to the configured backend. - -## How It Works - -1. The sandbox process sends HTTPS traffic to `inference.local`. -2. The sandbox proxy intercepts that explicit inference endpoint locally. -3. The proxy TLS-terminates, parses the HTTP request, and detects known - inference patterns (e.g., `POST /v1/chat/completions`). -4. Matching requests are forwarded to the configured backend via the sandbox's - local router. Non-inference requests are denied. +This example calls the NVIDIA API Catalog through its native OpenAI-compatible +endpoint. OpenShell supplies endpoint-bound credentials and network policy from +an explicitly imported provider profile; the Python client owns the endpoint, +model, request shape, timeout, and streaming behavior. ## Files | File | Description | |---|---| -| `inference.py` | Python script that tests streaming and non-streaming inference through `inference.local` | -| `sandbox-policy.yaml` | Minimal sandbox policy (no network access except `inference.local`) | -| `routes.yaml` | Example YAML route file for standalone (no-cluster) mode | - -## Quick Start (NVIDIA) - -Requires a running OpenShell gateway and `NVIDIA_API_KEY` set in your shell. - -```bash -# 1. Create a provider using your NVIDIA credentials -openshell provider create --name nvidia --type nvidia --credential NVIDIA_API_KEY - -# 2. Configure inference routing -openshell inference set --provider nvidia --model meta/llama-3.1-8b-instruct - -# 3. Run the test script in a sandbox -openshell sandbox create \ - --policy examples/local-inference/sandbox-policy.yaml \ - --upload examples/local-inference/inference.py \ - -- python3 /sandbox/inference.py -``` - -Expected output (with the streaming buffering bug present): - -```text -============================================================ -NON-STREAMING REQUEST -============================================================ - model = meta/llama-3.1-8b-instruct - content = Glowing screens abide - Whirring circuits, silent mind - Tech's gentle grasp - total = 0.96s - -============================================================ -STREAMING REQUEST -============================================================ - TTFB = 0.54s - model = meta/llama-3.1-8b-instruct - content = Glowing screens abide - Code and circuits whisper - Silent digital - total = 0.54s - - ** BUG: TTFB is 99% of total time — response was buffered, not streamed ** -``` - -When streaming works correctly, TTFB should be sub-second while total time -stays the same (tokens arrive incrementally). - -## Standalone (no cluster) - -Run the sandbox binary directly with a route file — no gateway needed: - -```bash -# 1. Edit routes.yaml to point at your local LLM (e.g. LM Studio on :1234) +| `nvidia-inference.yaml` | Example profile for the endpoint, credential, and allowed Python binaries | +| `inference.py` | Native endpoint streaming and non-streaming client | +| `sandbox-policy.yaml` | Minimal policy that lets Python install the OpenAI client from PyPI | -# 2. Run the sandbox with --inference-routes -openshell-sandbox \ - --inference-routes examples/local-inference/routes.yaml \ - --policy-rules \ - --policy-data examples/local-inference/sandbox-policy.yaml \ - -- python examples/local-inference/inference.py -``` - -The sandbox loads routes from the YAML file at startup and routes inference -requests locally — no gRPC server or cluster required. - -### With a gateway +## Run the Example -#### 1. Start an OpenShell gateway +Export the built-in profile as a starting point and compare it with the example +before import. A custom profile must use a new ID; built-in IDs are reserved. -```bash -mise run gateway:docker -openshell status +```shell +openshell provider profile export nvidia -o yaml > /tmp/nvidia-profile.yaml +diff -u /tmp/nvidia-profile.yaml examples/local-inference/nvidia-inference.yaml +openshell provider profile lint -f examples/local-inference/nvidia-inference.yaml +openshell provider profile import -f examples/local-inference/nvidia-inference.yaml ``` -#### 2. Configure gateway inference - -First make sure a provider record exists for the backend you want to use: - -```bash -openshell provider list -``` +Create the provider from the local `NVIDIA_API_KEY`, then attach it to the new +sandbox: -Then configure the gateway-managed `inference.local` route: - -```bash -# Example: use an existing provider record -openshell inference set \ - --provider openai-prod \ - --model nvidia/nemotron-3-nano-30b-a3b -``` - -Verify the active config: - -```bash -openshell inference get -``` +```shell +openshell provider create \ + --name nvidia-demo \ + --type nvidia-inference \ + --from-existing -#### 3. Run the example inside a sandbox - -```bash openshell sandbox create \ - --policy examples/local-inference/sandbox-policy.yaml \ --name inference-demo \ - -- python examples/local-inference/inference.py + --provider nvidia-demo \ + --policy examples/local-inference/sandbox-policy.yaml \ + --upload examples/local-inference/inference.py \ + -- python3 /sandbox/inference.py ``` -The script targets `https://inference.local/v1` directly. OpenShell -intercepts that connection and routes it to whatever backend gateway inference -is configured to use. +The profile contributes the NVIDIA endpoint to the effective network policy and +injects an opaque `NVIDIA_API_KEY` placeholder. The proxy substitutes the real +key only for requests that match the profile endpoint. Inspect the composed +policy with: -Expected output: - -```text -model= -content=NAV_OK +```shell +openshell policy get inference-demo --full ``` -#### 4. (Optional) Interactive session +To change the endpoint or allowed client binaries, export the custom profile, +edit it, and submit its `resource_version` with `profile update`. The workload +still needs a native client configuration that matches the profile. -```bash -openshell sandbox connect inference-demo -# Inside the sandbox: -python examples/local-inference/inference.py +```shell +openshell provider profile export nvidia-inference -o yaml > nvidia-inference.yaml +# Edit nvidia-inference.yaml. +openshell provider profile lint -f nvidia-inference.yaml +openshell provider profile update nvidia-inference -f nvidia-inference.yaml ``` -#### 5. Cleanup +Delete the sandbox when finished: -```bash +```shell openshell sandbox delete inference-demo ``` - -## Customizing Routes - -Edit `routes.yaml` to change which backend endpoint/model standalone mode uses. -In gateway mode, use `openshell inference set` instead. - -## Supported Protocols - -| Pattern | Protocol | Kind | -|---|---|---| -| `POST /v1/chat/completions` | `openai_chat_completions` | Chat completion | -| `POST /v1/completions` | `openai_completions` | Text completion | -| `POST /v1/responses` | `openai_responses` | Responses API | -| `POST /v1/messages` | `anthropic_messages` | Anthropic messages | diff --git a/examples/local-inference/inference.py b/examples/local-inference/inference.py index 2d04569ba7..b0e2035be6 100644 --- a/examples/local-inference/inference.py +++ b/examples/local-inference/inference.py @@ -1,25 +1,14 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Test inference routing through both inference.local and direct endpoint access. +"""Test native NVIDIA inference with non-streaming and streaming requests. -Exercises four scenarios to verify streaming works correctly: - 1. inference.local — non-streaming - 2. inference.local — streaming - 3. Direct NVIDIA endpoint (L7 TLS intercept) — non-streaming - 4. Direct NVIDIA endpoint (L7 TLS intercept) — streaming - -The direct endpoint tests verify that the L7 REST relay path (relay_chunked / -relay_until_eof) streams responses incrementally, in contrast with the -inference.local interception path which previously buffered the entire body. +OpenShell attaches a profile-backed provider to supply endpoint-bound +``NVIDIA_API_KEY`` credentials and matching network policy. The client selects +the native endpoint, model, request shape, and timeout. Usage: - # inference.local only (no provider attached): - openshell sandbox create --policy sandbox-policy.yaml --upload inference.py \ - -- python3 /sandbox/inference.py - - # All 4 tests (attach the nvidia provider so NVIDIA_API_KEY is available): - openshell sandbox create --provider nvidia --policy sandbox-policy.yaml \ + openshell sandbox create --provider nvidia-demo --policy sandbox-policy.yaml \ --upload inference.py -- python3 /sandbox/inference.py """ @@ -113,26 +102,16 @@ def run_streaming(client: OpenAI, label: str, model: str) -> None: def main() -> None: - # --- inference.local tests (router injects auth + model) --- - local_client = OpenAI(api_key="dummy", base_url="https://inference.local/v1") - - run_non_streaming(local_client, "inference.local", model="router") - run_streaming(local_client, "inference.local", model="router") - - # --- Direct endpoint tests (L7 TLS intercept path) --- - # The API key is available when the sandbox is started with --provider nvidia. api_key = os.environ.get("NVIDIA_API_KEY") - if api_key: - direct_client = OpenAI(api_key=api_key, base_url=DIRECT_URL) - - run_non_streaming(direct_client, f"direct ({DIRECT_URL})", model=DIRECT_MODEL) - run_streaming(direct_client, f"direct ({DIRECT_URL})", model=DIRECT_MODEL) - else: - print("=" * 60) - print("SKIPPED — direct endpoint tests (NVIDIA_API_KEY not set)") - print("=" * 60) - print(" Attach the nvidia provider to enable: --provider nvidia") - print() + if not api_key: + raise SystemExit( + "NVIDIA_API_KEY is unavailable; attach the nvidia-demo provider " + "and launch a new process" + ) + + client = OpenAI(api_key=api_key, base_url=DIRECT_URL, timeout=300) + run_non_streaming(client, DIRECT_URL, model=DIRECT_MODEL) + run_streaming(client, DIRECT_URL, model=DIRECT_MODEL) if __name__ == "__main__": diff --git a/examples/local-inference/nvidia-inference.yaml b/examples/local-inference/nvidia-inference.yaml new file mode 100644 index 0000000000..526a813bb9 --- /dev/null +++ b/examples/local-inference/nvidia-inference.yaml @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +id: nvidia-inference +display_name: NVIDIA Inference +description: NVIDIA API Catalog access for native Python clients +category: inference +# Informational only. This field does not enable routing or request transformation. +inference_capable: true +credentials: + - name: api_key + description: NVIDIA API key + env_vars: [NVIDIA_API_KEY] + required: true + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: integrate.api.nvidia.com + port: 443 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/python3 + - /usr/bin/python3.13 + - /usr/local/bin/python + - /sandbox/.venv/** diff --git a/examples/local-inference/routes.yaml b/examples/local-inference/routes.yaml deleted file mode 100644 index 6219fa0a56..0000000000 --- a/examples/local-inference/routes.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# Example route configuration for standalone (no-cluster) inference routing. -# -# Usage: -# openshell-sandbox \ -# --inference-routes examples/local-inference/routes.yaml \ -# --policy-rules policy.rego \ -# --policy-data examples/local-inference/sandbox-policy.yaml \ -# -- python examples/local-inference/inference.py - -routes: - - name: inference.local - endpoint: http://localhost:1234/v1 - model: local-model - protocols: - - openai_chat_completions - # Inline API key (for local models that require one): - api_key: lm-studio - - # Or reference an environment variable instead: - # api_key_env: OPENAI_API_KEY diff --git a/examples/local-inference/sandbox-policy.yaml b/examples/local-inference/sandbox-policy.yaml index a0d7ba1f41..25a2e635b9 100644 --- a/examples/local-inference/sandbox-policy.yaml +++ b/examples/local-inference/sandbox-policy.yaml @@ -30,18 +30,3 @@ network_policies: - { host: files.pythonhosted.org, port: 443 } binaries: - path: /usr/bin/python3.13 - - # Direct access to the NVIDIA inference API (L7 TLS intercept). - # Used to verify that the L7 REST relay path streams responses correctly - # (contrast with the inference.local interception path above). - nvidia_direct: - name: NVIDIA API (L7 intercept) - endpoints: - - host: integrate.api.nvidia.com - port: 443 - protocol: rest - tls: terminate - enforcement: enforce - access: full - binaries: - - path: /usr/bin/python3.13 diff --git a/openshell.spec b/openshell.spec index ac57d29ee9..df41843f1c 100644 --- a/openshell.spec +++ b/openshell.spec @@ -43,7 +43,7 @@ Recommends: podman OpenShell provides safe, sandboxed runtimes for autonomous AI agents. It offers a CLI for managing gateway registrations, sandboxes, and providers with policy-enforced egress routing, credential proxying, and privacy-aware -LLM inference routing. +profile-backed model-provider access. # --- Gateway sub-package --- %package gateway @@ -71,7 +71,7 @@ Recommends: %{name} %description -n python3-%{name} Python SDK for OpenShell providing programmatic access to sandbox -management, agent execution, and inference routing via gRPC. +management, agent execution, and provider access via gRPC. %prep %autosetup -n %{name}-%{version} diff --git a/proto/inference.proto b/proto/inference.proto deleted file mode 100644 index a28d7149e5..0000000000 --- a/proto/inference.proto +++ /dev/null @@ -1,173 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -syntax = "proto3"; - -package openshell.inference.v1; - -import "datamodel.proto"; -import "options.proto"; - -// Inference service provides workspace-scoped inference route configuration and bundle delivery. -service Inference { - // Return the resolved inference route bundle for sandbox-local execution. - rpc GetInferenceBundle(GetInferenceBundleRequest) - returns (GetInferenceBundleResponse) { - option (openshell.options.v1.authorization) = { - auth_mode: "sandbox" - }; - } - - // Set the inference route for a workspace. - // - // This controls how requests sent to `inference.local` are routed - // for sandboxes in the specified workspace. - rpc SetInferenceRoute(SetInferenceRouteRequest) - returns (SetInferenceRouteResponse) { - option (openshell.options.v1.authorization) = { - auth_mode: "bearer" - scope: "inference:write" - workspace_role: "admin" - }; - } - - // Get the inference route for a workspace. - rpc GetInferenceRoute(GetInferenceRouteRequest) - returns (GetInferenceRouteResponse) { - option (openshell.options.v1.authorization) = { - auth_mode: "bearer" - scope: "inference:read" - workspace_role: "user" - }; - } - - // Delete an inference route from a workspace. - rpc DeleteInferenceRoute(DeleteInferenceRouteRequest) - returns (DeleteInferenceRouteResponse) { - option (openshell.options.v1.authorization) = { - auth_mode: "bearer" - scope: "inference:write" - workspace_role: "admin" - }; - } -} - -// Persisted inference route configuration. -// -// Only `provider_name` and `model_id` are stored; endpoint, protocols, -// credentials, and auth style are resolved from the provider at bundle time. -message InferenceRouteConfig { - // Provider record name backing this route. - string provider_name = 1; - // Model identifier to force on generation calls. - string model_id = 2; - // Per-route request timeout in seconds. 0 means use default (60s). - uint64 timeout_secs = 3; -} - -// Storage envelope for a workspace-scoped inference route. -message InferenceRoute { - openshell.datamodel.v1.ObjectMeta metadata = 1; - InferenceRouteConfig config = 2; - // Monotonic version incremented on every update. - uint64 version = 3; -} - -message SetInferenceRouteRequest { - // Provider record name to use for credentials + endpoint mapping. - string provider_name = 1; - // Model identifier to force on generation calls. - string model_id = 2; - // Route name to target. Empty string defaults to "inference.local" (user-facing). - // Use "sandbox-system" for the sandbox system-level inference route. - string route_name = 3; - // Verify the resolved upstream endpoint synchronously before persistence. - bool verify = 4; - // Skip synchronous endpoint validation before persistence. - bool no_verify = 5; - // Per-route request timeout in seconds. 0 means use default (60s). - uint64 timeout_secs = 6; - // Target workspace. Empty string defaults to "default". - string workspace = 7; -} - -message ValidatedEndpoint { - string url = 1; - string protocol = 2; -} - -message SetInferenceRouteResponse { - string provider_name = 1; - string model_id = 2; - uint64 version = 3; - // Route name that was configured. - string route_name = 4; - // Whether endpoint verification ran as part of this request. - bool validation_performed = 5; - // The concrete endpoints that were probed during validation, when available. - repeated ValidatedEndpoint validated_endpoints = 6; - // Per-route request timeout in seconds that was persisted. - uint64 timeout_secs = 7; - // Workspace the route was configured in. - string workspace = 8; -} - -message GetInferenceRouteRequest { - // Route name to query. Empty string defaults to "inference.local" (user-facing). - // Use "sandbox-system" for the sandbox system-level inference route. - string route_name = 1; - // Target workspace. Empty string defaults to "default". - string workspace = 2; -} - -message GetInferenceRouteResponse { - string provider_name = 1; - string model_id = 2; - uint64 version = 3; - // Route name that was queried. - string route_name = 4; - // Per-route request timeout in seconds. 0 means default (60s). - uint64 timeout_secs = 5; - // Workspace the route belongs to. - string workspace = 6; -} - -message DeleteInferenceRouteRequest { - // Route name to delete. Empty string defaults to "inference.local" (user-facing). - // Use "sandbox-system" for the sandbox system-level inference route. - string route_name = 1; - // Target workspace. Empty string defaults to "default". - string workspace = 2; -} - -message DeleteInferenceRouteResponse { - // Whether a route was actually deleted. - bool deleted = 1; -} - -message GetInferenceBundleRequest {} - -// A single resolved route ready for sandbox-local execution. -message ResolvedRoute { - string name = 1; - string base_url = 2; - repeated string protocols = 3; - string api_key = 4 [(openshell.options.v1.secret) = true]; - string model_id = 5; - string provider_type = 6; - // Per-route request timeout in seconds. 0 means use default (60s). - uint64 timeout_secs = 7; - // When true, the model identifier is embedded in the URL path (e.g. Vertex AI). - bool model_in_path = 8; - // Optional override for the request path. When set, replaces the protocol-derived path. - // An empty string means POST directly to base_url/model_id with no additional path. - optional string request_path_override = 9; -} - -message GetInferenceBundleResponse { - repeated ResolvedRoute routes = 1; - // Opaque revision tag for cache freshness checks. - string revision = 2; - // Timestamp (epoch ms) when this bundle was generated. - int64 generated_at_ms = 3; -} diff --git a/proto/openshell.proto b/proto/openshell.proto index 138b973474..6e2b56b187 100644 --- a/proto/openshell.proto +++ b/proto/openshell.proto @@ -2130,7 +2130,7 @@ message UpdateConfigRequest { // The new policy to apply. // // Sandbox scope (`global=false`): - // - only network_policies and inference fields may differ from create-time + // - only network_policies may differ from create-time // policy; static fields must match version 1. // // Global scope (`global=true`): diff --git a/python/openshell/__init__.py b/python/openshell/__init__.py index 4001d2fb70..f20b76e9c2 100644 --- a/python/openshell/__init__.py +++ b/python/openshell/__init__.py @@ -9,8 +9,6 @@ ClientCredentialsAuth, ExecChunk, ExecResult, - InferenceRouteClient, - InferenceRouteConfig, Sandbox, SandboxClient, SandboxError, @@ -35,8 +33,6 @@ "ClientCredentialsAuth", "ExecChunk", "ExecResult", - "InferenceRouteClient", - "InferenceRouteConfig", "Sandbox", "SandboxClient", "SandboxError", diff --git a/python/openshell/sandbox.py b/python/openshell/sandbox.py index 07eeb6480e..c34a2c05fe 100644 --- a/python/openshell/sandbox.py +++ b/python/openshell/sandbox.py @@ -25,8 +25,6 @@ from ._proto import ( datamodel_pb2, - inference_pb2, - inference_pb2_grpc, openshell_pb2, openshell_pb2_grpc, ) @@ -1204,74 +1202,6 @@ def delete(self, name: str, *, workspace: str) -> bool: return bool(response.deleted) -@dataclass(frozen=True) -class InferenceRouteConfig: - provider_name: str - model_id: str - version: int - - -class InferenceRouteClient: - """gRPC client for workspace-scoped inference route configuration.""" - - def __init__(self, channel: grpc.Channel, *, timeout: float = 30.0) -> None: - self._stub = inference_pb2_grpc.InferenceStub(channel) - self._timeout = timeout - - @classmethod - def from_sandbox_client(cls, client: SandboxClient) -> InferenceRouteClient: - return cls(client._channel, timeout=client._timeout) - - def set_route( - self, - *, - workspace: str, - provider_name: str, - model_id: str, - no_verify: bool = False, - ) -> InferenceRouteConfig: - response = self._stub.SetInferenceRoute( - inference_pb2.SetInferenceRouteRequest( - workspace=workspace, - provider_name=provider_name, - model_id=model_id, - no_verify=no_verify, - ), - timeout=self._timeout, - ) - return InferenceRouteConfig( - provider_name=response.provider_name, - model_id=response.model_id, - version=response.version, - ) - - def get_route(self, *, workspace: str) -> InferenceRouteConfig: - response = self._stub.GetInferenceRoute( - inference_pb2.GetInferenceRouteRequest(workspace=workspace), - timeout=self._timeout, - ) - return InferenceRouteConfig( - provider_name=response.provider_name, - model_id=response.model_id, - version=response.version, - ) - - def delete_route( - self, - *, - workspace: str, - route_name: str = "", - ) -> bool: - response = self._stub.DeleteInferenceRoute( - inference_pb2.DeleteInferenceRouteRequest( - workspace=workspace, - route_name=route_name, - ), - timeout=self._timeout, - ) - return response.deleted - - @dataclass(frozen=True) class WorkspaceRef: name: str diff --git a/python/openshell/sandbox_test.py b/python/openshell/sandbox_test.py index 5e12d0c8a3..fd9e2bcb5b 100644 --- a/python/openshell/sandbox_test.py +++ b/python/openshell/sandbox_test.py @@ -23,7 +23,6 @@ _PYTHON_CLOUDPICKLE_BOOTSTRAP, _SANDBOX_PYTHON_BIN, ClientCredentialsAuth, - InferenceRouteClient, Sandbox, SandboxClient, SandboxError, @@ -403,34 +402,6 @@ def ExecSandbox( ) -class _FakeInferenceStub: - def __init__(self) -> None: - self.set_request = None - self.get_request = None - - def SetInferenceRoute(self, request: Any, timeout: float | None = None) -> Any: - self.set_request = request - _ = timeout - - class _Response: - provider_name = request.provider_name - model_id = request.model_id - version = 1 - - return _Response() - - def GetInferenceRoute(self, request: Any, timeout: float | None = None) -> Any: - self.get_request = request - _ = timeout - - class _Response: - provider_name = "openai-dev" - model_id = "gpt-4.1" - version = 2 - - return _Response() - - def _client_with_fake_stub(stub: object) -> SandboxClient: client = cast("SandboxClient", object.__new__(SandboxClient)) client._timeout = 30.0 @@ -1877,39 +1848,6 @@ def fake_from_active_cluster(**kwargs: Any) -> Any: assert captured["insecure"] is False -def test_inference_set_route_forwards_workspace_and_no_verify() -> None: - stub = _FakeInferenceStub() - client = cast("InferenceRouteClient", object.__new__(InferenceRouteClient)) - client._timeout = 30.0 - client._stub = cast("Any", stub) - - client.set_route( - workspace="production", - provider_name="openai-dev", - model_id="gpt-4.1", - no_verify=True, - ) - - assert stub.set_request is not None - assert stub.set_request.no_verify is True - assert stub.set_request.workspace == "production" - - -def test_inference_get_route_forwards_workspace() -> None: - stub = _FakeInferenceStub() - client = cast("InferenceRouteClient", object.__new__(InferenceRouteClient)) - client._timeout = 30.0 - client._stub = cast("Any", stub) - - config = client.get_route(workspace="staging") - - assert stub.get_request is not None - assert stub.get_request.workspace == "staging" - assert config.provider_name == "openai-dev" - assert config.model_id == "gpt-4.1" - assert config.version == 2 - - # --------------------------------------------------------------------------- # Encoding regression tests (utf-8 explicit on all config file reads/writes) # --------------------------------------------------------------------------- diff --git a/python/wheel_verifier_test.py b/python/wheel_verifier_test.py index 777f348f73..873fb75fbc 100644 --- a/python/wheel_verifier_test.py +++ b/python/wheel_verifier_test.py @@ -32,7 +32,7 @@ def _wheel_files() -> set[str]: "openshell/py.typed", "openshell/_proto/__init__.py", } - for stem in ("datamodel", "inference", "openshell", "options", "sandbox"): + for stem in ("datamodel", "openshell", "options", "sandbox"): files.add(f"openshell/_proto/{stem}_pb2.py") files.add(f"openshell/_proto/{stem}_pb2.pyi") files.add(f"openshell/_proto/{stem}_pb2_grpc.py") diff --git a/scripts/generate_third_party_notices.py b/scripts/generate_third_party_notices.py index 8329a76837..fc6d86f419 100755 --- a/scripts/generate_third_party_notices.py +++ b/scripts/generate_third_party_notices.py @@ -31,7 +31,6 @@ "openshell-core", "openshell-policy", "openshell-providers", - "openshell-router", "openshell-sandbox", "openshell-server", "openshell-tui", diff --git a/scripts/keycloak-realm.json b/scripts/keycloak-realm.json index 358d04865b..0664d6160e 100644 --- a/scripts/keycloak-realm.json +++ b/scripts/keycloak-realm.json @@ -250,24 +250,6 @@ "display.on.consent.screen": "true" } }, - { - "name": "inference:read", - "description": "Read inference resources", - "protocol": "openid-connect", - "attributes": { - "include.in.token.scope": "true", - "display.on.consent.screen": "true" - } - }, - { - "name": "inference:write", - "description": "Write inference resources", - "protocol": "openid-connect", - "attributes": { - "include.in.token.scope": "true", - "display.on.consent.screen": "true" - } - }, { "name": "workspace:read", "description": "Read workspace resources", @@ -328,7 +310,7 @@ } ], "defaultClientScopes": ["openid", "profile", "email", "roles", "web-origins", "acr"], - "optionalClientScopes": ["sandbox:read", "sandbox:write", "provider:read", "provider:write", "config:read", "config:write", "inference:read", "inference:write", "workspace:read", "workspace:write", "openshell:all"] + "optionalClientScopes": ["sandbox:read", "sandbox:write", "provider:read", "provider:write", "config:read", "config:write", "workspace:read", "workspace:write", "openshell:all"] }, { "clientId": "openshell-ci", diff --git a/sdk/go/README.md b/sdk/go/README.md index e011af01d8..cab09a296d 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -227,36 +227,6 @@ The pre-1.0 SDK intentionally includes source-incompatible API corrections: These changes are intentional while the module remains below v1. Update callers as one migration rather than relying on the v0.0.101 API shape. -### Inference Route Management - -Configure how inference requests are routed for a workspace: - -```go -// Set an inference route -route, err := client.Inference().SetRoute(ctx, "my-workspace", &v1.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "", // empty string = default route - TimeoutSecs: 120, -}) -if err != nil { - log.Fatal(err) -} -fmt.Printf("Route v%d: %s/%s\n", route.Version, route.ProviderName, route.ModelID) - -// Retrieve the route -route, err = client.Inference().GetRoute(ctx, "my-workspace", "") -if err != nil { - log.Fatal(err) -} - -// Delete the route -err = client.Inference().DeleteRoute(ctx, "my-workspace", "") -if err != nil { - log.Fatal(err) -} -``` - ## Architecture ``` @@ -270,7 +240,6 @@ Client │ ├── Profiles() → ProfileInterface (list, get, import, update, lint, delete) │ └── Refresh() → RefreshInterface (configure, status, rotate, delete) ├── Workspaces() → WorkspaceInterface (create, get, list, delete, members) - ├── Inference() → InferenceInterface (set, get, delete inference routes) └── Policy() → PolicyInterface (draft review, approve, reject, merge, status) ``` @@ -292,7 +261,7 @@ consumers import a single package. See the [Architecture](https://ro14nd.de/open | Policy management (draft review, approve, reject, merge, global policy) | `PolicyInterface` | [Policy](https://ro14nd.de/openshell-sdk-go/api/policy.html) | | Sandbox logs (streaming retrieval) | `SandboxInterface` | [Sandboxes](https://ro14nd.de/openshell-sdk-go/api/sandboxes.html) | | Workspace management (create, get, list, delete, members) | `WorkspaceInterface` | [Workspaces](https://ro14nd.de/openshell-sdk-go/api/workspaces.html) | -| Inference route management (set, get, delete) | `InferenceInterface` | [Inference](https://ro14nd.de/openshell-sdk-go/api/inference.html) | +| Sandbox provider attachment (attach, detach, list) | `SandboxInterface` | [Sandboxes](https://ro14nd.de/openshell-sdk-go/api/sandboxes.html) | | Gateway info and current user identity | `HealthInterface` | [Health](https://ro14nd.de/openshell-sdk-go/api/health.html) | | Health checking | `HealthInterface` | [Health](https://ro14nd.de/openshell-sdk-go/api/health.html) | | SSH tunneling and TCP forwarding | `SSHInterface`, `TCPInterface` | [SSH](https://ro14nd.de/openshell-sdk-go/api/ssh.html), [TCP](https://ro14nd.de/openshell-sdk-go/api/tcp.html) | diff --git a/sdk/go/buf.gen.yaml b/sdk/go/buf.gen.yaml index 626c71bb95..3d45c0ace7 100644 --- a/sdk/go/buf.gen.yaml +++ b/sdk/go/buf.gen.yaml @@ -10,7 +10,6 @@ version: v2 inputs: - proto_file: proto/openshell.proto - - proto_file: proto/inference.proto plugins: - local: protoc-gen-go @@ -22,7 +21,6 @@ plugins: - Mdatamodel.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/datamodelv1 - Msandbox.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/sandboxv1 - Moptions.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/optionsv1 - - Minference.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/inferencev1 - local: protoc-gen-go-grpc out: sdk/go include_imports: true @@ -32,4 +30,3 @@ plugins: - Mdatamodel.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/datamodelv1 - Msandbox.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/sandboxv1 - Moptions.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/optionsv1 - - Minference.proto=github.com/NVIDIA/OpenShell/sdk/go/proto/inferencev1 diff --git a/sdk/go/docs/src/api/fake.md b/sdk/go/docs/src/api/fake.md index ae96fc9d39..fd1e14bbfa 100644 --- a/sdk/go/docs/src/api/fake.md +++ b/sdk/go/docs/src/api/fake.md @@ -95,7 +95,6 @@ additive concrete-client accessors such as `SandboxTemplates()`: | `Providers()` | `ProviderInterface` | Full CRUD, Ensure | | `Workspaces()` | `WorkspaceInterface` | Full CRUD, Members | | `Health()` | `HealthInterface` | Configurable result | -| `Inference()` | `InferenceInterface` | Route CRUD | | `Policy()` | `PolicyInterface` | List, GetStatus (draft ops return Unimplemented) | | `Exec()` | `ExecInterface` | Returns Unimplemented | | `Files()` | `FileInterface` | Returns Unimplemented | diff --git a/sdk/go/openshell/v1/client.go b/sdk/go/openshell/v1/client.go index 06d8183360..c3ae75adda 100644 --- a/sdk/go/openshell/v1/client.go +++ b/sdk/go/openshell/v1/client.go @@ -31,7 +31,6 @@ type ClientInterface interface { Config() ConfigInterface Policy() PolicyInterface Workspaces() WorkspaceInterface - Inference() InferenceInterface Close() error } @@ -65,7 +64,6 @@ type Client struct { cfg ConfigInterface policy PolicyInterface workspaces WorkspaceInterface - inference InferenceInterface } // NewClient creates a new SDK client connected to the given gateway. @@ -112,7 +110,6 @@ func NewClient(cfg Config) (*Client, error) { c.cfg = newConfigClient(conn, c.sandboxes) c.policy = newPolicyClient(conn) c.workspaces = newWorkspaceClient(conn) - c.inference = newInferenceClient(conn) return c, nil } @@ -159,9 +156,6 @@ func (c *Client) Policy() PolicyInterface { return c.policy } // Workspaces returns the workspace management sub-client. func (c *Client) Workspaces() WorkspaceInterface { return c.workspaces } -// Inference returns the inference route management sub-client. -func (c *Client) Inference() InferenceInterface { return c.inference } - // Close closes the underlying gRPC connection. Safe to call multiple times. func (c *Client) Close() error { c.closeOnce.Do(func() { diff --git a/sdk/go/openshell/v1/client_test.go b/sdk/go/openshell/v1/client_test.go index ce3ca860e2..47c886a8e1 100644 --- a/sdk/go/openshell/v1/client_test.go +++ b/sdk/go/openshell/v1/client_test.go @@ -36,7 +36,6 @@ func TestNewClient_ValidConfig(t *testing.T) { assert.NotNil(t, client.Config()) assert.NotNil(t, client.Policy()) assert.NotNil(t, client.Workspaces()) - assert.NotNil(t, client.Inference()) err = client.Close() assert.NoError(t, err) diff --git a/sdk/go/openshell/v1/doc.go b/sdk/go/openshell/v1/doc.go index d088ae68fc..900d86c643 100644 --- a/sdk/go/openshell/v1/doc.go +++ b/sdk/go/openshell/v1/doc.go @@ -5,7 +5,7 @@ // // The SDK follows the Kubernetes client-go sub-client pattern: a single Client // provides typed accessors for each resource domain (Sandboxes, Providers, Exec, -// Files, Health, Services, SSH, TCP, Config, Policy, Workspaces, Inference). All operations accept a context.Context and return idiomatic +// Files, Health, Services, SSH, TCP, Config, Policy, and Workspaces). All operations accept a context.Context and return idiomatic // Go types. Proto-generated types never appear in the public API. // // # Quick Start @@ -441,31 +441,4 @@ // log.Fatal(err) // } // fmt.Printf("New settings revision: %d\n", result.SettingsRevision) -// -// # Inference Route Management -// -// Configure workspace-scoped inference routing to control how inference -// requests are forwarded to upstream providers: -// -// route, err := client.Inference().SetRoute(ctx, "my-workspace", &v1.InferenceRouteConfig{ -// ProviderName: "openai", -// ModelID: "gpt-4", -// RouteName: "", // empty string = default route -// TimeoutSecs: 120, -// }) -// if err != nil { -// log.Fatal(err) -// } -// fmt.Printf("Route v%d: %s/%s\n", route.Version, route.ProviderName, route.ModelID) -// -// route, err = client.Inference().GetRoute(ctx, "my-workspace", "") -// if err != nil { -// log.Fatal(err) -// } -// fmt.Printf("Provider: %s, Model: %s\n", route.ProviderName, route.ModelID) -// -// err = client.Inference().DeleteRoute(ctx, "my-workspace", "") -// if err != nil { -// log.Fatal(err) -// } package v1 diff --git a/sdk/go/openshell/v1/example_fake_test.go b/sdk/go/openshell/v1/example_fake_test.go index c59256413e..94c9ccbd7c 100644 --- a/sdk/go/openshell/v1/example_fake_test.go +++ b/sdk/go/openshell/v1/example_fake_test.go @@ -153,45 +153,3 @@ func ExampleNewClient_stopOnTerminal() { // Output: // Events received: 2 } - -// ExampleNewClient_inferenceRoute demonstrates setting and retrieving an -// inference route using the fake client. -func ExampleNewClient_inferenceRoute() { - client := fake.NewClient() - defer client.Close() //nolint:errcheck - - ctx := context.Background() - - // Set an inference route for a workspace - route, err := client.Inference().SetRoute(ctx, "my-workspace", &v1.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "", - TimeoutSecs: 120, - }) - if err != nil { - log.Fatal(err) - } - fmt.Printf("Set route v%d: %s/%s\n", route.Version, route.ProviderName, route.ModelID) - - // Retrieve the route - route, err = client.Inference().GetRoute(ctx, "my-workspace", "") - if err != nil { - log.Fatal(err) - } - fmt.Printf("Got route: %s/%s (timeout: %ds)\n", route.ProviderName, route.ModelID, route.TimeoutSecs) - - // Delete the route - err = client.Inference().DeleteRoute(ctx, "my-workspace", "") - if err != nil { - log.Fatal(err) - } - - // Verify deletion - _, err = client.Inference().GetRoute(ctx, "my-workspace", "") - fmt.Println("After delete:", v1.IsNotFound(err)) - // Output: - // Set route v1: openai/gpt-4 - // Got route: openai/gpt-4 (timeout: 120s) - // After delete: true -} diff --git a/sdk/go/openshell/v1/fake/fake.go b/sdk/go/openshell/v1/fake/fake.go index 16a35fed8a..7e73d2a994 100644 --- a/sdk/go/openshell/v1/fake/fake.go +++ b/sdk/go/openshell/v1/fake/fake.go @@ -35,7 +35,6 @@ type Client struct { cfg v1.ConfigInterface policy v1.PolicyInterface workspaces v1.WorkspaceInterface - inference v1.InferenceInterface closeOnce sync.Once closed bool @@ -95,7 +94,6 @@ func NewClient(opts ...ClientOption) *Client { fc.cfg = newFakeConfigClient(fc.isClosed) fc.policy = newFakePolicyClient(fc.isClosed) fc.workspaces = newFakeWorkspaceClient(fc.workspaceStore, fc.memberStore, fc.isClosed) - fc.inference = newFakeInferenceClient(fc.isClosed) for _, opt := range opts { opt(fc) @@ -154,9 +152,6 @@ func (fc *Client) Policy() v1.PolicyInterface { return fc.policy } // Workspaces returns the workspace management sub-client. func (fc *Client) Workspaces() v1.WorkspaceInterface { return fc.workspaces } -// Inference returns the inference route management sub-client. -func (fc *Client) Inference() v1.InferenceInterface { return fc.inference } - // Close marks the client as closed, stops all active watchers, and causes // subsequent sub-client calls to return Unavailable. Safe to call multiple // times. diff --git a/sdk/go/openshell/v1/fake/inference.go b/sdk/go/openshell/v1/fake/inference.go deleted file mode 100644 index cb4ef77e54..0000000000 --- a/sdk/go/openshell/v1/fake/inference.go +++ /dev/null @@ -1,119 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package fake - -import ( - "context" - "sync" - - "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/types" -) - -type fakeInferenceClient struct { - mu sync.RWMutex - routes map[string]*types.InferenceRoute // keyed by "workspace/routeName" - closedFunc func() bool -} - -func newFakeInferenceClient(closedFunc func() bool) *fakeInferenceClient { - return &fakeInferenceClient{ - routes: make(map[string]*types.InferenceRoute), - closedFunc: closedFunc, - } -} - -func inferenceKey(workspace, routeName string) string { - return workspace + "/" + routeName -} - -func copyInferenceRoute(r *types.InferenceRoute) *types.InferenceRoute { - if r == nil { - return nil - } - cp := *r - if r.ValidatedEndpoints != nil { - cp.ValidatedEndpoints = make([]types.ValidatedEndpoint, len(r.ValidatedEndpoints)) - copy(cp.ValidatedEndpoints, r.ValidatedEndpoints) - } - return &cp -} - -func (c *fakeInferenceClient) SetRoute(_ context.Context, workspace string, config *types.InferenceRouteConfig) (*types.InferenceRoute, error) { - if c.closedFunc() { - return nil, &types.StatusError{Code: types.ErrorUnavailable, Message: "client is closed"} - } - if workspace == "" { - return nil, &types.StatusError{Code: types.ErrorInvalidArgument, Message: "workspace must not be empty"} - } - if config == nil { - return nil, &types.StatusError{Code: types.ErrorInvalidArgument, Message: "config must not be nil"} - } - if config.ProviderName == "" { - return nil, &types.StatusError{Code: types.ErrorInvalidArgument, Message: "provider name must not be empty"} - } - if config.ModelID == "" { - return nil, &types.StatusError{Code: types.ErrorInvalidArgument, Message: "model ID must not be empty"} - } - - key := inferenceKey(workspace, config.RouteName) - - c.mu.Lock() - defer c.mu.Unlock() - - // Determine version: increment if route exists, start at 1 otherwise. - var version uint64 = 1 - if existing, ok := c.routes[key]; ok { - version = existing.Version + 1 - } - - route := &types.InferenceRoute{ - ProviderName: config.ProviderName, - ModelID: config.ModelID, - Version: version, - RouteName: config.RouteName, - TimeoutSecs: config.TimeoutSecs, - Workspace: workspace, - } - - c.routes[key] = copyInferenceRoute(route) - return copyInferenceRoute(route), nil -} - -func (c *fakeInferenceClient) GetRoute(_ context.Context, workspace, routeName string) (*types.InferenceRoute, error) { - if c.closedFunc() { - return nil, &types.StatusError{Code: types.ErrorUnavailable, Message: "client is closed"} - } - if workspace == "" { - return nil, &types.StatusError{Code: types.ErrorInvalidArgument, Message: "workspace must not be empty"} - } - - key := inferenceKey(workspace, routeName) - - c.mu.RLock() - defer c.mu.RUnlock() - - route, ok := c.routes[key] - if !ok { - return nil, &types.StatusError{Code: types.ErrorNotFound, Message: "route not found"} - } - return copyInferenceRoute(route), nil -} - -func (c *fakeInferenceClient) DeleteRoute(_ context.Context, workspace, routeName string) error { - if c.closedFunc() { - return &types.StatusError{Code: types.ErrorUnavailable, Message: "client is closed"} - } - if workspace == "" { - return &types.StatusError{Code: types.ErrorInvalidArgument, Message: "workspace must not be empty"} - } - - key := inferenceKey(workspace, routeName) - - c.mu.Lock() - defer c.mu.Unlock() - - // Idempotent: deleting a non-existent route is not an error. - delete(c.routes, key) - return nil -} diff --git a/sdk/go/openshell/v1/fake/inference_test.go b/sdk/go/openshell/v1/fake/inference_test.go deleted file mode 100644 index 21e69520f9..0000000000 --- a/sdk/go/openshell/v1/fake/inference_test.go +++ /dev/null @@ -1,273 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package fake - -import ( - "context" - "testing" - - "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/types" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestFakeInference_SetRoute_Success(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - route, err := fc.Inference().SetRoute(context.Background(), "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - TimeoutSecs: 120, - }) - - require.NoError(t, err) - require.NotNil(t, route) - assert.Equal(t, "openai", route.ProviderName) - assert.Equal(t, "gpt-4", route.ModelID) - assert.Equal(t, uint64(1), route.Version) - assert.Equal(t, "my-route", route.RouteName) - assert.Equal(t, uint64(120), route.TimeoutSecs) - assert.Equal(t, "ws", route.Workspace) -} - -func TestFakeInference_SetRoute_UpdateIncrementsVersion(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - ctx := context.Background() - - route1, err := fc.Inference().SetRoute(ctx, "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - }) - require.NoError(t, err) - assert.Equal(t, uint64(1), route1.Version) - - route2, err := fc.Inference().SetRoute(ctx, "ws", &types.InferenceRouteConfig{ - ProviderName: "anthropic", - ModelID: "claude-4", - RouteName: "my-route", - }) - require.NoError(t, err) - assert.Equal(t, uint64(2), route2.Version) - assert.Equal(t, "anthropic", route2.ProviderName) -} - -func TestFakeInference_SetRoute_EmptyWorkspace(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - _, err := fc.Inference().SetRoute(context.Background(), "", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - }) - - require.Error(t, err) - assert.True(t, types.IsInvalidArgument(err)) -} - -func TestFakeInference_SetRoute_NilConfig(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - _, err := fc.Inference().SetRoute(context.Background(), "ws", nil) - - require.Error(t, err) - assert.True(t, types.IsInvalidArgument(err)) -} - -func TestFakeInference_SetRoute_EmptyProviderName(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - _, err := fc.Inference().SetRoute(context.Background(), "ws", &types.InferenceRouteConfig{ - ProviderName: "", - ModelID: "gpt-4", - }) - - require.Error(t, err) - assert.True(t, types.IsInvalidArgument(err)) -} - -func TestFakeInference_SetRoute_EmptyModelID(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - _, err := fc.Inference().SetRoute(context.Background(), "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "", - }) - - require.Error(t, err) - assert.True(t, types.IsInvalidArgument(err)) -} - -func TestFakeInference_SetRoute_EmptyRouteName(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - route, err := fc.Inference().SetRoute(context.Background(), "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "", - }) - - require.NoError(t, err) - require.NotNil(t, route) - assert.Empty(t, route.RouteName) -} - -func TestFakeInference_GetRoute_Success(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - ctx := context.Background() - - _, err := fc.Inference().SetRoute(ctx, "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - TimeoutSecs: 120, - }) - require.NoError(t, err) - - route, err := fc.Inference().GetRoute(ctx, "ws", "my-route") - - require.NoError(t, err) - require.NotNil(t, route) - assert.Equal(t, "openai", route.ProviderName) - assert.Equal(t, "gpt-4", route.ModelID) - assert.Equal(t, "my-route", route.RouteName) - assert.Equal(t, uint64(120), route.TimeoutSecs) - assert.Equal(t, "ws", route.Workspace) -} - -func TestFakeInference_GetRoute_EmptyWorkspace(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - _, err := fc.Inference().GetRoute(context.Background(), "", "my-route") - - require.Error(t, err) - assert.True(t, types.IsInvalidArgument(err)) -} - -func TestFakeInference_GetRoute_NotFound(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - _, err := fc.Inference().GetRoute(context.Background(), "ws", "nonexistent") - - require.Error(t, err) - assert.True(t, types.IsNotFound(err)) -} - -func TestFakeInference_GetRoute_DeepCopy(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - ctx := context.Background() - - _, err := fc.Inference().SetRoute(ctx, "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - }) - require.NoError(t, err) - - route1, err := fc.Inference().GetRoute(ctx, "ws", "my-route") - require.NoError(t, err) - - // Mutate the returned route; it should not affect the stored copy. - route1.ProviderName = "mutated" - - route2, err := fc.Inference().GetRoute(ctx, "ws", "my-route") - require.NoError(t, err) - assert.Equal(t, "openai", route2.ProviderName) -} - -func TestFakeInference_DeleteRoute_Success(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - ctx := context.Background() - - _, err := fc.Inference().SetRoute(ctx, "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - }) - require.NoError(t, err) - - err = fc.Inference().DeleteRoute(ctx, "ws", "my-route") - require.NoError(t, err) - - // Subsequent get should return NotFound. - _, err = fc.Inference().GetRoute(ctx, "ws", "my-route") - require.Error(t, err) - assert.True(t, types.IsNotFound(err)) -} - -func TestFakeInference_DeleteRoute_EmptyWorkspace(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - err := fc.Inference().DeleteRoute(context.Background(), "", "my-route") - - require.Error(t, err) - assert.True(t, types.IsInvalidArgument(err)) -} - -func TestFakeInference_DeleteRoute_Idempotent(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - // Deleting a non-existent route should not error. - err := fc.Inference().DeleteRoute(context.Background(), "ws", "nonexistent") - require.NoError(t, err) -} - -func TestFakeInference_WorkspaceIsolation(t *testing.T) { - fc := NewClient() - defer fc.Close() //nolint:errcheck - - ctx := context.Background() - - _, err := fc.Inference().SetRoute(ctx, "ws1", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "shared-name", - }) - require.NoError(t, err) - - // Different workspace should not see the route. - _, err = fc.Inference().GetRoute(ctx, "ws2", "shared-name") - require.Error(t, err) - assert.True(t, types.IsNotFound(err)) -} - -func TestFakeInference_ClosedClient(t *testing.T) { - fc := NewClient() - _ = fc.Close() - - ctx := context.Background() - - _, err := fc.Inference().SetRoute(ctx, "ws", &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - }) - require.Error(t, err) - assert.True(t, types.IsUnavailable(err)) - - _, err = fc.Inference().GetRoute(ctx, "ws", "route") - require.Error(t, err) - assert.True(t, types.IsUnavailable(err)) - - err = fc.Inference().DeleteRoute(ctx, "ws", "route") - require.Error(t, err) - assert.True(t, types.IsUnavailable(err)) -} diff --git a/sdk/go/openshell/v1/inference.go b/sdk/go/openshell/v1/inference.go deleted file mode 100644 index 7dfe98ae72..0000000000 --- a/sdk/go/openshell/v1/inference.go +++ /dev/null @@ -1,38 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package v1 - -import ( - "context" - - "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/types" -) - -// InferenceRouteConfig holds parameters for setting an inference route. -type InferenceRouteConfig = types.InferenceRouteConfig - -// InferenceRoute represents a configured inference route as returned by the -// gateway. -type InferenceRoute = types.InferenceRoute - -// ValidatedEndpoint represents an endpoint probed during route validation. -type ValidatedEndpoint = types.ValidatedEndpoint - -// InferenceInterface defines inference route management operations. -// Accessed via client.Inference(). -type InferenceInterface interface { - // SetRoute configures an inference route for a workspace. - // Returns ErrorInvalidArgument if workspace, providerName, or modelID is empty. - SetRoute(ctx context.Context, workspace string, config *InferenceRouteConfig) (*InferenceRoute, error) - - // GetRoute retrieves the inference route for a workspace by route name. - // Returns ErrorInvalidArgument if workspace is empty. - // Returns ErrorNotFound if no route exists for the given name. - GetRoute(ctx context.Context, workspace, routeName string) (*InferenceRoute, error) - - // DeleteRoute removes an inference route from a workspace. - // Returns ErrorInvalidArgument if workspace is empty. - // Idempotent: deleting a non-existent route is not an error. - DeleteRoute(ctx context.Context, workspace, routeName string) error -} diff --git a/sdk/go/openshell/v1/inference_client.go b/sdk/go/openshell/v1/inference_client.go deleted file mode 100644 index 821aca35ea..0000000000 --- a/sdk/go/openshell/v1/inference_client.go +++ /dev/null @@ -1,72 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package v1 - -import ( - "context" - - "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/internal/converter" - pb "github.com/NVIDIA/OpenShell/sdk/go/proto/inferencev1" - "google.golang.org/grpc" -) - -type inferenceClient struct { - client pb.InferenceClient -} - -func newInferenceClient(conn grpc.ClientConnInterface) *inferenceClient { - return &inferenceClient{client: pb.NewInferenceClient(conn)} -} - -func (c *inferenceClient) SetRoute(ctx context.Context, workspace string, config *InferenceRouteConfig) (*InferenceRoute, error) { - if workspace == "" { - return nil, &StatusError{Code: ErrorInvalidArgument, Message: "workspace must not be empty"} - } - if config == nil { - return nil, &StatusError{Code: ErrorInvalidArgument, Message: "config must not be nil"} - } - if config.ProviderName == "" { - return nil, &StatusError{Code: ErrorInvalidArgument, Message: "provider name must not be empty"} - } - if config.ModelID == "" { - return nil, &StatusError{Code: ErrorInvalidArgument, Message: "model ID must not be empty"} - } - - req := converter.InferenceRouteConfigToProto(workspace, config) - resp, err := c.client.SetInferenceRoute(ctx, req) - if err != nil { - return nil, converter.FromGRPCError(err) - } - return converter.InferenceRouteFromSetResponse(resp), nil -} - -func (c *inferenceClient) GetRoute(ctx context.Context, workspace, routeName string) (*InferenceRoute, error) { - if workspace == "" { - return nil, &StatusError{Code: ErrorInvalidArgument, Message: "workspace must not be empty"} - } - - resp, err := c.client.GetInferenceRoute(ctx, &pb.GetInferenceRouteRequest{ - Workspace: workspace, - RouteName: routeName, - }) - if err != nil { - return nil, converter.FromGRPCError(err) - } - return converter.InferenceRouteFromGetResponse(resp), nil -} - -func (c *inferenceClient) DeleteRoute(ctx context.Context, workspace, routeName string) error { - if workspace == "" { - return &StatusError{Code: ErrorInvalidArgument, Message: "workspace must not be empty"} - } - - _, err := c.client.DeleteInferenceRoute(ctx, &pb.DeleteInferenceRouteRequest{ - Workspace: workspace, - RouteName: routeName, - }) - if err != nil { - return converter.FromGRPCError(err) - } - return nil -} diff --git a/sdk/go/openshell/v1/inference_client_test.go b/sdk/go/openshell/v1/inference_client_test.go deleted file mode 100644 index a75e6e97fe..0000000000 --- a/sdk/go/openshell/v1/inference_client_test.go +++ /dev/null @@ -1,405 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package v1 - -import ( - "context" - "net" - "testing" - - pb "github.com/NVIDIA/OpenShell/sdk/go/proto/inferencev1" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "google.golang.org/grpc" - "google.golang.org/grpc/codes" - "google.golang.org/grpc/credentials/insecure" - "google.golang.org/grpc/status" - "google.golang.org/grpc/test/bufconn" -) - -type mockInferenceServer struct { - pb.UnimplementedInferenceServer - - setResp *pb.SetInferenceRouteResponse - getResp *pb.GetInferenceRouteResponse - deleteResp *pb.DeleteInferenceRouteResponse - err error - - lastSetReq *pb.SetInferenceRouteRequest - lastGetReq *pb.GetInferenceRouteRequest - lastDeleteReq *pb.DeleteInferenceRouteRequest -} - -func (s *mockInferenceServer) SetInferenceRoute(_ context.Context, req *pb.SetInferenceRouteRequest) (*pb.SetInferenceRouteResponse, error) { - s.lastSetReq = req - if s.err != nil { - return nil, s.err - } - return s.setResp, nil -} - -func (s *mockInferenceServer) GetInferenceRoute(_ context.Context, req *pb.GetInferenceRouteRequest) (*pb.GetInferenceRouteResponse, error) { - s.lastGetReq = req - if s.err != nil { - return nil, s.err - } - return s.getResp, nil -} - -func (s *mockInferenceServer) DeleteInferenceRoute(_ context.Context, req *pb.DeleteInferenceRouteRequest) (*pb.DeleteInferenceRouteResponse, error) { - s.lastDeleteReq = req - if s.err != nil { - return nil, s.err - } - return s.deleteResp, nil -} - -func newMockInferenceServer(mock *mockInferenceServer) (*grpc.ClientConn, func()) { - lis := bufconn.Listen(bufSize) - srv := grpc.NewServer() - pb.RegisterInferenceServer(srv, mock) - - go func() { _ = srv.Serve(lis) }() - - conn, err := grpc.NewClient("passthrough:///bufconn", - grpc.WithContextDialer(func(_ context.Context, _ string) (net.Conn, error) { - return lis.Dial() - }), - grpc.WithTransportCredentials(insecure.NewCredentials()), - ) - if err != nil { - srv.Stop() - panic("grpc.NewClient failed: " + err.Error()) - } - - return conn, func() { - _ = conn.Close() - srv.Stop() - } -} - -// --- SetRoute tests --- - -func TestSetRoute_Success(t *testing.T) { - mock := &mockInferenceServer{ - setResp: &pb.SetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 1, - RouteName: "my-route", - ValidationPerformed: true, - ValidatedEndpoints: []*pb.ValidatedEndpoint{ - {Url: "https://api.openai.com/v1", Protocol: "openai"}, - }, - TimeoutSecs: 120, - Workspace: "team-alpha", - }, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - route, err := ic.SetRoute(context.Background(), "team-alpha", &InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - NoVerify: false, - TimeoutSecs: 120, - }) - - require.NoError(t, err) - require.NotNil(t, route) - assert.Equal(t, "openai", route.ProviderName) - assert.Equal(t, "gpt-4", route.ModelID) - assert.Equal(t, uint64(1), route.Version) - assert.Equal(t, "my-route", route.RouteName) - assert.True(t, route.ValidationPerformed) - require.Len(t, route.ValidatedEndpoints, 1) - assert.Equal(t, "https://api.openai.com/v1", route.ValidatedEndpoints[0].URL) - assert.Equal(t, "openai", route.ValidatedEndpoints[0].Protocol) - assert.Equal(t, uint64(120), route.TimeoutSecs) - assert.Equal(t, "team-alpha", route.Workspace) - - // Verify the proto request was correctly constructed. - assert.Equal(t, "openai", mock.lastSetReq.GetProviderName()) - assert.Equal(t, "gpt-4", mock.lastSetReq.GetModelId()) - assert.Equal(t, "my-route", mock.lastSetReq.GetRouteName()) - assert.Equal(t, "team-alpha", mock.lastSetReq.GetWorkspace()) - assert.Equal(t, uint64(120), mock.lastSetReq.GetTimeoutSecs()) -} - -func TestSetRoute_EmptyWorkspace(t *testing.T) { - mock := &mockInferenceServer{} - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.SetRoute(context.Background(), "", &InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - }) - - require.Error(t, err) - assert.True(t, IsInvalidArgument(err)) -} - -func TestSetRoute_NilConfig(t *testing.T) { - mock := &mockInferenceServer{} - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.SetRoute(context.Background(), "ws", nil) - - require.Error(t, err) - assert.True(t, IsInvalidArgument(err)) -} - -func TestSetRoute_EmptyProviderName(t *testing.T) { - mock := &mockInferenceServer{} - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.SetRoute(context.Background(), "ws", &InferenceRouteConfig{ - ProviderName: "", - ModelID: "gpt-4", - }) - - require.Error(t, err) - assert.True(t, IsInvalidArgument(err)) -} - -func TestSetRoute_EmptyModelID(t *testing.T) { - mock := &mockInferenceServer{} - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.SetRoute(context.Background(), "ws", &InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "", - }) - - require.Error(t, err) - assert.True(t, IsInvalidArgument(err)) -} - -func TestSetRoute_EmptyRouteName(t *testing.T) { - mock := &mockInferenceServer{ - setResp: &pb.SetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 1, - RouteName: "", - Workspace: "ws", - }, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - route, err := ic.SetRoute(context.Background(), "ws", &InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "", - }) - - require.NoError(t, err) - require.NotNil(t, route) - assert.Empty(t, route.RouteName) -} - -func TestSetRoute_PermissionDenied(t *testing.T) { - mock := &mockInferenceServer{ - err: status.Error(codes.PermissionDenied, "workspace admin required"), - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.SetRoute(context.Background(), "ws", &InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - }) - - require.Error(t, err) - assert.True(t, IsPermissionDenied(err)) -} - -func TestSetRoute_NoVerify(t *testing.T) { - mock := &mockInferenceServer{ - setResp: &pb.SetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 1, - Workspace: "ws", - }, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.SetRoute(context.Background(), "ws", &InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - NoVerify: true, - }) - - require.NoError(t, err) - assert.True(t, mock.lastSetReq.GetNoVerify()) -} - -// --- GetRoute tests --- - -func TestGetRoute_Success(t *testing.T) { - mock := &mockInferenceServer{ - getResp: &pb.GetInferenceRouteResponse{ - ProviderName: "vertex", - ModelId: "gemini-pro", - Version: 3, - RouteName: "default", - TimeoutSecs: 60, - Workspace: "prod", - }, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - route, err := ic.GetRoute(context.Background(), "prod", "default") - - require.NoError(t, err) - require.NotNil(t, route) - assert.Equal(t, "vertex", route.ProviderName) - assert.Equal(t, "gemini-pro", route.ModelID) - assert.Equal(t, uint64(3), route.Version) - assert.Equal(t, "default", route.RouteName) - assert.Equal(t, uint64(60), route.TimeoutSecs) - assert.Equal(t, "prod", route.Workspace) - assert.False(t, route.ValidationPerformed) - assert.Nil(t, route.ValidatedEndpoints) - - assert.Equal(t, "prod", mock.lastGetReq.GetWorkspace()) - assert.Equal(t, "default", mock.lastGetReq.GetRouteName()) -} - -func TestGetRoute_EmptyWorkspace(t *testing.T) { - mock := &mockInferenceServer{} - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.GetRoute(context.Background(), "", "my-route") - - require.Error(t, err) - assert.True(t, IsInvalidArgument(err)) -} - -func TestGetRoute_NotFound(t *testing.T) { - mock := &mockInferenceServer{ - err: status.Error(codes.NotFound, "route not found"), - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - _, err := ic.GetRoute(context.Background(), "ws", "missing-route") - - require.Error(t, err) - assert.True(t, IsNotFound(err)) -} - -func TestGetRoute_EmptyRouteName(t *testing.T) { - mock := &mockInferenceServer{ - getResp: &pb.GetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 1, - RouteName: "", - Workspace: "ws", - }, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - route, err := ic.GetRoute(context.Background(), "ws", "") - - require.NoError(t, err) - require.NotNil(t, route) - assert.Empty(t, route.RouteName) - assert.Empty(t, mock.lastGetReq.GetRouteName()) -} - -// --- DeleteRoute tests --- - -func TestDeleteRoute_Success(t *testing.T) { - mock := &mockInferenceServer{ - deleteResp: &pb.DeleteInferenceRouteResponse{Deleted: true}, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - err := ic.DeleteRoute(context.Background(), "ws", "my-route") - - require.NoError(t, err) - assert.Equal(t, "ws", mock.lastDeleteReq.GetWorkspace()) - assert.Equal(t, "my-route", mock.lastDeleteReq.GetRouteName()) -} - -func TestDeleteRoute_EmptyWorkspace(t *testing.T) { - mock := &mockInferenceServer{} - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - err := ic.DeleteRoute(context.Background(), "", "my-route") - - require.Error(t, err) - assert.True(t, IsInvalidArgument(err)) -} - -func TestDeleteRoute_Idempotent(t *testing.T) { - // Deleting a non-existent route should succeed (gateway returns OK). - mock := &mockInferenceServer{ - deleteResp: &pb.DeleteInferenceRouteResponse{Deleted: false}, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - err := ic.DeleteRoute(context.Background(), "ws", "nonexistent") - - require.NoError(t, err) -} - -func TestDeleteRoute_PermissionDenied(t *testing.T) { - mock := &mockInferenceServer{ - err: status.Error(codes.PermissionDenied, "workspace admin required"), - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - err := ic.DeleteRoute(context.Background(), "ws", "my-route") - - require.Error(t, err) - assert.True(t, IsPermissionDenied(err)) -} - -func TestDeleteRoute_EmptyRouteName(t *testing.T) { - mock := &mockInferenceServer{ - deleteResp: &pb.DeleteInferenceRouteResponse{Deleted: true}, - } - conn, cleanup := newMockInferenceServer(mock) - defer cleanup() - - ic := newInferenceClient(conn) - err := ic.DeleteRoute(context.Background(), "ws", "") - - require.NoError(t, err) - assert.Empty(t, mock.lastDeleteReq.GetRouteName()) -} diff --git a/sdk/go/openshell/v1/internal/converter/inference.go b/sdk/go/openshell/v1/internal/converter/inference.go deleted file mode 100644 index 8d5f2aebd3..0000000000 --- a/sdk/go/openshell/v1/internal/converter/inference.go +++ /dev/null @@ -1,75 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package converter - -import ( - "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/types" - pb "github.com/NVIDIA/OpenShell/sdk/go/proto/inferencev1" -) - -// InferenceRouteConfigToProto converts an SDK InferenceRouteConfig plus -// workspace into a proto SetInferenceRouteRequest. -func InferenceRouteConfigToProto(workspace string, cfg *types.InferenceRouteConfig) *pb.SetInferenceRouteRequest { - if cfg == nil { - return &pb.SetInferenceRouteRequest{Workspace: workspace} - } - return &pb.SetInferenceRouteRequest{ - ProviderName: cfg.ProviderName, - ModelId: cfg.ModelID, - RouteName: cfg.RouteName, - NoVerify: cfg.NoVerify, - TimeoutSecs: cfg.TimeoutSecs, - Workspace: workspace, - } -} - -// InferenceRouteFromSetResponse converts a proto SetInferenceRouteResponse -// to an SDK InferenceRoute. -func InferenceRouteFromSetResponse(resp *pb.SetInferenceRouteResponse) *types.InferenceRoute { - if resp == nil { - return nil - } - return &types.InferenceRoute{ - ProviderName: resp.GetProviderName(), - ModelID: resp.GetModelId(), - Version: resp.GetVersion(), - RouteName: resp.GetRouteName(), - TimeoutSecs: resp.GetTimeoutSecs(), - Workspace: resp.GetWorkspace(), - ValidationPerformed: resp.GetValidationPerformed(), - ValidatedEndpoints: validatedEndpointsFromProto(resp.GetValidatedEndpoints()), - } -} - -// InferenceRouteFromGetResponse converts a proto GetInferenceRouteResponse -// to an SDK InferenceRoute. -func InferenceRouteFromGetResponse(resp *pb.GetInferenceRouteResponse) *types.InferenceRoute { - if resp == nil { - return nil - } - return &types.InferenceRoute{ - ProviderName: resp.GetProviderName(), - ModelID: resp.GetModelId(), - Version: resp.GetVersion(), - RouteName: resp.GetRouteName(), - TimeoutSecs: resp.GetTimeoutSecs(), - Workspace: resp.GetWorkspace(), - } -} - -// validatedEndpointsFromProto converts a slice of proto ValidatedEndpoint -// to SDK ValidatedEndpoint values. Returns nil for nil or empty input. -func validatedEndpointsFromProto(eps []*pb.ValidatedEndpoint) []types.ValidatedEndpoint { - if len(eps) == 0 { - return nil - } - result := make([]types.ValidatedEndpoint, len(eps)) - for i, ep := range eps { - result[i] = types.ValidatedEndpoint{ - URL: ep.GetUrl(), - Protocol: ep.GetProtocol(), - } - } - return result -} diff --git a/sdk/go/openshell/v1/internal/converter/inference_test.go b/sdk/go/openshell/v1/internal/converter/inference_test.go deleted file mode 100644 index c5b9befb62..0000000000 --- a/sdk/go/openshell/v1/internal/converter/inference_test.go +++ /dev/null @@ -1,168 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package converter - -import ( - "testing" - - "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/types" - pb "github.com/NVIDIA/OpenShell/sdk/go/proto/inferencev1" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestInferenceRouteConfigToProto(t *testing.T) { - cfg := &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "my-route", - NoVerify: true, - TimeoutSecs: 120, - } - - req := InferenceRouteConfigToProto("team-alpha", cfg) - - assert.Equal(t, "openai", req.GetProviderName()) - assert.Equal(t, "gpt-4", req.GetModelId()) - assert.Equal(t, "my-route", req.GetRouteName()) - assert.True(t, req.GetNoVerify()) - assert.False(t, req.GetVerify()) - assert.Equal(t, uint64(120), req.GetTimeoutSecs()) - assert.Equal(t, "team-alpha", req.GetWorkspace()) -} - -func TestInferenceRouteConfigToProto_NilConfig(t *testing.T) { - req := InferenceRouteConfigToProto("ws", nil) - - assert.Equal(t, "ws", req.GetWorkspace()) - assert.Empty(t, req.GetProviderName()) -} - -func TestInferenceRouteConfigToProto_EmptyRouteName(t *testing.T) { - cfg := &types.InferenceRouteConfig{ - ProviderName: "openai", - ModelID: "gpt-4", - RouteName: "", - } - - req := InferenceRouteConfigToProto("ws", cfg) - - assert.Empty(t, req.GetRouteName()) -} - -func TestInferenceRouteFromSetResponse(t *testing.T) { - resp := &pb.SetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 5, - RouteName: "my-route", - ValidationPerformed: true, - ValidatedEndpoints: []*pb.ValidatedEndpoint{ - {Url: "https://api.openai.com/v1", Protocol: "openai"}, - {Url: "https://backup.openai.com/v1", Protocol: "openai"}, - }, - TimeoutSecs: 120, - Workspace: "team-alpha", - } - - route := InferenceRouteFromSetResponse(resp) - - require.NotNil(t, route) - assert.Equal(t, "openai", route.ProviderName) - assert.Equal(t, "gpt-4", route.ModelID) - assert.Equal(t, uint64(5), route.Version) - assert.Equal(t, "my-route", route.RouteName) - assert.True(t, route.ValidationPerformed) - require.Len(t, route.ValidatedEndpoints, 2) - assert.Equal(t, "https://api.openai.com/v1", route.ValidatedEndpoints[0].URL) - assert.Equal(t, "openai", route.ValidatedEndpoints[0].Protocol) - assert.Equal(t, "https://backup.openai.com/v1", route.ValidatedEndpoints[1].URL) - assert.Equal(t, uint64(120), route.TimeoutSecs) - assert.Equal(t, "team-alpha", route.Workspace) -} - -func TestInferenceRouteFromSetResponse_Nil(t *testing.T) { - route := InferenceRouteFromSetResponse(nil) - assert.Nil(t, route) -} - -func TestInferenceRouteFromSetResponse_NoEndpoints(t *testing.T) { - resp := &pb.SetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 1, - ValidationPerformed: false, - } - - route := InferenceRouteFromSetResponse(resp) - - require.NotNil(t, route) - assert.Nil(t, route.ValidatedEndpoints) - assert.False(t, route.ValidationPerformed) -} - -func TestInferenceRouteFromGetResponse(t *testing.T) { - resp := &pb.GetInferenceRouteResponse{ - ProviderName: "vertex", - ModelId: "gemini-pro", - Version: 3, - RouteName: "default", - TimeoutSecs: 60, - Workspace: "prod", - } - - route := InferenceRouteFromGetResponse(resp) - - require.NotNil(t, route) - assert.Equal(t, "vertex", route.ProviderName) - assert.Equal(t, "gemini-pro", route.ModelID) - assert.Equal(t, uint64(3), route.Version) - assert.Equal(t, "default", route.RouteName) - assert.Equal(t, uint64(60), route.TimeoutSecs) - assert.Equal(t, "prod", route.Workspace) - assert.False(t, route.ValidationPerformed) - assert.Nil(t, route.ValidatedEndpoints) -} - -func TestInferenceRouteFromGetResponse_Nil(t *testing.T) { - route := InferenceRouteFromGetResponse(nil) - assert.Nil(t, route) -} - -func TestInferenceRouteFromSetResponse_DeepCopy(t *testing.T) { - protoEndpoints := []*pb.ValidatedEndpoint{ - {Url: "https://original.com", Protocol: "openai"}, - } - resp := &pb.SetInferenceRouteResponse{ - ProviderName: "openai", - ModelId: "gpt-4", - Version: 1, - ValidatedEndpoints: protoEndpoints, - } - - route := InferenceRouteFromSetResponse(resp) - - // Mutate the proto source; SDK value should be unaffected. - protoEndpoints[0].Url = "https://mutated.com" - assert.Equal(t, "https://original.com", route.ValidatedEndpoints[0].URL) -} - -func TestInferenceRoundTrip(t *testing.T) { - cfg := &types.InferenceRouteConfig{ - ProviderName: "anthropic", - ModelID: "claude-4", - RouteName: "inference-route", - NoVerify: false, - TimeoutSecs: 90, - } - - req := InferenceRouteConfigToProto("my-ws", cfg) - - assert.Equal(t, cfg.ProviderName, req.GetProviderName()) - assert.Equal(t, cfg.ModelID, req.GetModelId()) - assert.Equal(t, cfg.RouteName, req.GetRouteName()) - assert.Equal(t, cfg.NoVerify, req.GetNoVerify()) - assert.Equal(t, cfg.TimeoutSecs, req.GetTimeoutSecs()) - assert.Equal(t, "my-ws", req.GetWorkspace()) -} diff --git a/sdk/go/openshell/v1/types/inference.go b/sdk/go/openshell/v1/types/inference.go deleted file mode 100644 index 945fd8a7f5..0000000000 --- a/sdk/go/openshell/v1/types/inference.go +++ /dev/null @@ -1,67 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -package types - -// InferenceRouteConfig holds parameters for setting an inference route. -// ProviderName and ModelID are required; the SDK validates them before -// sending the request to the gateway. -type InferenceRouteConfig struct { - // ProviderName is the provider record name for credentials and endpoint mapping. - ProviderName string - - // ModelID is the model identifier to force on generation calls. - ModelID string - - // RouteName is the route name to target. An empty string represents the - // default user-facing route. - RouteName string - - // NoVerify skips synchronous endpoint validation before persistence when true. - NoVerify bool - - // TimeoutSecs is the per-route request timeout in seconds. 0 means use the - // default (60s). - TimeoutSecs uint64 -} - -// InferenceRoute represents a configured inference route as returned by the -// gateway. For SetRoute responses, ValidationPerformed and ValidatedEndpoints -// contain verification metadata; for GetRoute responses they are zero-valued. -type InferenceRoute struct { - // ProviderName is the provider record name. - ProviderName string - - // ModelID is the model identifier. - ModelID string - - // Version is the server-assigned version for the route. - Version uint64 - - // RouteName is the route name that was configured or queried. - RouteName string - - // TimeoutSecs is the per-route request timeout in seconds. - TimeoutSecs uint64 - - // Workspace is the workspace the route belongs to. - Workspace string - - // ValidationPerformed indicates whether endpoint verification ran during - // this request. Only populated for SetRoute responses. - ValidationPerformed bool - - // ValidatedEndpoints lists endpoints probed during validation, if any. - // Only populated for SetRoute responses. - ValidatedEndpoints []ValidatedEndpoint -} - -// ValidatedEndpoint represents an endpoint that was probed during route -// validation. -type ValidatedEndpoint struct { - // URL is the endpoint URL that was validated. - URL string - - // Protocol is the protocol used (e.g., "openai", "vertex"). - Protocol string -} diff --git a/sdk/go/proto/inferencev1/inference.pb.go b/sdk/go/proto/inferencev1/inference.pb.go deleted file mode 100644 index decc6c4f39..0000000000 --- a/sdk/go/proto/inferencev1/inference.pb.go +++ /dev/null @@ -1,1018 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -// Code generated by protoc-gen-go. DO NOT EDIT. -// versions: -// protoc-gen-go v1.36.11 -// protoc (unknown) -// source: inference.proto - -package inferencev1 - -import ( - datamodelv1 "github.com/NVIDIA/OpenShell/sdk/go/proto/datamodelv1" - _ "github.com/NVIDIA/OpenShell/sdk/go/proto/optionsv1" - protoreflect "google.golang.org/protobuf/reflect/protoreflect" - protoimpl "google.golang.org/protobuf/runtime/protoimpl" - reflect "reflect" - sync "sync" - unsafe "unsafe" -) - -const ( - // Verify that this generated code is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) - // Verify that runtime/protoimpl is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) -) - -// Persisted inference route configuration. -// -// Only `provider_name` and `model_id` are stored; endpoint, protocols, -// credentials, and auth style are resolved from the provider at bundle time. -type InferenceRouteConfig struct { - state protoimpl.MessageState `protogen:"open.v1"` - // Provider record name backing this route. - ProviderName string `protobuf:"bytes,1,opt,name=provider_name,json=providerName,proto3" json:"provider_name,omitempty"` - // Model identifier to force on generation calls. - ModelId string `protobuf:"bytes,2,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` - // Per-route request timeout in seconds. 0 means use default (60s). - TimeoutSecs uint64 `protobuf:"varint,3,opt,name=timeout_secs,json=timeoutSecs,proto3" json:"timeout_secs,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *InferenceRouteConfig) Reset() { - *x = InferenceRouteConfig{} - mi := &file_inference_proto_msgTypes[0] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *InferenceRouteConfig) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*InferenceRouteConfig) ProtoMessage() {} - -func (x *InferenceRouteConfig) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[0] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use InferenceRouteConfig.ProtoReflect.Descriptor instead. -func (*InferenceRouteConfig) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{0} -} - -func (x *InferenceRouteConfig) GetProviderName() string { - if x != nil { - return x.ProviderName - } - return "" -} - -func (x *InferenceRouteConfig) GetModelId() string { - if x != nil { - return x.ModelId - } - return "" -} - -func (x *InferenceRouteConfig) GetTimeoutSecs() uint64 { - if x != nil { - return x.TimeoutSecs - } - return 0 -} - -// Storage envelope for a workspace-scoped inference route. -type InferenceRoute struct { - state protoimpl.MessageState `protogen:"open.v1"` - Metadata *datamodelv1.ObjectMeta `protobuf:"bytes,1,opt,name=metadata,proto3" json:"metadata,omitempty"` - Config *InferenceRouteConfig `protobuf:"bytes,2,opt,name=config,proto3" json:"config,omitempty"` - // Monotonic version incremented on every update. - Version uint64 `protobuf:"varint,3,opt,name=version,proto3" json:"version,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *InferenceRoute) Reset() { - *x = InferenceRoute{} - mi := &file_inference_proto_msgTypes[1] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *InferenceRoute) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*InferenceRoute) ProtoMessage() {} - -func (x *InferenceRoute) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[1] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use InferenceRoute.ProtoReflect.Descriptor instead. -func (*InferenceRoute) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{1} -} - -func (x *InferenceRoute) GetMetadata() *datamodelv1.ObjectMeta { - if x != nil { - return x.Metadata - } - return nil -} - -func (x *InferenceRoute) GetConfig() *InferenceRouteConfig { - if x != nil { - return x.Config - } - return nil -} - -func (x *InferenceRoute) GetVersion() uint64 { - if x != nil { - return x.Version - } - return 0 -} - -type SetInferenceRouteRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - // Provider record name to use for credentials + endpoint mapping. - ProviderName string `protobuf:"bytes,1,opt,name=provider_name,json=providerName,proto3" json:"provider_name,omitempty"` - // Model identifier to force on generation calls. - ModelId string `protobuf:"bytes,2,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` - // Route name to target. Empty string defaults to "inference.local" (user-facing). - // Use "sandbox-system" for the sandbox system-level inference route. - RouteName string `protobuf:"bytes,3,opt,name=route_name,json=routeName,proto3" json:"route_name,omitempty"` - // Verify the resolved upstream endpoint synchronously before persistence. - Verify bool `protobuf:"varint,4,opt,name=verify,proto3" json:"verify,omitempty"` - // Skip synchronous endpoint validation before persistence. - NoVerify bool `protobuf:"varint,5,opt,name=no_verify,json=noVerify,proto3" json:"no_verify,omitempty"` - // Per-route request timeout in seconds. 0 means use default (60s). - TimeoutSecs uint64 `protobuf:"varint,6,opt,name=timeout_secs,json=timeoutSecs,proto3" json:"timeout_secs,omitempty"` - // Target workspace. Empty string defaults to "default". - Workspace string `protobuf:"bytes,7,opt,name=workspace,proto3" json:"workspace,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *SetInferenceRouteRequest) Reset() { - *x = SetInferenceRouteRequest{} - mi := &file_inference_proto_msgTypes[2] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *SetInferenceRouteRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SetInferenceRouteRequest) ProtoMessage() {} - -func (x *SetInferenceRouteRequest) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[2] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SetInferenceRouteRequest.ProtoReflect.Descriptor instead. -func (*SetInferenceRouteRequest) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{2} -} - -func (x *SetInferenceRouteRequest) GetProviderName() string { - if x != nil { - return x.ProviderName - } - return "" -} - -func (x *SetInferenceRouteRequest) GetModelId() string { - if x != nil { - return x.ModelId - } - return "" -} - -func (x *SetInferenceRouteRequest) GetRouteName() string { - if x != nil { - return x.RouteName - } - return "" -} - -func (x *SetInferenceRouteRequest) GetVerify() bool { - if x != nil { - return x.Verify - } - return false -} - -func (x *SetInferenceRouteRequest) GetNoVerify() bool { - if x != nil { - return x.NoVerify - } - return false -} - -func (x *SetInferenceRouteRequest) GetTimeoutSecs() uint64 { - if x != nil { - return x.TimeoutSecs - } - return 0 -} - -func (x *SetInferenceRouteRequest) GetWorkspace() string { - if x != nil { - return x.Workspace - } - return "" -} - -type ValidatedEndpoint struct { - state protoimpl.MessageState `protogen:"open.v1"` - Url string `protobuf:"bytes,1,opt,name=url,proto3" json:"url,omitempty"` - Protocol string `protobuf:"bytes,2,opt,name=protocol,proto3" json:"protocol,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *ValidatedEndpoint) Reset() { - *x = ValidatedEndpoint{} - mi := &file_inference_proto_msgTypes[3] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *ValidatedEndpoint) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*ValidatedEndpoint) ProtoMessage() {} - -func (x *ValidatedEndpoint) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[3] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use ValidatedEndpoint.ProtoReflect.Descriptor instead. -func (*ValidatedEndpoint) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{3} -} - -func (x *ValidatedEndpoint) GetUrl() string { - if x != nil { - return x.Url - } - return "" -} - -func (x *ValidatedEndpoint) GetProtocol() string { - if x != nil { - return x.Protocol - } - return "" -} - -type SetInferenceRouteResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` - ProviderName string `protobuf:"bytes,1,opt,name=provider_name,json=providerName,proto3" json:"provider_name,omitempty"` - ModelId string `protobuf:"bytes,2,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` - Version uint64 `protobuf:"varint,3,opt,name=version,proto3" json:"version,omitempty"` - // Route name that was configured. - RouteName string `protobuf:"bytes,4,opt,name=route_name,json=routeName,proto3" json:"route_name,omitempty"` - // Whether endpoint verification ran as part of this request. - ValidationPerformed bool `protobuf:"varint,5,opt,name=validation_performed,json=validationPerformed,proto3" json:"validation_performed,omitempty"` - // The concrete endpoints that were probed during validation, when available. - ValidatedEndpoints []*ValidatedEndpoint `protobuf:"bytes,6,rep,name=validated_endpoints,json=validatedEndpoints,proto3" json:"validated_endpoints,omitempty"` - // Per-route request timeout in seconds that was persisted. - TimeoutSecs uint64 `protobuf:"varint,7,opt,name=timeout_secs,json=timeoutSecs,proto3" json:"timeout_secs,omitempty"` - // Workspace the route was configured in. - Workspace string `protobuf:"bytes,8,opt,name=workspace,proto3" json:"workspace,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *SetInferenceRouteResponse) Reset() { - *x = SetInferenceRouteResponse{} - mi := &file_inference_proto_msgTypes[4] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *SetInferenceRouteResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SetInferenceRouteResponse) ProtoMessage() {} - -func (x *SetInferenceRouteResponse) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[4] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SetInferenceRouteResponse.ProtoReflect.Descriptor instead. -func (*SetInferenceRouteResponse) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{4} -} - -func (x *SetInferenceRouteResponse) GetProviderName() string { - if x != nil { - return x.ProviderName - } - return "" -} - -func (x *SetInferenceRouteResponse) GetModelId() string { - if x != nil { - return x.ModelId - } - return "" -} - -func (x *SetInferenceRouteResponse) GetVersion() uint64 { - if x != nil { - return x.Version - } - return 0 -} - -func (x *SetInferenceRouteResponse) GetRouteName() string { - if x != nil { - return x.RouteName - } - return "" -} - -func (x *SetInferenceRouteResponse) GetValidationPerformed() bool { - if x != nil { - return x.ValidationPerformed - } - return false -} - -func (x *SetInferenceRouteResponse) GetValidatedEndpoints() []*ValidatedEndpoint { - if x != nil { - return x.ValidatedEndpoints - } - return nil -} - -func (x *SetInferenceRouteResponse) GetTimeoutSecs() uint64 { - if x != nil { - return x.TimeoutSecs - } - return 0 -} - -func (x *SetInferenceRouteResponse) GetWorkspace() string { - if x != nil { - return x.Workspace - } - return "" -} - -type GetInferenceRouteRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - // Route name to query. Empty string defaults to "inference.local" (user-facing). - // Use "sandbox-system" for the sandbox system-level inference route. - RouteName string `protobuf:"bytes,1,opt,name=route_name,json=routeName,proto3" json:"route_name,omitempty"` - // Target workspace. Empty string defaults to "default". - Workspace string `protobuf:"bytes,2,opt,name=workspace,proto3" json:"workspace,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *GetInferenceRouteRequest) Reset() { - *x = GetInferenceRouteRequest{} - mi := &file_inference_proto_msgTypes[5] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *GetInferenceRouteRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*GetInferenceRouteRequest) ProtoMessage() {} - -func (x *GetInferenceRouteRequest) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[5] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use GetInferenceRouteRequest.ProtoReflect.Descriptor instead. -func (*GetInferenceRouteRequest) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{5} -} - -func (x *GetInferenceRouteRequest) GetRouteName() string { - if x != nil { - return x.RouteName - } - return "" -} - -func (x *GetInferenceRouteRequest) GetWorkspace() string { - if x != nil { - return x.Workspace - } - return "" -} - -type GetInferenceRouteResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` - ProviderName string `protobuf:"bytes,1,opt,name=provider_name,json=providerName,proto3" json:"provider_name,omitempty"` - ModelId string `protobuf:"bytes,2,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` - Version uint64 `protobuf:"varint,3,opt,name=version,proto3" json:"version,omitempty"` - // Route name that was queried. - RouteName string `protobuf:"bytes,4,opt,name=route_name,json=routeName,proto3" json:"route_name,omitempty"` - // Per-route request timeout in seconds. 0 means default (60s). - TimeoutSecs uint64 `protobuf:"varint,5,opt,name=timeout_secs,json=timeoutSecs,proto3" json:"timeout_secs,omitempty"` - // Workspace the route belongs to. - Workspace string `protobuf:"bytes,6,opt,name=workspace,proto3" json:"workspace,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *GetInferenceRouteResponse) Reset() { - *x = GetInferenceRouteResponse{} - mi := &file_inference_proto_msgTypes[6] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *GetInferenceRouteResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*GetInferenceRouteResponse) ProtoMessage() {} - -func (x *GetInferenceRouteResponse) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[6] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use GetInferenceRouteResponse.ProtoReflect.Descriptor instead. -func (*GetInferenceRouteResponse) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{6} -} - -func (x *GetInferenceRouteResponse) GetProviderName() string { - if x != nil { - return x.ProviderName - } - return "" -} - -func (x *GetInferenceRouteResponse) GetModelId() string { - if x != nil { - return x.ModelId - } - return "" -} - -func (x *GetInferenceRouteResponse) GetVersion() uint64 { - if x != nil { - return x.Version - } - return 0 -} - -func (x *GetInferenceRouteResponse) GetRouteName() string { - if x != nil { - return x.RouteName - } - return "" -} - -func (x *GetInferenceRouteResponse) GetTimeoutSecs() uint64 { - if x != nil { - return x.TimeoutSecs - } - return 0 -} - -func (x *GetInferenceRouteResponse) GetWorkspace() string { - if x != nil { - return x.Workspace - } - return "" -} - -type DeleteInferenceRouteRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - // Route name to delete. Empty string defaults to "inference.local" (user-facing). - // Use "sandbox-system" for the sandbox system-level inference route. - RouteName string `protobuf:"bytes,1,opt,name=route_name,json=routeName,proto3" json:"route_name,omitempty"` - // Target workspace. Empty string defaults to "default". - Workspace string `protobuf:"bytes,2,opt,name=workspace,proto3" json:"workspace,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *DeleteInferenceRouteRequest) Reset() { - *x = DeleteInferenceRouteRequest{} - mi := &file_inference_proto_msgTypes[7] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *DeleteInferenceRouteRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*DeleteInferenceRouteRequest) ProtoMessage() {} - -func (x *DeleteInferenceRouteRequest) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[7] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use DeleteInferenceRouteRequest.ProtoReflect.Descriptor instead. -func (*DeleteInferenceRouteRequest) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{7} -} - -func (x *DeleteInferenceRouteRequest) GetRouteName() string { - if x != nil { - return x.RouteName - } - return "" -} - -func (x *DeleteInferenceRouteRequest) GetWorkspace() string { - if x != nil { - return x.Workspace - } - return "" -} - -type DeleteInferenceRouteResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` - // Whether a route was actually deleted. - Deleted bool `protobuf:"varint,1,opt,name=deleted,proto3" json:"deleted,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *DeleteInferenceRouteResponse) Reset() { - *x = DeleteInferenceRouteResponse{} - mi := &file_inference_proto_msgTypes[8] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *DeleteInferenceRouteResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*DeleteInferenceRouteResponse) ProtoMessage() {} - -func (x *DeleteInferenceRouteResponse) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[8] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use DeleteInferenceRouteResponse.ProtoReflect.Descriptor instead. -func (*DeleteInferenceRouteResponse) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{8} -} - -func (x *DeleteInferenceRouteResponse) GetDeleted() bool { - if x != nil { - return x.Deleted - } - return false -} - -type GetInferenceBundleRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *GetInferenceBundleRequest) Reset() { - *x = GetInferenceBundleRequest{} - mi := &file_inference_proto_msgTypes[9] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *GetInferenceBundleRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*GetInferenceBundleRequest) ProtoMessage() {} - -func (x *GetInferenceBundleRequest) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[9] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use GetInferenceBundleRequest.ProtoReflect.Descriptor instead. -func (*GetInferenceBundleRequest) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{9} -} - -// A single resolved route ready for sandbox-local execution. -type ResolvedRoute struct { - state protoimpl.MessageState `protogen:"open.v1"` - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - BaseUrl string `protobuf:"bytes,2,opt,name=base_url,json=baseUrl,proto3" json:"base_url,omitempty"` - Protocols []string `protobuf:"bytes,3,rep,name=protocols,proto3" json:"protocols,omitempty"` - ApiKey string `protobuf:"bytes,4,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"` - ModelId string `protobuf:"bytes,5,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` - ProviderType string `protobuf:"bytes,6,opt,name=provider_type,json=providerType,proto3" json:"provider_type,omitempty"` - // Per-route request timeout in seconds. 0 means use default (60s). - TimeoutSecs uint64 `protobuf:"varint,7,opt,name=timeout_secs,json=timeoutSecs,proto3" json:"timeout_secs,omitempty"` - // When true, the model identifier is embedded in the URL path (e.g. Vertex AI). - ModelInPath bool `protobuf:"varint,8,opt,name=model_in_path,json=modelInPath,proto3" json:"model_in_path,omitempty"` - // Optional override for the request path. When set, replaces the protocol-derived path. - // An empty string means POST directly to base_url/model_id with no additional path. - RequestPathOverride *string `protobuf:"bytes,9,opt,name=request_path_override,json=requestPathOverride,proto3,oneof" json:"request_path_override,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *ResolvedRoute) Reset() { - *x = ResolvedRoute{} - mi := &file_inference_proto_msgTypes[10] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *ResolvedRoute) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*ResolvedRoute) ProtoMessage() {} - -func (x *ResolvedRoute) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[10] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use ResolvedRoute.ProtoReflect.Descriptor instead. -func (*ResolvedRoute) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{10} -} - -func (x *ResolvedRoute) GetName() string { - if x != nil { - return x.Name - } - return "" -} - -func (x *ResolvedRoute) GetBaseUrl() string { - if x != nil { - return x.BaseUrl - } - return "" -} - -func (x *ResolvedRoute) GetProtocols() []string { - if x != nil { - return x.Protocols - } - return nil -} - -func (x *ResolvedRoute) GetApiKey() string { - if x != nil { - return x.ApiKey - } - return "" -} - -func (x *ResolvedRoute) GetModelId() string { - if x != nil { - return x.ModelId - } - return "" -} - -func (x *ResolvedRoute) GetProviderType() string { - if x != nil { - return x.ProviderType - } - return "" -} - -func (x *ResolvedRoute) GetTimeoutSecs() uint64 { - if x != nil { - return x.TimeoutSecs - } - return 0 -} - -func (x *ResolvedRoute) GetModelInPath() bool { - if x != nil { - return x.ModelInPath - } - return false -} - -func (x *ResolvedRoute) GetRequestPathOverride() string { - if x != nil && x.RequestPathOverride != nil { - return *x.RequestPathOverride - } - return "" -} - -type GetInferenceBundleResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` - Routes []*ResolvedRoute `protobuf:"bytes,1,rep,name=routes,proto3" json:"routes,omitempty"` - // Opaque revision tag for cache freshness checks. - Revision string `protobuf:"bytes,2,opt,name=revision,proto3" json:"revision,omitempty"` - // Timestamp (epoch ms) when this bundle was generated. - GeneratedAtMs int64 `protobuf:"varint,3,opt,name=generated_at_ms,json=generatedAtMs,proto3" json:"generated_at_ms,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *GetInferenceBundleResponse) Reset() { - *x = GetInferenceBundleResponse{} - mi := &file_inference_proto_msgTypes[11] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *GetInferenceBundleResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*GetInferenceBundleResponse) ProtoMessage() {} - -func (x *GetInferenceBundleResponse) ProtoReflect() protoreflect.Message { - mi := &file_inference_proto_msgTypes[11] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use GetInferenceBundleResponse.ProtoReflect.Descriptor instead. -func (*GetInferenceBundleResponse) Descriptor() ([]byte, []int) { - return file_inference_proto_rawDescGZIP(), []int{11} -} - -func (x *GetInferenceBundleResponse) GetRoutes() []*ResolvedRoute { - if x != nil { - return x.Routes - } - return nil -} - -func (x *GetInferenceBundleResponse) GetRevision() string { - if x != nil { - return x.Revision - } - return "" -} - -func (x *GetInferenceBundleResponse) GetGeneratedAtMs() int64 { - if x != nil { - return x.GeneratedAtMs - } - return 0 -} - -var File_inference_proto protoreflect.FileDescriptor - -const file_inference_proto_rawDesc = "" + - "\n" + - "\x0finference.proto\x12\x16openshell.inference.v1\x1a\x0fdatamodel.proto\x1a\roptions.proto\"y\n" + - "\x14InferenceRouteConfig\x12#\n" + - "\rprovider_name\x18\x01 \x01(\tR\fproviderName\x12\x19\n" + - "\bmodel_id\x18\x02 \x01(\tR\amodelId\x12!\n" + - "\ftimeout_secs\x18\x03 \x01(\x04R\vtimeoutSecs\"\xb0\x01\n" + - "\x0eInferenceRoute\x12>\n" + - "\bmetadata\x18\x01 \x01(\v2\".openshell.datamodel.v1.ObjectMetaR\bmetadata\x12D\n" + - "\x06config\x18\x02 \x01(\v2,.openshell.inference.v1.InferenceRouteConfigR\x06config\x12\x18\n" + - "\aversion\x18\x03 \x01(\x04R\aversion\"\xef\x01\n" + - "\x18SetInferenceRouteRequest\x12#\n" + - "\rprovider_name\x18\x01 \x01(\tR\fproviderName\x12\x19\n" + - "\bmodel_id\x18\x02 \x01(\tR\amodelId\x12\x1d\n" + - "\n" + - "route_name\x18\x03 \x01(\tR\trouteName\x12\x16\n" + - "\x06verify\x18\x04 \x01(\bR\x06verify\x12\x1b\n" + - "\tno_verify\x18\x05 \x01(\bR\bnoVerify\x12!\n" + - "\ftimeout_secs\x18\x06 \x01(\x04R\vtimeoutSecs\x12\x1c\n" + - "\tworkspace\x18\a \x01(\tR\tworkspace\"A\n" + - "\x11ValidatedEndpoint\x12\x10\n" + - "\x03url\x18\x01 \x01(\tR\x03url\x12\x1a\n" + - "\bprotocol\x18\x02 \x01(\tR\bprotocol\"\xe4\x02\n" + - "\x19SetInferenceRouteResponse\x12#\n" + - "\rprovider_name\x18\x01 \x01(\tR\fproviderName\x12\x19\n" + - "\bmodel_id\x18\x02 \x01(\tR\amodelId\x12\x18\n" + - "\aversion\x18\x03 \x01(\x04R\aversion\x12\x1d\n" + - "\n" + - "route_name\x18\x04 \x01(\tR\trouteName\x121\n" + - "\x14validation_performed\x18\x05 \x01(\bR\x13validationPerformed\x12Z\n" + - "\x13validated_endpoints\x18\x06 \x03(\v2).openshell.inference.v1.ValidatedEndpointR\x12validatedEndpoints\x12!\n" + - "\ftimeout_secs\x18\a \x01(\x04R\vtimeoutSecs\x12\x1c\n" + - "\tworkspace\x18\b \x01(\tR\tworkspace\"W\n" + - "\x18GetInferenceRouteRequest\x12\x1d\n" + - "\n" + - "route_name\x18\x01 \x01(\tR\trouteName\x12\x1c\n" + - "\tworkspace\x18\x02 \x01(\tR\tworkspace\"\xd5\x01\n" + - "\x19GetInferenceRouteResponse\x12#\n" + - "\rprovider_name\x18\x01 \x01(\tR\fproviderName\x12\x19\n" + - "\bmodel_id\x18\x02 \x01(\tR\amodelId\x12\x18\n" + - "\aversion\x18\x03 \x01(\x04R\aversion\x12\x1d\n" + - "\n" + - "route_name\x18\x04 \x01(\tR\trouteName\x12!\n" + - "\ftimeout_secs\x18\x05 \x01(\x04R\vtimeoutSecs\x12\x1c\n" + - "\tworkspace\x18\x06 \x01(\tR\tworkspace\"Z\n" + - "\x1bDeleteInferenceRouteRequest\x12\x1d\n" + - "\n" + - "route_name\x18\x01 \x01(\tR\trouteName\x12\x1c\n" + - "\tworkspace\x18\x02 \x01(\tR\tworkspace\"8\n" + - "\x1cDeleteInferenceRouteResponse\x12\x18\n" + - "\adeleted\x18\x01 \x01(\bR\adeleted\"\x1b\n" + - "\x19GetInferenceBundleRequest\"\xd5\x02\n" + - "\rResolvedRoute\x12\x12\n" + - "\x04name\x18\x01 \x01(\tR\x04name\x12\x19\n" + - "\bbase_url\x18\x02 \x01(\tR\abaseUrl\x12\x1c\n" + - "\tprotocols\x18\x03 \x03(\tR\tprotocols\x12\x1d\n" + - "\aapi_key\x18\x04 \x01(\tB\x04\x88\xb5\x18\x01R\x06apiKey\x12\x19\n" + - "\bmodel_id\x18\x05 \x01(\tR\amodelId\x12#\n" + - "\rprovider_type\x18\x06 \x01(\tR\fproviderType\x12!\n" + - "\ftimeout_secs\x18\a \x01(\x04R\vtimeoutSecs\x12\"\n" + - "\rmodel_in_path\x18\b \x01(\bR\vmodelInPath\x127\n" + - "\x15request_path_override\x18\t \x01(\tH\x00R\x13requestPathOverride\x88\x01\x01B\x18\n" + - "\x16_request_path_override\"\x9f\x01\n" + - "\x1aGetInferenceBundleResponse\x12=\n" + - "\x06routes\x18\x01 \x03(\v2%.openshell.inference.v1.ResolvedRouteR\x06routes\x12\x1a\n" + - "\brevision\x18\x02 \x01(\tR\brevision\x12&\n" + - "\x0fgenerated_at_ms\x18\x03 \x01(\x03R\rgeneratedAtMs2\x82\x05\n" + - "\tInference\x12\x8a\x01\n" + - "\x12GetInferenceBundle\x121.openshell.inference.v1.GetInferenceBundleRequest\x1a2.openshell.inference.v1.GetInferenceBundleResponse\"\r\x82\xb5\x18\t\n" + - "\asandbox\x12\x9e\x01\n" + - "\x11SetInferenceRoute\x120.openshell.inference.v1.SetInferenceRouteRequest\x1a1.openshell.inference.v1.SetInferenceRouteResponse\"$\x82\xb5\x18 \n" + - "\x06bearer\x12\x05admin\"\x0finference:write\x12\x9c\x01\n" + - "\x11GetInferenceRoute\x120.openshell.inference.v1.GetInferenceRouteRequest\x1a1.openshell.inference.v1.GetInferenceRouteResponse\"\"\x82\xb5\x18\x1e\n" + - "\x06bearer\x12\x04user\"\x0einference:read\x12\xa7\x01\n" + - "\x14DeleteInferenceRoute\x123.openshell.inference.v1.DeleteInferenceRouteRequest\x1a4.openshell.inference.v1.DeleteInferenceRouteResponse\"$\x82\xb5\x18 \n" + - "\x06bearer\x12\x05admin\"\x0finference:writeb\x06proto3" - -var ( - file_inference_proto_rawDescOnce sync.Once - file_inference_proto_rawDescData []byte -) - -func file_inference_proto_rawDescGZIP() []byte { - file_inference_proto_rawDescOnce.Do(func() { - file_inference_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_inference_proto_rawDesc), len(file_inference_proto_rawDesc))) - }) - return file_inference_proto_rawDescData -} - -var file_inference_proto_msgTypes = make([]protoimpl.MessageInfo, 12) -var file_inference_proto_goTypes = []any{ - (*InferenceRouteConfig)(nil), // 0: openshell.inference.v1.InferenceRouteConfig - (*InferenceRoute)(nil), // 1: openshell.inference.v1.InferenceRoute - (*SetInferenceRouteRequest)(nil), // 2: openshell.inference.v1.SetInferenceRouteRequest - (*ValidatedEndpoint)(nil), // 3: openshell.inference.v1.ValidatedEndpoint - (*SetInferenceRouteResponse)(nil), // 4: openshell.inference.v1.SetInferenceRouteResponse - (*GetInferenceRouteRequest)(nil), // 5: openshell.inference.v1.GetInferenceRouteRequest - (*GetInferenceRouteResponse)(nil), // 6: openshell.inference.v1.GetInferenceRouteResponse - (*DeleteInferenceRouteRequest)(nil), // 7: openshell.inference.v1.DeleteInferenceRouteRequest - (*DeleteInferenceRouteResponse)(nil), // 8: openshell.inference.v1.DeleteInferenceRouteResponse - (*GetInferenceBundleRequest)(nil), // 9: openshell.inference.v1.GetInferenceBundleRequest - (*ResolvedRoute)(nil), // 10: openshell.inference.v1.ResolvedRoute - (*GetInferenceBundleResponse)(nil), // 11: openshell.inference.v1.GetInferenceBundleResponse - (*datamodelv1.ObjectMeta)(nil), // 12: openshell.datamodel.v1.ObjectMeta -} -var file_inference_proto_depIdxs = []int32{ - 12, // 0: openshell.inference.v1.InferenceRoute.metadata:type_name -> openshell.datamodel.v1.ObjectMeta - 0, // 1: openshell.inference.v1.InferenceRoute.config:type_name -> openshell.inference.v1.InferenceRouteConfig - 3, // 2: openshell.inference.v1.SetInferenceRouteResponse.validated_endpoints:type_name -> openshell.inference.v1.ValidatedEndpoint - 10, // 3: openshell.inference.v1.GetInferenceBundleResponse.routes:type_name -> openshell.inference.v1.ResolvedRoute - 9, // 4: openshell.inference.v1.Inference.GetInferenceBundle:input_type -> openshell.inference.v1.GetInferenceBundleRequest - 2, // 5: openshell.inference.v1.Inference.SetInferenceRoute:input_type -> openshell.inference.v1.SetInferenceRouteRequest - 5, // 6: openshell.inference.v1.Inference.GetInferenceRoute:input_type -> openshell.inference.v1.GetInferenceRouteRequest - 7, // 7: openshell.inference.v1.Inference.DeleteInferenceRoute:input_type -> openshell.inference.v1.DeleteInferenceRouteRequest - 11, // 8: openshell.inference.v1.Inference.GetInferenceBundle:output_type -> openshell.inference.v1.GetInferenceBundleResponse - 4, // 9: openshell.inference.v1.Inference.SetInferenceRoute:output_type -> openshell.inference.v1.SetInferenceRouteResponse - 6, // 10: openshell.inference.v1.Inference.GetInferenceRoute:output_type -> openshell.inference.v1.GetInferenceRouteResponse - 8, // 11: openshell.inference.v1.Inference.DeleteInferenceRoute:output_type -> openshell.inference.v1.DeleteInferenceRouteResponse - 8, // [8:12] is the sub-list for method output_type - 4, // [4:8] is the sub-list for method input_type - 4, // [4:4] is the sub-list for extension type_name - 4, // [4:4] is the sub-list for extension extendee - 0, // [0:4] is the sub-list for field type_name -} - -func init() { file_inference_proto_init() } -func file_inference_proto_init() { - if File_inference_proto != nil { - return - } - file_inference_proto_msgTypes[10].OneofWrappers = []any{} - type x struct{} - out := protoimpl.TypeBuilder{ - File: protoimpl.DescBuilder{ - GoPackagePath: reflect.TypeOf(x{}).PkgPath(), - RawDescriptor: unsafe.Slice(unsafe.StringData(file_inference_proto_rawDesc), len(file_inference_proto_rawDesc)), - NumEnums: 0, - NumMessages: 12, - NumExtensions: 0, - NumServices: 1, - }, - GoTypes: file_inference_proto_goTypes, - DependencyIndexes: file_inference_proto_depIdxs, - MessageInfos: file_inference_proto_msgTypes, - }.Build() - File_inference_proto = out.File - file_inference_proto_goTypes = nil - file_inference_proto_depIdxs = nil -} diff --git a/sdk/go/proto/inferencev1/inference_grpc.pb.go b/sdk/go/proto/inferencev1/inference_grpc.pb.go deleted file mode 100644 index 61f74348c0..0000000000 --- a/sdk/go/proto/inferencev1/inference_grpc.pb.go +++ /dev/null @@ -1,256 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -// Code generated by protoc-gen-go-grpc. DO NOT EDIT. -// versions: -// - protoc-gen-go-grpc v1.6.2 -// - protoc (unknown) -// source: inference.proto - -package inferencev1 - -import ( - context "context" - grpc "google.golang.org/grpc" - codes "google.golang.org/grpc/codes" - status "google.golang.org/grpc/status" -) - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the grpc package it is being compiled against. -// Requires gRPC-Go v1.64.0 or later. -const _ = grpc.SupportPackageIsVersion9 - -const ( - Inference_GetInferenceBundle_FullMethodName = "/openshell.inference.v1.Inference/GetInferenceBundle" - Inference_SetInferenceRoute_FullMethodName = "/openshell.inference.v1.Inference/SetInferenceRoute" - Inference_GetInferenceRoute_FullMethodName = "/openshell.inference.v1.Inference/GetInferenceRoute" - Inference_DeleteInferenceRoute_FullMethodName = "/openshell.inference.v1.Inference/DeleteInferenceRoute" -) - -// InferenceClient is the client API for Inference service. -// -// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. -// -// Inference service provides workspace-scoped inference route configuration and bundle delivery. -type InferenceClient interface { - // Return the resolved inference route bundle for sandbox-local execution. - GetInferenceBundle(ctx context.Context, in *GetInferenceBundleRequest, opts ...grpc.CallOption) (*GetInferenceBundleResponse, error) - // Set the inference route for a workspace. - // - // This controls how requests sent to `inference.local` are routed - // for sandboxes in the specified workspace. - SetInferenceRoute(ctx context.Context, in *SetInferenceRouteRequest, opts ...grpc.CallOption) (*SetInferenceRouteResponse, error) - // Get the inference route for a workspace. - GetInferenceRoute(ctx context.Context, in *GetInferenceRouteRequest, opts ...grpc.CallOption) (*GetInferenceRouteResponse, error) - // Delete an inference route from a workspace. - DeleteInferenceRoute(ctx context.Context, in *DeleteInferenceRouteRequest, opts ...grpc.CallOption) (*DeleteInferenceRouteResponse, error) -} - -type inferenceClient struct { - cc grpc.ClientConnInterface -} - -func NewInferenceClient(cc grpc.ClientConnInterface) InferenceClient { - return &inferenceClient{cc} -} - -func (c *inferenceClient) GetInferenceBundle(ctx context.Context, in *GetInferenceBundleRequest, opts ...grpc.CallOption) (*GetInferenceBundleResponse, error) { - cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(GetInferenceBundleResponse) - err := c.cc.Invoke(ctx, Inference_GetInferenceBundle_FullMethodName, in, out, cOpts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *inferenceClient) SetInferenceRoute(ctx context.Context, in *SetInferenceRouteRequest, opts ...grpc.CallOption) (*SetInferenceRouteResponse, error) { - cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(SetInferenceRouteResponse) - err := c.cc.Invoke(ctx, Inference_SetInferenceRoute_FullMethodName, in, out, cOpts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *inferenceClient) GetInferenceRoute(ctx context.Context, in *GetInferenceRouteRequest, opts ...grpc.CallOption) (*GetInferenceRouteResponse, error) { - cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(GetInferenceRouteResponse) - err := c.cc.Invoke(ctx, Inference_GetInferenceRoute_FullMethodName, in, out, cOpts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *inferenceClient) DeleteInferenceRoute(ctx context.Context, in *DeleteInferenceRouteRequest, opts ...grpc.CallOption) (*DeleteInferenceRouteResponse, error) { - cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(DeleteInferenceRouteResponse) - err := c.cc.Invoke(ctx, Inference_DeleteInferenceRoute_FullMethodName, in, out, cOpts...) - if err != nil { - return nil, err - } - return out, nil -} - -// InferenceServer is the server API for Inference service. -// All implementations must embed UnimplementedInferenceServer -// for forward compatibility. -// -// Inference service provides workspace-scoped inference route configuration and bundle delivery. -type InferenceServer interface { - // Return the resolved inference route bundle for sandbox-local execution. - GetInferenceBundle(context.Context, *GetInferenceBundleRequest) (*GetInferenceBundleResponse, error) - // Set the inference route for a workspace. - // - // This controls how requests sent to `inference.local` are routed - // for sandboxes in the specified workspace. - SetInferenceRoute(context.Context, *SetInferenceRouteRequest) (*SetInferenceRouteResponse, error) - // Get the inference route for a workspace. - GetInferenceRoute(context.Context, *GetInferenceRouteRequest) (*GetInferenceRouteResponse, error) - // Delete an inference route from a workspace. - DeleteInferenceRoute(context.Context, *DeleteInferenceRouteRequest) (*DeleteInferenceRouteResponse, error) - mustEmbedUnimplementedInferenceServer() -} - -// UnimplementedInferenceServer must be embedded to have -// forward compatible implementations. -// -// NOTE: this should be embedded by value instead of pointer to avoid a nil -// pointer dereference when methods are called. -type UnimplementedInferenceServer struct{} - -func (UnimplementedInferenceServer) GetInferenceBundle(context.Context, *GetInferenceBundleRequest) (*GetInferenceBundleResponse, error) { - return nil, status.Error(codes.Unimplemented, "method GetInferenceBundle not implemented") -} -func (UnimplementedInferenceServer) SetInferenceRoute(context.Context, *SetInferenceRouteRequest) (*SetInferenceRouteResponse, error) { - return nil, status.Error(codes.Unimplemented, "method SetInferenceRoute not implemented") -} -func (UnimplementedInferenceServer) GetInferenceRoute(context.Context, *GetInferenceRouteRequest) (*GetInferenceRouteResponse, error) { - return nil, status.Error(codes.Unimplemented, "method GetInferenceRoute not implemented") -} -func (UnimplementedInferenceServer) DeleteInferenceRoute(context.Context, *DeleteInferenceRouteRequest) (*DeleteInferenceRouteResponse, error) { - return nil, status.Error(codes.Unimplemented, "method DeleteInferenceRoute not implemented") -} -func (UnimplementedInferenceServer) mustEmbedUnimplementedInferenceServer() {} -func (UnimplementedInferenceServer) testEmbeddedByValue() {} - -// UnsafeInferenceServer may be embedded to opt out of forward compatibility for this service. -// Use of this interface is not recommended, as added methods to InferenceServer will -// result in compilation errors. -type UnsafeInferenceServer interface { - mustEmbedUnimplementedInferenceServer() -} - -func RegisterInferenceServer(s grpc.ServiceRegistrar, srv InferenceServer) { - // If the following call panics, it indicates UnimplementedInferenceServer was - // embedded by pointer and is nil. This will cause panics if an - // unimplemented method is ever invoked, so we test this at initialization - // time to prevent it from happening at runtime later due to I/O. - if t, ok := srv.(interface{ testEmbeddedByValue() }); ok { - t.testEmbeddedByValue() - } - s.RegisterService(&Inference_ServiceDesc, srv) -} - -func _Inference_GetInferenceBundle_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(GetInferenceBundleRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(InferenceServer).GetInferenceBundle(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: Inference_GetInferenceBundle_FullMethodName, - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(InferenceServer).GetInferenceBundle(ctx, req.(*GetInferenceBundleRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Inference_SetInferenceRoute_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(SetInferenceRouteRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(InferenceServer).SetInferenceRoute(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: Inference_SetInferenceRoute_FullMethodName, - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(InferenceServer).SetInferenceRoute(ctx, req.(*SetInferenceRouteRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Inference_GetInferenceRoute_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(GetInferenceRouteRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(InferenceServer).GetInferenceRoute(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: Inference_GetInferenceRoute_FullMethodName, - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(InferenceServer).GetInferenceRoute(ctx, req.(*GetInferenceRouteRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Inference_DeleteInferenceRoute_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(DeleteInferenceRouteRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(InferenceServer).DeleteInferenceRoute(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: Inference_DeleteInferenceRoute_FullMethodName, - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(InferenceServer).DeleteInferenceRoute(ctx, req.(*DeleteInferenceRouteRequest)) - } - return interceptor(ctx, in, info, handler) -} - -// Inference_ServiceDesc is the grpc.ServiceDesc for Inference service. -// It's only intended for direct use with grpc.RegisterService, -// and not to be introspected or modified (even as a copy) -var Inference_ServiceDesc = grpc.ServiceDesc{ - ServiceName: "openshell.inference.v1.Inference", - HandlerType: (*InferenceServer)(nil), - Methods: []grpc.MethodDesc{ - { - MethodName: "GetInferenceBundle", - Handler: _Inference_GetInferenceBundle_Handler, - }, - { - MethodName: "SetInferenceRoute", - Handler: _Inference_SetInferenceRoute_Handler, - }, - { - MethodName: "GetInferenceRoute", - Handler: _Inference_GetInferenceRoute_Handler, - }, - { - MethodName: "DeleteInferenceRoute", - Handler: _Inference_DeleteInferenceRoute_Handler, - }, - }, - Streams: []grpc.StreamDesc{}, - Metadata: "inference.proto", -} diff --git a/sdk/go/proto/openshellv1/openshell.pb.go b/sdk/go/proto/openshellv1/openshell.pb.go index f9e7f39030..fe1f88cb71 100644 --- a/sdk/go/proto/openshellv1/openshell.pb.go +++ b/sdk/go/proto/openshellv1/openshell.pb.go @@ -8921,7 +8921,7 @@ type UpdateConfigRequest struct { // The new policy to apply. // // Sandbox scope (`global=false`): - // - only network_policies and inference fields may differ from create-time + // - only network_policies may differ from create-time // policy; static fields must match version 1. // // Global scope (`global=true`): diff --git a/sdk/typescript/buf.gen.yaml b/sdk/typescript/buf.gen.yaml index 757f0bd73e..e4caeb64da 100644 --- a/sdk/typescript/buf.gen.yaml +++ b/sdk/typescript/buf.gen.yaml @@ -5,7 +5,7 @@ # validation policy live in the repo-level buf.yaml; this template only drives # generation. buf compiles the module with its own compiler (no protoc) and # runs the connect-es plugin from this package's devDependencies. Limited to -# the client-surface closure so we don't emit the unused inference/compute/test +# the client-surface closure so we don't emit unused compute/test # protos; well-known types resolve through @bufbuild/protobuf/wkt and are not # generated. version: v2 diff --git a/skills/debug-inference/SKILL.md b/skills/debug-inference/SKILL.md index c26dff37c6..c888bfb043 100644 --- a/skills/debug-inference/SKILL.md +++ b/skills/debug-inference/SKILL.md @@ -1,415 +1,130 @@ --- name: debug-inference -description: Debug why inference.local, direct external inference, or supervisor-only system inference is failing. Use when the user cannot reach a local model server, has provider base URL issues, sees inference verification failures, hits protocol mismatches, or needs to diagnose inference on local vs remote gateways. Trigger keywords - debug inference, inference.local, system inference, sandbox-system, local inference, ollama, vllm, sglang, trtllm, NIM, inference failing, model server unreachable, failed to verify inference endpoint, host.openshell.internal. +description: Debug inference clients that use an attached provider and its native endpoint, including hosted APIs and host-local Ollama, vLLM, SGLang, TRT-LLM, LM Studio, or NIM. Use for provider attachment, endpoint policy, credential substitution, topology, and migration from the removed inference.local endpoint. Trigger keywords - debug inference, inference.local, local inference, ollama, lm studio, vllm, sglang, trtllm, NIM, inference failing, model server unreachable, credential_endpoint_mismatch, host.openshell.internal. --- # Debug Inference -Diagnose why OpenShell inference is failing and recommend exact fix commands. +Diagnose inference as ordinary provider-authorized network traffic. OpenShell no +longer supplies a managed inference route, rewrites request shapes, or selects a +model. The application calls the provider's native endpoint and owns its base +URL, model, request format, and timeout. -Use `openshell` CLI commands to inspect the active gateway, provider records, managed inference config, and sandbox behavior. Use a short sandbox probe when needed to confirm end-to-end routing. +Use installed `openshell --help` output as the authority for command syntax. +Refer to the published [provider management guide](https://docs.nvidia.com/openshell/latest/sandboxes/manage-providers.md) +and [provider profile guide](https://docs.nvidia.com/openshell/latest/providers/profiles.md) +for current behavior. -## Overview +## Diagnostic Workflow -OpenShell supports three inference paths. Diagnose the correct one first. - -1. **Managed inference** through `https://inference.local` - - Configured by `openshell inference set` - - Shared by every sandbox on the active gateway - - Credentials and model are injected by OpenShell -2. **Direct external inference** to hosts like `api.openai.com` - - Controlled by `network_policies` - - Requires the application to call the external host directly - - Requires provider attachment and network access to be configured separately -3. **System inference** used by platform functions - - Configured by `openshell inference set --system` - - Uses the `sandbox-system` route - - Consumed in-process by the sandbox supervisor and not exposed to sandbox user code through `inference.local` - -For local or self-hosted engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments, the most common managed inference pattern is an `openai` provider with `OPENAI_BASE_URL` pointing at a host the gateway can reach. - -## Prerequisites - -- `openshell` is on the PATH -- The active gateway is running -- You know the failing setup, or can infer it from commands and config - -Use `openshell --help` and nested `--help` output as the authority for the installed CLI version. Use the published [inference routing guide](https://docs.nvidia.com/openshell/latest/sandboxes/inference-routing.md) for route behavior and [provider management guide](https://docs.nvidia.com/openshell/latest/sandboxes/manage-providers.md) for credential and profile concepts. - -## Tools Available - -Use these commands first: +### 1. Confirm Gateway and Sandbox Context ```bash -# Which gateway is active, and can the CLI reach it? openshell status - -# Show both the user-facing and system inference routes -openshell inference get - -# Show only the supervisor-only system route -openshell inference get --system - -# Inspect the provider record referenced by the relevant route -openshell provider get - -# Inspect gateway topology details when remote/local confusion is suspected openshell gateway info - -# Run a minimal end-to-end probe from a sandbox -openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' -``` - -## Workflow - -When the user asks to debug inference, run diagnostics automatically in this order. Stop and report findings as soon as a root cause is identified. - -### Determine Context - -Establish these facts first: - -1. Is sandbox code calling `https://inference.local`, is the application calling a direct external host, or is a platform function using system inference? -2. Which gateway is active, and is it local, remote, or cloud? -3. Which provider, model, and timeout are configured for the relevant route? -4. Is the upstream local to the gateway host, or somewhere else? - -### Step 0: Check the Active Gateway - -Run: - -```bash -openshell status -openshell gateway info -``` - -Look for: - -- Active gateway name and endpoint -- Whether the gateway is local or remote -- Whether `host.openshell.internal` would point to the local machine or a remote host - -Common mistake: - -- **Laptop-local model + remote gateway**: `host.openshell.internal` points to the remote gateway host, not your laptop. A laptop-local Ollama or vLLM server will not be reachable without a tunnel or shared reachable network path. - -### Step 1: Check Whether the Relevant Route Is Configured - -Run: - -```bash -openshell inference get -openshell inference get --system +openshell sandbox get ``` -Interpretation: - -- `openshell inference get` shows both the user-facing `inference.local` route and the system route. `--system` isolates the system route. -- **The `inference.local` route is `Not configured`**: managed inference has no backend. Configure it without `--system`: - - ```bash - openshell inference set --provider --model - ``` - -- **System inference is `Not configured`**: platform functions have no system backend. Configure it separately: - - ```bash - openshell inference set --system --provider --model - ``` +For a host-local model server, `host.openshell.internal` identifies the machine +running the gateway. It does not identify the operator's laptop when the gateway +is remote. A server listening only on `127.0.0.1` may also be unreachable from a +container; bind it to an address reachable from the gateway runtime. -- **Provider, model, and timeout shown**: Continue to provider inspection for the relevant route. - -### Step 2: Inspect the Provider Record - -Run: +### 2. Inspect the Provider and Its Profile ```bash -openshell provider get +openshell provider get +openshell provider profile export -o yaml ``` -Check: - -- Provider type matches the client API shape and is supported for managed inference - - `openai` for OpenAI-compatible engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments - - `anthropic` for Anthropic Messages API - - `nvidia` for NVIDIA-hosted OpenAI-compatible endpoints - - `deepinfra` for DeepInfra's OpenAI-compatible endpoint - - `google-vertex-ai` for Vertex AI; Claude models use Anthropic Messages and other models use OpenAI Chat Completions - - `aws-bedrock` only through a configured Bedrock-compatible bridge today -- Required credential key exists -- `*_BASE_URL` override is correct when using a self-hosted endpoint - -An `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` override is route-only. It does not -authorize the fixed public-vendor endpoint or expose the alternate-upstream key -for substitution there. Use an imported endpoint-bearing custom profile when a -sandbox needs direct access to the alternate upstream. - -Fix examples: - -```bash -openshell provider create --name ollama --type openai --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 - -openshell provider update ollama --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 -``` - -`provider update` preserves the provider type and does not accept `--type`. Prefer bare credential keys, such as `--credential OPENAI_API_KEY`, when reading a real secret from the CLI environment. - -### Step 3: Check Local Host Reachability - -For host-backed local inference, confirm the upstream server: - -- Binds to `0.0.0.0`, not only `127.0.0.1` -- Runs on the same machine as the gateway -- Is reachable through `host.openshell.internal`, the host's LAN IP, or another reachable hostname - -Common mistakes: - -- **Base URL uses `127.0.0.1` or `localhost`**: usually wrong for managed inference. Replace with `host.openshell.internal` or the host's LAN IP. -- **Server binds only to loopback**: reconfigure it to bind to `0.0.0.0`. -- **Inference engine runs as a system service**: changing the bind address may require updating the service configuration and restarting the service before the new listener becomes reachable. - -### Step 4: Check Request Shape - -User-facing managed inference only works for `https://inference.local` and supported inference API paths. - -Supported patterns include: - -- `POST /v1/chat/completions` -- `POST /v1/completions` -- `POST /v1/responses` -- `POST /v1/embeddings` -- `POST /v1/messages` -- `GET /v1/models` -- `GET /v1/models/*` -- `POST /model/{modelId}/invoke` for bridge-fronted `aws-bedrock` - -Common mistakes: - -- **Wrong scheme**: `http://inference.local` instead of `https://inference.local` -- **Unsupported path**: request does not match a known inference API -- **Protocol mismatch**: Anthropic client against an `openai` provider, or vice versa -- **Provider-specific mismatch**: Vertex Claude requests must use `/v1/messages`; other Vertex models currently use `/v1/chat/completions`; Bedrock uses its model-in-path invoke shape +Check that the profile: -Fix guidance: +- Names the exact endpoint host, port, and protocol the client calls. +- Allows the client binary. +- Declares the credential key and intended authentication style. +- Uses narrow HTTP rules when the provider should expose only part of an API. -- Use a supported path and provider type -- Point OpenAI-compatible SDKs at `https://inference.local/v1` -- If the SDK requires an API key, pass any non-empty placeholder such as `test` - -### Step 5: Probe from a Sandbox - -This probe validates the user-facing `inference.local` route. It does not exercise supervisor-only system inference. - -Run a minimal request from inside a sandbox: +For a custom or self-hosted OpenAI-compatible endpoint, import an +endpoint-bearing profile. A base URL stored only in provider configuration does +not authorize a new endpoint. ```bash -openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' +openshell provider profile lint -f ./provider-profile.yaml +openshell provider profile import -f ./provider-profile.yaml +openshell provider create --name --type ``` -Interpretation: - -- **`cluster inference is not configured`**: set the managed gateway route with `openshell inference set` -- **`connection not allowed by policy`** on `inference.local`: unsupported method or path -- **`no compatible route`**: provider type and client API shape do not match -- **Connection refused / upstream unavailable / verification failures**: base URL, bind address, topology, or credentials are wrong +Add the required `--credential KEY` or `--credential KEY=VALUE` arguments shown +by the profile. Never broaden endpoint policy merely to silence a credential +binding error. -For system inference failures, inspect the platform function and sandbox supervisor/network logs after confirming `openshell inference get --system`. User code cannot call the `sandbox-system` route directly. - -### Step 6: Reapply or Repair the Managed Route - -After fixing the provider, use `update` for a partial change or `set` to replace the route: +### 3. Confirm Attachment ```bash -openshell inference set --provider --model -openshell inference update --provider -openshell inference update --model -openshell inference update --timeout 120 +openshell sandbox provider list +openshell sandbox provider attach ``` -Add `--system` to target the system route. Without it, these commands target `inference.local`. A timeout of `0` uses the 60-second default; increase it for models with long reasoning or idle streaming phases. - -If the endpoint is intentionally offline and you only want to save the config: +Launch a new process after attaching a provider so it inherits newly available +credential placeholders: ```bash -openshell inference set --provider --model --no-verify +openshell sandbox exec -- env ``` -Use `--no-verify` only when the endpoint is intentionally offline or the provider protocol cannot be verified, such as the current bridge-fronted Bedrock flow. Inference updates are hot-reloaded to running sandboxes within about 5 seconds by default. - -### Step 7: Diagnose Direct External Inference - -If the application calls `api.openai.com`, `api.anthropic.com`, or another external host directly, this is not a managed inference issue. - -Check instead: - -1. The application is configured to call the external hostname directly -2. A provider with the needed credentials exists -3. The sandbox has that provider attached (`openshell sandbox provider list [name]`) -4. `network_policies` allow that host, port, and HTTP rules - -If the response reports `credential_endpoint_mismatch`, the provider is attached -but its credential profile does not authorize that request recipient. Run -`openshell provider get ` to identify the provider type, then -inspect its profile endpoints with -`openshell provider profile export -o yaml`. That export uses the current -workspace scope; add `--global` when the provider was created with -`--global-profile`. Compare the profile's endpoint host, port, and path with the -direct request. Correct the provider selection or profile endpoint when that -recipient is intentional. Do not widen the sandbox network policy to work around -the mismatch: policy admission and credential endpoint authorization are -separate checks, and the provider profile should authorize only intended -credential recipients. - -If the response reports `request_authority_mismatch`, compare the HTTP request -authority with the CONNECT tunnel endpoint. The host and effective port must -match. For a tunnel to `api.example.com:8443`, send -`Host: api.example.com:8443`; omitting the non-default port makes the request -authority use the transport default and OpenShell rejects it. An absolute-form -request target must use the same authority. - -Attach or detach a provider on an existing sandbox with `openshell sandbox provider attach ` and `openshell sandbox provider detach `. - -Use the `generate-sandbox-policy` skill when the user needs help authoring policy YAML. - -## Fix: Local Host Inference Timeouts (Firewall) - -Use this fix when a sandbox can reach `https://inference.local`, but OpenShell reports an upstream timeout against a host-local backend such as Ollama. - -Example symptom: - -```json -{"error":"request to http://host.docker.internal:11434/v1/models timed out"} -``` - -### When This Happens - -This failure commonly appears on Linux hosts that: - -- Run the OpenShell gateway in Docker -- Route `inference.local` to a host-local OpenAI-compatible endpoint such as Ollama -- Have a host firewall or networking configuration that denies container-to-host traffic by default - -In this case, OpenShell routing is usually working correctly. The failing hop is container-to-host traffic on the backend port. - -### Why CoreDNS Is Not the Cause - -This is not the same issue as the Colima CoreDNS fix. - -OpenShell injects `host.docker.internal` and `host.openshell.internal` into sandbox workloads when the selected compute platform supports it. That path bypasses runtime DNS lookup. If the request still times out, the usual cause is host firewall or network policy, not DNS. - -### Verify the Problem - -1. Confirm the model server works on the host: - - ```bash - curl -sS http://127.0.0.1:11434/v1/models - ``` - -2. Confirm the host gateway address also works on the host: - - ```bash - curl -sS http://172.17.0.1:11434/v1/models - ``` - -3. Test the same endpoint from a gateway or sandbox container on the Docker network: - - ```bash - docker ps --filter name=openshell --format '{{.Names}}' - docker exec wget -qO- -T 5 http://host.docker.internal:11434/v1/models - ``` - -If steps 1 and 2 succeed but step 3 times out, the host firewall or network configuration is blocking the container-to-host path. - -### Fix - -Allow the Docker bridge network used by the OpenShell gateway and sandbox containers to reach the host-local inference port. The exact command depends on your firewall tooling (iptables, nftables, firewalld, UFW, etc.), but the rule should allow: - -- **Source**: the Docker bridge subnet used by OpenShell containers (commonly `172.18.0.0/16`) -- **Destination**: the host gateway IP injected into sandbox workloads for `host.docker.internal` (commonly `172.17.0.1`) -- **Port**: the inference server port (e.g. `11434/tcp` for Ollama) - -To find the actual values on your system: +Do not print or copy credential values into diagnostic output. Detaching a +provider revokes its policy and credential access: ```bash -# Docker bridge subnet for the OpenShell network -docker network inspect $(docker network ls --filter name=openshell -q) --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' - -# Host gateway IP visible from inside the container -docker exec cat /etc/hosts | grep host.docker.internal +openshell sandbox provider detach ``` -Adjust the source subnet, destination IP, or port to match your local Docker network layout. - -### Verify the Fix - -1. Re-run the container network check: - - ```bash - docker exec wget -qO- -T 5 http://host.docker.internal:11434/v1/models - ``` - -2. Re-test from a sandbox: +### 4. Verify Native Client Configuration - ```bash - curl -sS https://inference.local/v1/models - ``` +The application must use the real upstream contract: -Both commands should return the upstream model list. +- Native provider base URL, not `https://inference.local`. +- Real model ID, not a placeholder that OpenShell used to rewrite. +- Native OpenAI, Anthropic, Vertex, or other provider request shape. +- Application-owned timeout and retry settings. +- The credential environment variable declared by the attached profile. -### If It Still Fails +Probe the exact endpoint from a newly launched sandbox process. Start with a +non-secret discovery endpoint when the provider offers one, then send a minimal +inference request using the provider's documented API shape. -- Confirm the backend listens on a host-reachable address: `ss -ltnp | rg ':11434\b'` -- Confirm the provider points at the host alias path you expect: `openshell provider get ` -- Confirm the active inference route: `openshell inference get` -- Inspect sandbox logs for upstream timeout details: `openshell logs --since 10m` - -## Common Failure Patterns +### 5. Interpret Common Failures | Symptom | Likely cause | Fix | -|---------|--------------|-----| -| `openshell inference get` shows `Not configured` | No managed inference route configured | `openshell inference set --provider --model ` | -| System inference is `Not configured` | Platform-only route has no backend | `openshell inference set --system --provider --model ` | -| `failed to verify inference endpoint` | Bad base URL, wrong credentials, wrong provider type, or upstream not reachable | Fix provider config, then rerun `openshell inference set`; use `--no-verify` only when the endpoint is intentionally offline | -| Base URL uses `127.0.0.1` | Loopback points at the wrong runtime | Use `host.openshell.internal` or another gateway-reachable host | -| Local engine works only when gateway is local | Gateway moved to remote host | Run the engine on the gateway host, add a tunnel, or use direct external access | -| `connection not allowed by policy` on `inference.local` | Unsupported path or method | Use a supported inference API path | -| `no compatible route` | Provider type does not match request shape | Create or select a provider of the matching type, or change the client API | -| `inference.local` works but a platform function fails | User route is configured but `sandbox-system` is missing or wrong | `openshell inference get --system`; configure or update with `--system`; inspect supervisor logs | -| Direct call to external host is denied | Missing policy or provider attachment | Update `network_policies` and launch sandbox with the right provider | -| Direct call returns `credential_endpoint_mismatch` | Attached provider profile does not authorize the request host, port, or path | Inspect the provider profile endpoints; select or update the profile only if it intentionally authorizes that recipient | -| Direct call returns `request_authority_mismatch` | HTTP authority does not match the CONNECT host and effective port | Include the explicit non-default port in `Host` and use the same authority in absolute-form targets | -| SDK fails on empty auth token | Client requires a non-empty API key even though OpenShell injects the real one | Use any placeholder token such as `test` | -| Upstream timeout from container to host-local backend | Host firewall or network config blocks container-to-host traffic | Allow the Docker bridge subnet to reach the inference port on the host gateway IP (see firewall fix section above) | - -## Full Diagnostic Dump - -Run this when you want a compact report before deciding on a fix: - -```bash -echo "=== Gateway Status ===" -openshell status - -echo "=== Gateway Info ===" -openshell gateway info - -echo "=== Managed Inference ===" -openshell inference get - -echo "=== System Inference Only ===" -openshell inference get --system - -echo "=== Providers ===" -openshell provider list - -echo "=== Selected Provider ===" -openshell provider get - -echo "=== Sandbox Probe ===" -openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' -``` - -When you report back, state: - -1. Which inference path is failing (`inference.local`, direct external, or system inference) -2. Whether gateway topology is part of the problem -3. The most likely root cause -4. The exact fix commands the user should run +|---|---|---| +| `Could not resolve host: inference.local` | Client still uses the removed managed endpoint | Configure the provider's native base URL and attach an endpoint-bearing provider profile | +| Direct request is denied | Missing attachment, endpoint policy, HTTP rule, or binary authorization | Inspect the attached provider profile and sandbox effective policy | +| `credential_endpoint_mismatch` | Credential profile does not authorize the request recipient | Correct the host/port/path or import a narrowly scoped profile for the intended endpoint | +| `request_authority_mismatch` | HTTP authority differs from the CONNECT destination | Use the same host and effective port in both authorities | +| Credential variable is absent | Provider was not attached when this process launched, or profiles collide on a key | Attach the provider and launch a new process; resolve duplicate keys explicitly | +| Upstream rejects the model or body | Client relied on removed model/request rewriting | Configure the real model and provider-native request format in the application | +| `127.0.0.1` works on the host but not in the sandbox | Loopback refers to different runtime | Use `host.openshell.internal` or another gateway-reachable endpoint and profile | +| Host-local request times out | Server bind address, gateway topology, or host firewall blocks container-to-host traffic | Verify the listener and permit only the required gateway network path and port | + +## Host-Local Inference Checklist + +For Ollama, LM Studio, vLLM, SGLang, TRT-LLM, and local NIM deployments: + +1. Verify the engine from the gateway host. +2. Verify it listens on an address reachable from the gateway runtime. +3. Import a custom profile naming `host.openshell.internal` and the actual port. +4. Restrict the profile to the intended binaries and API paths. +5. Create and attach the provider. +6. Configure the application's base URL, model, and timeout. +7. Probe the native endpoint from a newly launched sandbox process. + +## Reporting + +Report: + +1. The active gateway and whether topology contributes to the failure. +2. The provider, profile, attachment, endpoint, and client binary involved. +3. The exact failed host, port, path, and request authority without secrets. +4. Whether the client still relies on removed managed-routing behavior. +5. The narrowest profile, attachment, or application configuration change that + resolves the problem. diff --git a/skills/generate-sandbox-policy/examples.md b/skills/generate-sandbox-policy/examples.md index b4c4b6f822..2cbc21b6b6 100644 --- a/skills/generate-sandbox-policy/examples.md +++ b/skills/generate-sandbox-policy/examples.md @@ -857,8 +857,8 @@ network_policies: The agent notes that `filesystem_policy` and `landlock` are sensible defaults that may need adjustment. Process identity is omitted so the compute driver can -select it. Gateway inference is configured separately via `openshell inference -set/get` rather than an `inference` policy block. +select it. When inference credentials are needed, attach an endpoint-bearing +provider profile; its policy is merged with the sandbox policy. --- diff --git a/skills/openshell-cli/SKILL.md b/skills/openshell-cli/SKILL.md index c46f569789..85a840b592 100644 --- a/skills/openshell-cli/SKILL.md +++ b/skills/openshell-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: openshell-cli -description: Guide agents through using the OpenShell CLI (openshell) for sandbox management, gateway registration, provider configuration and refresh, policy iteration, settings, service exposure, BYOC workflows, and inference routing. Covers basic through advanced multi-step workflows. Trigger keywords - openshell, sandbox create, sandbox exec, sandbox connect, logs, provider create, provider profile, provider refresh, policy set, policy get, settings, service expose, forward, port forward, BYOC, bring your own container, inference, use openshell, run openshell, CLI usage, manage sandbox, manage provider, gateway add, gateway select. +description: Guide agents through using the OpenShell CLI (openshell) for sandbox management, gateway registration, provider configuration and refresh, policy iteration, settings, service exposure, BYOC workflows, and attached-provider inference. Covers basic through advanced multi-step workflows. Trigger keywords - openshell, sandbox create, sandbox exec, sandbox connect, logs, provider create, provider profile, provider refresh, policy set, policy get, settings, service expose, forward, port forward, BYOC, bring your own container, inference, use openshell, run openshell, CLI usage, manage sandbox, manage provider, gateway add, gateway select. --- # OpenShell CLI @@ -9,7 +9,7 @@ Guide agents through using the `openshell` CLI for sandbox and platform manageme ## Overview -The OpenShell CLI (`openshell`) is the primary interface for managing sandboxes, providers, policies, settings, exposed services, inference routes, and gateway registrations. Gateway service lifecycle is handled outside the CLI by packages, systemd, or Helm. This skill teaches agents how to orchestrate CLI commands for common and complex workflows. +The OpenShell CLI (`openshell`) is the primary interface for managing sandboxes, providers, policies, settings, exposed services, and gateway registrations. Gateway service lifecycle is handled outside the CLI by packages, systemd, or Helm. This skill teaches agents how to orchestrate CLI commands for common and complex workflows. **Companion skill**: For creating or modifying sandbox policy YAML content (network rules, L7 inspection, access presets), use the `generate-sandbox-policy` skill. This skill covers the CLI *commands* for the policy lifecycle; `generate-sandbox-policy` covers policy *content authoring*. @@ -700,29 +700,23 @@ The user does not need to disconnect. Policy updates are hot-reloaded; `--wait` openshell sandbox delete work-session ``` -## Workflow 7: Managed Inference +## Workflow 7: Inference with Attached Providers -Configure the user-facing `inference.local` route or the system inference route used by platform functions. - -Ensure the provider exists, then set the route: - -```bash -openshell provider list -openshell inference set \ - --provider nvidia \ - --model nvidia/nemotron-3-nano-30b-a3b -``` - -This updates the managed `inference.local` route. Endpoint verification runs before the route is saved. Use `--no-verify` only when verification is intentionally impossible, and use `--timeout SECONDS` to configure the request timeout. Add `--system` to `set` or `update` for the platform-only system route. - -Inspect both configurations: +Inference uses the same provider attachment workflow as other credentialed +services. Import or select a profile that authorizes the provider's native +endpoint, create the provider, and attach it only to sandboxes that need it: ```bash -openshell inference get -openshell inference get --system +openshell provider profile import -f ./inference-provider.yaml +openshell provider create --name model-provider --type --credential +openshell sandbox provider attach work-session model-provider +openshell sandbox exec work-session -- ``` -Agents send HTTPS requests to `inference.local`; the sandbox intercepts them and routes them through the configured inference route. Sandbox policy remains separate from inference route configuration. +The application owns the native base URL, model, request shape, and timeout. +Launch a new process after attaching a provider so it inherits the provider +credential placeholder. Use the `debug-inference` skill for endpoint, policy, +credential-binding, or migration failures. ## Workflow 8: Gateway Management @@ -835,4 +829,4 @@ $ openshell sandbox upload --help |-------|------------| | `generate-sandbox-policy` | Creating or modifying policy YAML content (network rules, L7 inspection, access presets, endpoint configuration, and network middleware) | | `debug-openshell-cluster` | Diagnosing gateway deployment, runtime, or health failures | -| `debug-inference` | Diagnosing `inference.local`, host-backed local inference, and provider base URL issues | +| `debug-inference` | Diagnosing attached-provider inference, native endpoints, host-backed models, and migration from `inference.local` | diff --git a/snapcraft.yaml b/snapcraft.yaml index f567c63c47..e2d47f7dd3 100644 --- a/snapcraft.yaml +++ b/snapcraft.yaml @@ -9,7 +9,7 @@ description: | OpenShell provides safe, sandboxed runtimes for autonomous AI agents. It offers a CLI for managing gateways, sandboxes, and providers with policy-enforced egress routing, credential proxying, and privacy-aware - LLM inference routing. + profile-backed model-provider access. The OpenShell snap ships a CLI (`openshell`), a terminal UI (`openshell.term`), and a managed gateway daemon (`openshell.gateway`). diff --git a/tasks/scripts/generate_python_proto.py b/tasks/scripts/generate_python_proto.py index b29510a085..76664eb415 100644 --- a/tasks/scripts/generate_python_proto.py +++ b/tasks/scripts/generate_python_proto.py @@ -9,7 +9,6 @@ from pathlib import Path PROTO_FILES = [ - "proto/inference.proto", "proto/openshell.proto", "proto/datamodel.proto", "proto/options.proto", @@ -17,22 +16,6 @@ ] LINE_REWRITES = { - "python/openshell/_proto/inference_pb2.py": [ - ( - r"^import datamodel_pb2 as datamodel__pb2$", - "from . import datamodel_pb2 as datamodel__pb2", - ), - ( - r"^import options_pb2 as options__pb2$", - "from . import options_pb2 as options__pb2", - ), - ], - "python/openshell/_proto/inference_pb2_grpc.py": [ - ( - r"^import inference_pb2 as inference__pb2$", - "from . import inference_pb2 as inference__pb2", - ), - ], "python/openshell/_proto/openshell_pb2_grpc.py": [ ( r"^import openshell_pb2 as openshell__pb2$", diff --git a/tasks/scripts/verify-python-wheel.py b/tasks/scripts/verify-python-wheel.py index 6234c0cdb6..0539bd88a0 100644 --- a/tasks/scripts/verify-python-wheel.py +++ b/tasks/scripts/verify-python-wheel.py @@ -13,7 +13,7 @@ from pathlib import Path from zipfile import BadZipFile, ZipFile -PROTO_STEMS = ("datamodel", "inference", "openshell", "options", "sandbox") +PROTO_STEMS = ("datamodel", "openshell", "options", "sandbox") BYTECODE_SUFFIXES = (".pyc", ".pyo") NATIVE_SUFFIXES = (".dll", ".dylib", ".exe", ".pyd", ".so")