refactor(inference): remove managed inference routes - #3195
Open
johntmyers wants to merge 1 commit into
Open
Conversation
johntmyers
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 4, 2026 20:48
13 tasks
|
🌿 Preview your docs: https://nvidia-preview-pr-3195.docs.buildwithfern.com/openshell |
Collaborator
Author
Docker E2E Test AttestationLocal Docker E2E tests passed. CI does not currently run this lane, so this comment serves as the verification record.
Test SummaryThe Tests Executed
|
Collaborator
Author
Python E2E Test AttestationLocal Python E2E tests passed against a Docker-backed gateway. CI does not currently run this lane, so this comment serves as the verification record.
Test SummaryThe 81 tests under Tests Executed |
Closes #3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
johntmyers
force-pushed
the
refactor/3172-remove-managed-inference/johntmyers
branch
from
September 4, 2026 21:12
6599bfe to
36b3abd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remove OpenShell's managed inference routing control plane and
inference.localdata path. Inference workloads now use explicitly imported, use-case-specific provider profiles attached to sandboxes and call provider-native endpoints directly;inference_capableremains informational for future observability hooks.Related Issue
Closes #3172
Changes
proto/,crates/openshell-core/,crates/openshell-server/, and SDKs: remove inference-route APIs, commands, generated clients, auth scopes, and persisted route behavior.crates/openshell-supervisor-network/andcrates/openshell-sandbox/: removeinference.local, request-shape matching, route refresh/caching, and inference-route supervisor configuration.crates/openshell-router/: remove the built-in privacy router crate and its packaging/dependency surface.crates/openshell-server/migrations/: delete legacyinference_routeobjects during SQLite and PostgreSQL upgrades.crates/openshell-providers/: retain provider aliases and Vertex constants, keepinference_capableinformational, and preserve endpoint credential-binding guards.e2e/: replace managed-route tests with explicitly imported OpenAI-style and Anthropic-style profile fixtures against native endpoints, including a negativeinference.localassertion.docs/,architecture/,examples/local-inference/, and agent skills: document the breaking migration and provide human-readable Now/After workflows.Deviations from Plan
None — implemented as planned in the approved issue comment and follow-up decisions.
Testing
mise run pre-commitpassesmise run testpassesmise run cipassesmise run e2e:dockerpassesmise run e2e:pythonpasses (86 passed, 81 OIDC-only skipped)Tests added:
x-api-keyprofiles calling native endpoints, secret substitution/model body assertions, and negativeinference.localbehavior.Checklist
Documentation updated:
docs/sandboxes/inference-routing.mdx: provider-backed inference workflow, breaking migration, and Now/After UX.docs/get-started/tutorials/andexamples/local-inference/: native endpoint examples using explicitly imported profiles.architecture/, provider/security/observability/reference docs, and agent skills: remove managed-route assumptions and align troubleshooting/workflows.