Skip to content

feat(inference)!: replace managed routes with providers - #3194

Open
varshaprasad96 wants to merge 1 commit into
NVIDIA:mainfrom
varshaprasad96:3172-remove-managed-inference-and-refactor-cli/varshaprasad96
Open

feat(inference)!: replace managed routes with providers#3194
varshaprasad96 wants to merge 1 commit into
NVIDIA:mainfrom
varshaprasad96:3172-remove-managed-inference-and-refactor-cli/varshaprasad96

Conversation

@varshaprasad96

Copy link
Copy Markdown
Contributor

Summary

Remove the managed inference-route stack now that provider v2 supplies native provider endpoints, and fold the next service/settings CLI module extraction into the same clean replacement branch. This keeps the #2304 refactor aligned with the architecture established by #3172.

Related Issue

Closes #3172
Relates to #2304
Supersedes #3188

Changes

  • Remove the managed inference CLI, gRPC API, router crate, supervisor interception path, SDK clients, generated bindings, and obsolete examples/tests.
  • Add SQLite and PostgreSQL migrations that remove legacy inference-route objects while preserving all other stored objects.
  • Keep inference access provider-native, including provider type normalization and custom endpoint-bearing profiles, and document the upgrade path.
  • Extract service and settings command implementations from the CLI run module into dedicated command modules.
  • Update architecture, published docs, man pages, agent skills, Go/Python SDK surfaces, and repository metadata.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (obsolete managed-route cases removed; provider endpoint coverage retained)
  • mise run ci passes on the rebased branch
  • mise run go:ci passes
  • Focused Python SDK and wheel tests pass (137 tests)
  • mise run e2e could not complete locally because the Docker daemon is unavailable

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: remove managed inference routes in favor of providers v2

1 participant