The litellm example plugin rejects any Step.CallModel, and its README rules out classifier-backed routers on that basis. That keeps llm_task_classifier out of the LiteLLM integration entirely.
We built a routing plugin that serves those intermediate calls instead of rejecting them: it drives run_stream(), answers each judge call in-plugin, narrows candidate_models from the RoutingOutcome, and reports judge errors with ModelCall.fail() so the classifier's own fallback kicks in. It works today against main (f4152d07) and litellm 1.100.0. In our deep-research workload, routine steps go to the efficient deployment (judge p_solve 0.90+) and a hard synthesis step escalates (p_solve 0.05–0.18). Cost is real, though: the judge adds 6–21s and ~1.4k prompt tokens per routed call.
I would be curious for the team's thoughts on a potential PR for this (we could upstream the changes we did for our plugin internally) or whether this is an intentional limitation. One design question: our judge uses its own OpenAI client, which bypasses LiteLLM's spend tracking and needs separate credentials. Routing the judge through LiteLLM against a designated non-routed deployment seems cleaner (recursion-safe, spend-tracked, one credential model), but an injectable judge client is also an option. Which shape would you want?
Happy to contribute the PR with tests if there's interest in this.
The litellm example plugin rejects any
Step.CallModel, and its README rules out classifier-backed routers on that basis. That keepsllm_task_classifierout of the LiteLLM integration entirely.We built a routing plugin that serves those intermediate calls instead of rejecting them: it drives
run_stream(), answers each judge call in-plugin, narrowscandidate_modelsfrom theRoutingOutcome, and reports judge errors withModelCall.fail()so the classifier's own fallback kicks in. It works today against main (f4152d07) and litellm 1.100.0. In our deep-research workload, routine steps go to the efficient deployment (judge p_solve 0.90+) and a hard synthesis step escalates (p_solve 0.05–0.18). Cost is real, though: the judge adds 6–21s and ~1.4k prompt tokens per routed call.I would be curious for the team's thoughts on a potential PR for this (we could upstream the changes we did for our plugin internally) or whether this is an intentional limitation. One design question: our judge uses its own OpenAI client, which bypasses LiteLLM's spend tracking and needs separate credentials. Routing the judge through LiteLLM against a designated non-routed deployment seems cleaner (recursion-safe, spend-tracked, one credential model), but an injectable judge client is also an option. Which shape would you want?
Happy to contribute the PR with tests if there's interest in this.