Feature request
Add an ensemble routing method that combines independent responses from multiple models into one synthesized answer.
Instead of selecting a single model for each request, Switchyard should send the request to multiple candidate models concurrently, then pass their drafts to a configured synthesizer model to produce the final response.
Motivation
Explore whether combining two smaller or less capable models can provide useful answer quality compared with a stronger single-model baseline, particularly for coding tasks. Quality, latency, and total upstream usage should be evaluated rather than assuming that an ensemble is better or cheaper.
Proposed behavior
- Configure candidate model targets and a synthesizer target per route.
- Generate candidate drafts concurrently.
- Give candidates independent output budgets.
- Require a configurable minimum number of successful candidates before synthesis.
- Synthesize usable drafts into a single response while preserving the caller-facing API and final-response streaming settings.
- Handle candidate failures explicitly and keep internal reasoning blocks out of synthesis inputs.
Implementation PR
PR #657: feat(routing): add ensemble response synthesis implements this method with two to four candidate targets, configurable synthesis, validation, and tests.
The PR also includes exploratory comparisons of SOL + Opus → SOL against Astra. These small-sample results are preliminary and do not establish general quality equivalence, superiority, or a cost advantage. Candidate usage must be included when comparing total cost.
Acceptance criteria
- An ensemble route invokes its candidates concurrently and synthesizes their usable drafts.
- Candidate budgets, synthesizer selection, and the minimum-success threshold are configurable and validated.
- Failure handling and final-response streaming behavior are covered by tests.
- Documentation explains configuration, additional upstream calls, and usage-accounting limitations.
This request is specifically for the ensemble routing method; frontend configuration is separate.
Feature request
Add an ensemble routing method that combines independent responses from multiple models into one synthesized answer.
Instead of selecting a single model for each request, Switchyard should send the request to multiple candidate models concurrently, then pass their drafts to a configured synthesizer model to produce the final response.
Motivation
Explore whether combining two smaller or less capable models can provide useful answer quality compared with a stronger single-model baseline, particularly for coding tasks. Quality, latency, and total upstream usage should be evaluated rather than assuming that an ensemble is better or cheaper.
Proposed behavior
Implementation PR
PR #657: feat(routing): add ensemble response synthesis implements this method with two to four candidate targets, configurable synthesis, validation, and tests.
The PR also includes exploratory comparisons of SOL + Opus → SOL against Astra. These small-sample results are preliminary and do not establish general quality equivalence, superiority, or a cost advantage. Candidate usage must be included when comparing total cost.
Acceptance criteria
This request is specifically for the ensemble routing method; frontend configuration is separate.