A High-Performance Serving System for Speech Language Models
VoxServe delivers low-latency, high-throughput inference for Speech Language Models (SpeechLMs), including text-to-speech (TTS) and speech-to-speech (STS) models.
- [2025-02] Blog post: Light-Speed Qwen3-TTS Serving at Scale with VoxServe
- [2025-02] Paper released: VoxServe: A Streaming-Centric Serving System for Speech Language Models
Install via pip and start the server:
pip install vox-serve
vox-serve --model <model-name> --port <port-number>Or install from source:
git clone https://github.com/vox-serve/vox-serve.git
cd vox-serve
pip install -e .
python -m vox_serve.launch --model <model-name> --port <port-number>Send requests to the server:
# Text-to-speech
curl -X POST "http://localhost:<port-number>/generate" \
-F "text=Hello world" -F "streaming=true" -o output.wav
# Speech-to-speech (for models with audio input support)
curl -X POST "http://localhost:<port-number>/generate" \
-F "text=Hello world" -F "@input.wav" -F "streaming=true" -o output.wavSee the examples/ directory for more usage examples.
VoxServe supports the following TTS and STS models:
| Model | Type | Link |
|---|---|---|
chatterbox |
TTS | Chatterbox TTS |
cosyvoice2 |
TTS | CosyVoice2-0.5B |
csm |
TTS | CSM-1B |
orpheus |
TTS | Orpheus-3B |
qwen3-tts |
TTS | Qwen3-TTS-1.7B |
zonos |
TTS | Zonos-v0.1 |
glm |
STS | GLM-4-Voice-9B |
step |
STS | Step-Audio-2-Mini |
See the models documentation for detailed information. More models coming soon.
VoxServe is optimized for real-time speech synthesis. The demo below shows a TTS request achieving 40 ms Time-To-First-Audio (TTFA) on an NVIDIA H100 GPU with Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice.
Qwen3-TTS supports incremental text input, enabling seamless integration with LLMs for voice chatbots. The demo below shows VoxServe connected to a local LLM with low end-to-end latency.
VoxServe includes a web-based playground for interactive testing. Use the browser UI to manage servers, generate audio, and view real-time logs.
See examples/playground/README.md for setup instructions.