kitten_tts2.md

Kitten TTS 2

Powered by Stellon Labs.

kitten_tts2 supports offline multilingual TTS, preset voices, reference-audio cloning, and chunked long-form speech. The complete inference path is native C++/GGML: Qwen3, S3 meanflow, HiFT, S3 tokenizer, CAMPPlus, and XVectorSincNet. Python, LibTorch, TorchScript, and ONNX Runtime are not runtime dependencies.

The family is experimental; a compatible community GGUF is available below. CPU and NVIDIA CUDA are validated. Other GPU backends, streaming, automatic reference transcription, the upstream text normalizer, and the smaller student_w4/student_w8 decoders are not implemented or validated by this port. The full default decoder is used.

Build

Shared S3 code is included through the chatterbox dependency. No Kitten-specific build flag or Torch package is needed.

This port uses the fixes from Chatterbox S3 encoder correctness PR #778 (relative-shift padding and attention/feed-forward LayerNorm epsilon), which merged before the Kitten integration. Current upstream includes both changes. The original Kitten integration also added build registration, documentation indexes and the WebUI catalog with its generated bundle. The updated weight profiles below update the Kitten implementation, exporter, package metadata, tests, documentation and Kitten WebUI entries. CPU sessions also reuse exact-shape S3 encoder graphs; other S3Gen users retain their existing graph-release policy.

cmake -S . -B build-kitten2 -DAUDIOCPP_MODEL_SET=custom \
  -DAUDIOCPP_MODELS=kitten_tts2 -DENGINE_ENABLE_CUDA=OFF \
  -DENGINE_BUILD_MODEL_TESTS=ON
cmake --build build-kitten2 --config Release \
  --target audiocpp_cli audiocpp_server audiocpp_gguf audiocpp_kitten_tts2_prepare_voices --parallel 8

Visual Studio binaries are in build-kitten2/bin/Release/; single-configuration builds use build-kitten2/bin/. Run examples from the audio.cpp repository root and replace model/source path placeholders with their actual locations. Commands use POSIX-shell \ line continuations; in PowerShell, join the lines or use backticks instead. Examples using audiocpp_cli assume its binary directory is on PATH.

For NVIDIA, configure with -DENGINE_ENABLE_CUDA=ON and select --backend cuda at runtime. The Kitten networks use the selected GGML backend; reference resampling, sampling, and some small host computations remain on CPU. Windows Release builds and preset/cloned speech were tested on a 16 GB RTX 4060 Ti with CUDA Toolkit 13.3 and driver 610.88. Set -DCMAKE_CUDA_ARCHITECTURES=89 when building specifically for this card; choose the architecture for other cards. CUDA 13.3 places its runtime DLLs in CUDA_PATH/bin/x64; ensure that directory is on PATH when launching the executable on Windows.

The speaker's temporal max pooling uses a CUDA-supported 3x1 window. Inference uses the existing backend precision and dispatch policy. For controlled CUDA component comparisons, set NVIDIA_TF32_OVERRIDE=0 before starting the native probes, as described in the test setup. This is a parity-test setting; ordinary synthesis is also tested without it. See the validation record for the current port's build, server/cloning checks, measured RTF and component errors.

The WebUI shows 7 GB estimated VRAM for the native Q8 package. An earlier CUDA server run, before removing the shared precision edits, used a 238-character passage and peaked approximately 4.7 GiB above the desktop baseline for preset speech and 5.9 GiB for cloning. This measurement has not been repeated after the cleanup. The estimate includes headroom; it is guidance, not a tested minimum for every input or weight type.

Download and preset speech

The refreshed packages contain the full native decoder, speaker encoders, and 124 prepared voice entries. The checkpoint date in each filename distinguishes them from the older packages. Q8 is the default and recommended package; Q4 preserves the BF16 embedding and tied output head. Previous BF16 and hybrid Q4 packages remain available; see weight profiles.

python tools/model_manager_v2.py install kitten_tts2 --models-root models

The family ID resolves to kitten_tts2_20261021_q8_0, which installs the GGUF under models/KittenTTS2-GGUF/. No Hugging Face token is needed.

Direct download: kitten-tts2-20261021-q8_0.gguf. Separate model components or the original preset recordings are not needed at runtime. Custom voice cloning still requires the reference clip and transcript you supply.

audiocpp_cli --family kitten_tts2 \
  --model models/KittenTTS2-GGUF/kitten-tts2-20261021-q8_0.gguf \
  --backend cpu --threads 8 --task tts --voice-id Bruno \
  --request-option seed=1234 --text "Hello there. This is native Kitten speech." \
  --out hello.wav

Upstream source assets

These assets are for direct safetensors loading, local conversion, or developer validation. They are not additional downloads required by the published GGUF.

Use the upstream KittenML/kitten-tts-2 layout, including the full lm/model.safetensors, LM configuration/tokenizer, config.json, cpp/default/voices.json, and speaker/model.safetensors. Preserve LICENSE.md and speaker/LICENSE. For multilingual preset preparation, also keep voices/voices.json and its referenced WAV/NPZ files. The upstream cpp/default/decoder.pt is needed only for the optional PyTorch component comparisons described in the test guide; the native runtime and GGUF converter do not use it.

Add the official S3 meanflow checkpoint as native/s3gen_meanflow.safetensors, and the Chatterbox MIT license as native/LICENSE. Validated decoder revision: 749d1c1a46eb10492095d68fbcf55691ccf137cd; SHA-256: d65cb687a2ed581ee6cc297e919ffefa63386944f42364ae13b78a594945514f. The runtime never downloads assets.

The upstream custom cpp/model-tq2_1.gguf and packed ternary/emb4 safetensors are not inputs to this loader. Use the full safetensors or convert to audio.cpp GGUF.

audiocpp_cli --family kitten_tts2 --model /path/to/kitten-tts-2 \
  --backend cpu --threads 8 --task tts --voice-id Bruno \
  --request-option seed=1234 --text "Hello there. This is native Kitten speech." \
  --out hello.wav

The published v2 package contains 47 upstream voices and PreparedBruno. Converting the current upstream checkpoint includes all 123 upstream voices and the PreparedBruno alias. Select a voice present in the loaded package. Output is mono 24 kHz audio, written with audio.cpp's existing WAV support.

Languages

Use the language-named preset for Arabic, Chinese, French, German, Hindi, Italian, Portuguese, Russian, or Spanish. The 38 named English voices remain available in the published v2 package. Current upstream conversions additionally include Dutch, Indonesian, Japanese, Korean, Polish, and Turkish presets, such as DutchBaker, IndonesianTourGuide, JapaneseTrainConductor, KoreanNewsAnchor, PolishNewsAnchor, and TurkishNewsAnchor.

audiocpp_cli --family kitten_tts2 --model models/kitten-tts2-v2/kitten-tts2-q8-v2-multilingual.gguf \
  --backend cpu --task tts --voice-id German \
  --text "Guten Morgen. Dies ist ein Test der deutschen Sprachausgabe." --out german.wav

The server accepts "voice": "German" (or another language preset) with UTF-8 text in input. A reference clip and transcript in the target language can also be supplied for cloning. There is no separate language switch: the voice reference supplies the accent and pronunciation context.

Text reaches the original Qwen tokenizer without English normalization, matching upstream's recommendation to use normalize=False for other languages. Write numbers, dates and abbreviations as spoken words in the target language. The shared chunker preserves Unicode codepoints and supports CJK punctuation.

Upstream's README advertises 20 languages, but its published voice index provides presets for English and the nine languages above. The remaining ten languages have no shipped language presets in this snapshot and are not validated here. Generation and tokenization checks do not establish pronunciation quality; see the validation record for exactly what was exercised.

Voice cloning

Supply a one-to-thirty-second reference clip and its transcript:

audiocpp_cli --family kitten_tts2 --model models/kitten-tts2-v2/kitten-tts2-q8-v2-multilingual.gguf \
  --backend cpu --threads 8 --task clon --voice-ref reference.wav \
  --reference-text "The exact words spoken in the reference clip." \
  --text "The new words to say in this voice." --request-option seed=1234 \
  --out cloned.wav

Cloning also works in TTS sessions. The C++ request accepts voice.speaker.audio or audio_input, with reference_text in request options. Reference audio takes precedence over a preset ID. The server speech endpoint accepts voice_ref (path or base64) and reference_text.

Channels are averaged. Native sinc resampling supplies 16 kHz speaker/tokenizer inputs and the 24 kHz decoder reference. The identity encoder uses the complete clip. Decoder conditioning repeats short clips to at least six seconds and takes at most ten seconds, following the upstream default decoder policy. Upstream uses librosa/soxr for some S3 preprocessing, so reference tokens need not be byte-identical with this native sinc resampler.

Reference encoders load on the first clone request. A session caches its last clip and transcript; changing either invalidates conditioning and the LM prefix. This in-memory cache does not register persistent voice IDs.

Sampling and long-form text

Stable sampling defaults to temperature 0.8, top-p 0.8, top-k 50. preset=expressive uses temperature/top-p 0.9. Explicit request options override the preset; temperature zero selects greedy decoding.

The sampler uses the upstream 50-token repetition window, silence/end exemptions, and token-run penalty. The tied head projects only speech codes and the two end markers, deliberately excluding stray text tokens. Native RNG and arithmetic differ from PyTorch CPU; equal seeds do not promise Python-identical waveforms.

Text defaults to 380-codepoint chunks in tag_aware mode. Each chunk reuses the reference prefix and starts fresh target generation. Joins trim edge silence, apply short fades, and insert 160 ms gaps. Expression tags reach the model verbatim and enable its emotion prefix. Expand numbers/abbreviations when needed.

LM tensors retain their stored types by default. An explicit --session-option kitten_tts2.weight_type=... changes projection storage only; the embedding and tied output head retain their stored type. The decoder and reference encoders use F32.

CPU projection storage

The refreshed kitten_tts2_20261021_q4_0 package stores Q4 projections directly, with the BF16 embedding and output head preserved. Leave weight_type at native to avoid runtime repacking.

For the published Q8-v2 GGUF or a current Q8 conversion, try --session-option kitten_tts2.weight_type=q4_0 on CPU. This packs the ternary projection values exactly into Q4 blocks while retaining the stored BF16 embedding and output head. It is not the hybrid Q4 download, whose embedding is also quantized. Repacking happens once when the session loads; it adds startup work but reduces the cost of subsequent generation. The default remains native, and the GGUF file is not modified.

audiocpp_cli --family kitten_tts2 \
  --model models/kitten-tts2-v2/kitten-tts2-q8-v2-multilingual.gguf \
  --backend cpu --threads 8 --task tts \
  --session-option kitten_tts2.weight_type=q4_0 \
  --request-option voice_id=Bruno --request-option seed=42 \
  --text "Please close the window before you leave. The books beside the desk must stay dry." \
  --out hello.wav --log

If output is cut short by the automatic token budget, set a larger ceiling, for example --request-option max_tokens=1000.

Execution plans

The default GGML path is unchanged. kitten_tts2.exec_plan is a session option; existing GGUFs also accept it without reconversion. Components and parameters omitted from the plan retain their defaults.

NEON vocoder

Build with -DENGINE_ENABLE_NEON=ON, then select the HiFT neural decoder:

audiocpp_cli --task tts --family kitten_tts2 --model <model.gguf> \
  --backend cpu --threads 8 --text "Hello from Kitten TTS 2." \
  --session-option 'kitten_tts2.exec_plan={"vocoder":{"impl":"neon","backend":"cpu"}}' \
  --out output.wav --metrics --log

The same plan can be used with --backend metal: the rest of the model retains Metal execution, while the HiFT convolution stack runs on the NEON CPU library. Pitch prediction uses the existing framework API. Kitten keeps excitation model-local and uses the existing STFT/ISTFT APIs for spectral processing. The NEON library has no GGML dependency. Sampling settings, token budgets, inference steps and language-model/flow execution remain unchanged. Executor selection and weight adaptation stay in Kitten. The NEON path receives plain mel/F0 arrays. The GGML HiFT module is unchanged.

Updated weight profiles

Use the kitten_tts2 model family and leave kitten_tts2.weight_type unset (or select native) to retain the GGUF's stored precision. The loader reads tensor types directly; no custom export selector is embedded in the package. The WebUI sends no Kitten precision override by default.

Export --type	Download filename	Source LM	Body / tied embedding	Approximate complete size
q8_0	kitten-tts2-20261021-q8_0.gguf	Refreshed ternary checkpoint	Q8_0 / BF16	3.34 GB
q4_0 --embedding-type orig	kitten-tts2-20261021-q4_0.gguf	Refreshed BF16 checkpoint	Q4_0 / BF16	2.64 GB
bf16	kitten-tts2-bf16-multilingual.gguf	model.safetensors	BF16 / BF16	4.60 GB
q8_0	kitten-tts2-q8-v2-multilingual.gguf	model-ternary.safetensors	Q8_0 / BF16	3.28 GB
q4_0	kitten-tts2-q4-multilingual.gguf	model.safetensors	Q4_0 / Q8_0	2.27 GB

install kitten_tts2 selects Q8. Install a selected package with:

python tools/model_manager_v2.py install kitten_tts2_bf16 --models-root models
python tools/model_manager_v2.py install kitten_tts2_20261021_q8_0 --models-root models
python tools/model_manager_v2.py install kitten_tts2_20261021_q4_0 --models-root models
python tools/model_manager_v2.py install kitten_tts2_q8_v2 --models-root models
python tools/model_manager_v2.py install kitten_tts2_q4 --models-root models

The three previous-checkpoint packages install into models/kitten-tts2-v2/, with distinct GGUF filenames. They can coexist and share the license/NOTICE files. Select the chosen GGUF path in the CLI/server. Each package also downloads LICENSE, NOTICE, native/LICENSE, speaker/LICENSE, and SHA256SUMS from the same repository.

Every profile uses the existing default S3 MeanFlow decoder, reference encoders and multilingual voice conditioning. No shared framework, S3 or ggml changes are required. The student decoders and upstream TL2/emb4 format are not part of these profiles.

The Q8 and Q4 body matrices preserve the updated BF16 checkpoint's ternary values exactly in standard GGML blocks. Q4 is a mixed-precision export: its embedding is quantized directly from full BF16 to Q8_0, with approximately 0.44% relative L2 weight error overall and 0.41% across speech-token rows. These are reconstruction errors, not audio quality scores. Norms and speaker projections retain BF16; S3 and reference components retain their original types. Hybrid Q4 remains experimental: local ASR checks found missing words in longer CPU samples. See the validation record for speech-quality checks and limitations.

On CUDA, Q4 feed-forward down projections are loaded as BF16 to avoid non-finite results with this model's large activations. Their values stay exact; this adds about 483 MiB of resident weights compared with retaining those projections as Q4. CPU keeps the Q4 body. File size is not a RAM/VRAM estimate.

Updated conversion requires NumPy, PyTorch and safetensors offline only. The full lm/model.safetensors is needed for every updated profile to refresh the preset speaker projections from the shipped NPZ identities. The existing decoder conditioning and PreparedBruno alias are retained.

The exporter defaults to --type q8_0. Each type selects its source checkpoint and tensor storage recipe, including the embedding precision shown above. For exact Q4 projections with the original BF16 embedding and tied output head, use --type q4_0 --embedding-type orig. This package loads directly with the default native weight type, without runtime Q4 conversion.

For CPU-only ternary storage, --type tq2_1 packs the 196 projections into 2.125 bits per weight, preserving independent scales for each 128-weight group. The embedding/head stays BF16 and the codec stays F32. Load with native weight storage; no session option is needed. This opt-in format requires an audio.cpp binary with TQ2_1 support and is not a CUDA/Vulkan/Metal package. Existing package defaults are unchanged. The CPU kernel is adapted from Kitten's C++ implementation. audio.cpp uses GGUF type ID 44: the other fork's ID 43 is already used by audio.cpp's I2_S format, so its ternary GGUFs are not interchangeable.

python tools/community_models/prepare_kitten_tts2_gguf.py \
  --model /path/to/updated-kitten-tts-2 --converter build-kitten2/bin/audiocpp_gguf \
  --type bf16 --output kitten-tts2-bf16-multilingual.gguf
python tools/community_models/prepare_kitten_tts2_gguf.py \
  --model /path/to/updated-kitten-tts-2 --converter build-kitten2/bin/audiocpp_gguf \
  --type q8_0 --output kitten-tts2-q8-v2-multilingual.gguf
python tools/community_models/prepare_kitten_tts2_gguf.py \
  --model /path/to/updated-kitten-tts-2 --converter build-kitten2/bin/audiocpp_gguf \
  --type q4_0 --output kitten-tts2-q4-multilingual.gguf

Use --s3gen and --s3-license when the original default decoder assets are stored separately.

Self-contained GGUF

The conversion commands above produce complete runtime packages. They require the native converter and audiocpp_kitten_tts2_prepare_voices beside it.

The upstream prepared cpp/default/voices.json omits the nine language presets; the helper fills those from voices/voices.json, the NPZ embeddings/tokens, and reference WAVs. It refreshes every preset speaker projection with the updated full LM while preserving decoder conditioning. These preparation dependencies are not needed for inference.

Preparation uses CPU by default; --voice-backend cuda selects a CUDA-enabled preparer. --voice-preparer can select an executable elsewhere. Save the complete index with --voices-output voices.json, then reuse it with --prepared-voices voices.json on later conversions. Incomplete indexes are rejected so a multilingual spec cannot silently ship English-only voices. For direct safetensors loading, install that complete index as MODEL/cpp/default/voices.json; the original upstream index alone lacks the language presets. The GGUF embeds it and needs no adjacent reference clips.

--s3gen and --s3-license can select assets outside the model directory. All three tensor sources, preset conditioning, configuration, tokenizer, spec, licenses, and NOTICE are packaged. No .pt file is embedded. The package supports cloning without adjacent model files. Use the GGUF path as --model.

Weights retain the Stellon Labs Community License, including commercial registration/limits and attribution requirements. Decoder and speaker components retain their respective upstream terms. No model weights are committed with the integration.

See validation instructions and measured results.