Skip to content

whistle_asr.md

Latest commit

 

History

History
192 lines (162 loc) · 10.3 KB

whistle_asr.md

File metadata and controls

192 lines (162 loc) · 10.3 KB
Raw

Whistle ASR (experimental)

The whistle_asr family runs Cactus Compute Whistle locally through audio.cpp. The current implementation supports offline transcription of 16 kHz mono WAV files up to 30 seconds. The encoder uses the selected backend and the decoder runs on the host CPU. It detects one of the model's seven languages (English, German, French, Spanish, Italian, Dutch, or Polish); --language en, de, fr, es, it, nl, or pl forces a language. These are supported language codes, not seven-language validation evidence. The focused validation report records the English and German checks separately. French, Spanish, Italian, Dutch, and Polish each have one ElevenLabs synthetic clip checked through automatic and forced-language CLI requests. See the five-language spot checks. This limited coverage does not qualify general accuracy in all seven languages.

GGUF package

The default package is whistle_asr_f32, containing Whistle-GGUF/whistle-f32.gguf in audio-cpp/audio.cpp-gguf (about 211 MiB).

audiocpp_cli --task asr --family whistle_asr --backend cuda --threads 4 \
  --model models/Whistle-GGUF/whistle-f32.gguf \
  --audio assets/resources/sample_16k.wav --log --metrics

Use --backend vulkan, cpu, or metal for those backends. The decoder runs on the host CPU in every case.

Local conversion

To convert the official model locally, use the exact revision b358ddadd89b7a713b5aa131f23032d3cca1b251 and put checkpoints/whistle.safetensors in a dedicated directory as whistle.safetensors, together with config.json and LICENSE. Keep whistle.cact outside that directory. The converter checks their byte lengths and SHA-256 hashes, extracts the tokenizer, mel filterbank, and Hadamard permutations from the official .cact, and embeds those six sidecars plus the Whistle model spec in the GGUF:

cmake --build build/release --target audiocpp_gguf audiocpp_cli
python3 tools/community_models/whistle_asr/convert.py \
  --source /path/to/whistle-source \
  --cact /path/to/whistle.cact \
  --converter build/release/bin/audiocpp_gguf \
  --output /path/to/whistle-f32.gguf
audiocpp_cli --task asr --family whistle_asr --backend cpu \
  --model /path/to/whistle-f32.gguf \
  --audio assets/resources/sample_16k.wav --log --metrics

The official .cact uses Cactus Quants. The GGUF stores the published FP32 Safetensors checkpoint instead. The two weight sets can yield different spellings or number formatting for the same speech. The runtime accepts FP32 tensors only, including for an extracted Safetensors directory with the same five required assets. Raw upstream Safetensors alone are insufficient. Other GGUF precisions are rejected. Conversion and inference do not require the Cactus engine, ONNX Runtime, or Python at recognition time.

Streaming, keyword biasing, word timestamps, speech embeddings, resampling, and longer recordings are not supported by this family. CPU, Metal, CUDA, and Vulkan are supported encoder backends. HIP has not been validated. The runtime rejects unsupported sample rates, channel counts, and durations rather than silently changing the input. Decoding uses the checkpoint's 320-token ceiling; a non-terminating decode fails explicitly instead of reporting a truncated transcript as complete. No model is downloaded automatically.

Constant-RMS speech is passed to the model rather than rejected by an amplitude-based noise gate. Digital silence returns an empty transcript. Inputs shorter than 40 milliseconds also return an empty transcript. This family has no speech activity detector, so noise-only recordings may produce words.

Runtime structure

The mel frontend reuses the shared NemoMelFrontend (512-point FFT, 400-sample symmetric Hann window, 160-sample hop, constant padding, natural-log mel energies, per-bin mean and unbiased-variance normalization). Whistle's 99th-percentile block RMS gain normalization is applied locally before it.

The encoder runs as one GGML graph on the selected backend. It is built from the framework Conv2dModule, DepthwiseConv2dModule, DepthwiseConv1dModule, LinearModule, RMSNormModule, GemmaRMSNormModule, RoPEModule, SiluModule, SigmoidModule, SoftmaxModule, and GLUModule. The stem does not use the shared DepthwiseConvSubsamplingModule because that module hardcodes ReLU and Whistle's stem uses SiLU. Whistle-specific math stays model-local in graph.cpp, shared by the encoder graph: the four-lane manifold hyper-connection mixing with its Sinkhorn normalization, the conditioned Kronecker-factored Hadamard MLP with its two fixed permutations, and the per-head RMS norm and rotary embedding over 48-wide heads. Checkpoint tensors are JAX [in, out] kernels; the loader transposes them once into the backend weight store.

The autoregressive decoder uses the original host implementation from PR #824, with OpenMP workers bounded by --threads. Its weights are retained across requests; self-attention caches and three-tap projection history reset for each recording. The encoder graph computes cross-attention keys and values, which are transferred to the host decoder once per request. Grouped-query and cross-attention preserve the model's 48-wide queries/keys and 64-wide values. Engram lookup, projections, gates and greedy argmax also run on the host.

This avoids thousands of small GGML decoder nodes per token. Four-thread CPU and Metal sessions benefit on the tested M4 Pro; one-thread decoding is slower than the previous one-thread GGML graph.

Build and request options

Follow the platform build instructions on Windows. For a focused build, select AUDIOCPP_MODEL_SET=custom and AUDIOCPP_MODELS=whistle_asr. A Windows helper example is:

.\scripts\build_windows.ps1 -Preset windows-cpu-release -ModelSet custom -Models "whistle_asr" -Target audiocpp_cli
cmake --build build/windows-cpu-release --target audiocpp_gguf

On macOS the Metal helper configures the same model set:

scripts/build_metal.sh --build-dir build/whistle --build-type Release --openmp auto --with-tests --deployment-build --model-set custom --models whistle_asr --target audiocpp_cli --target audiocpp_gguf

Use --backend to select an available backend: cpu, metal, cuda, or vulkan. --threads sets the CPU encoder's thread count and bounds the host decoder's OpenMP workers; the runtime accepts 1 to 64. Build with OpenMP enabled for parallel host decoding. Without OpenMP, the host decoder runs serially. An experimental decoder implementation reuses the framework NEON library for the FP32 vocabulary projection, retaining the same host attention, mHC, Hadamard and cache logic. Build with -DENGINE_ENABLE_NEON=ON on AArch64 and opt in with:

--session-option 'whistle_asr.exec_plan={"decoder":{"impl":"neon"}}'

The default is host; impl: "default" also selects it. Both implementations run on CPU independently of the encoder backend and use the session thread count. No artifacts or parameters are accepted. Existing embedded GGUF specs can use this option without reconversion. The spec exposes the optional language request option and exec_plan session option. Use --language de or --request-option language=de to force German. Omit the language option for model language selection. Decoding is greedy, with at most 320 text tokens.

Provenance and licensing

Use the pinned upstream revision. The converter preserves the upstream LICENSE inside the GGUF. The repository license does not grant rights to the weights or extracted assets. See the provenance record and model license table for source-specific terms and unresolved questions. No weights or extracted assets are included in this port.

Validation references

  • Official model, checkpoint, and license
  • Published Whistle reference reconstruction
  • whistle_frontend_test checks mel features against independently computed values.
  • whistle_tokenizer_test checks byte-fallback UTF-8 replacement and metaspace decoding without model weights.
  • whistle_assets_test <gguf> assets/resources/sample_16k.wav "Some call me Nature." checks a converted GGUF against the public sample's first three seconds. --backend cpu|metal|cuda|vulkan and --threads n select the backend; --full uses the whole recording.
  • whistle_assets_test <gguf> <wav> --reference <dir> [--tolerance f] compares the mel features, encoder memory, and cross-attention keys/values against .f32 reference dumps in <dir> and reports the largest difference as a fraction of each reference tensor's largest magnitude (default tolerance 2e-3; Metal's half-precision matmul operand staging needs --tolerance 5e-3). The same modes also run the decoder on the full recording. --dump writes decoder_logits_<position>.f32 (the full 8199-wide logits of each step), decoder_tokens.i32 (input tokens, BOS and language token first) and decoder_transcript.txt. --reference compares every logits file with the same scaled metric and requires identical tokens and transcript. In the validation report, the encoder dumps came from the earlier host-only implementation at commit 6e0a40c4. The decoder dumps came from the host decoder, before the step graph replaced it.

To run the native tests through CTest, configure with -DENGINE_BUILD_TESTS=ON and -DAUDIOCPP_WHISTLE_TEST_MODEL=<absolute-path-to-converted-GGUF>, then build whistle_frontend_test, whistle_tokenizer_test, and whistle_assets_test, then run ctest --test-dir <build-directory> -R whistle_ --output-on-failure. The opt-in tests check both the three-second excerpt and the entire public WAV, plus quiet and constant-RMS speech, silence, the 30-second limit, and decoder reuse.

See validation evidence for exact local commands, observed results, performance, and readiness blockers. Comparisons with the quantized official runtime are transcript checks, not exact FP32 numerical parity.