{"slug": "llama-cpp-0-9-0-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the", "title": "llama.cpp 0.9.0 Ships Chatterbox TTS and OCP FP8 — Local Speech Still Dies on the Windows Capture Path", "summary": "Llama.cpp v0.9.0, released 4 November 2026, adds OCP FP8 E4M3 quantization, a flash_attn_ext_rows op that restores unified KV cache prefill performance for multi-sequence work, early NCCL multi-node tensor parallel via ggml 0.29.0, and first-class Chatterbox TTS and Nemotron 3.5 Transcribe support through a new llama-tts tool. The release makes Chatterbox Turbo and Multilingual V3 convertible to GGUF for fully local TTS and STT without a Python runtime or cloud round-trip. The writeup notes that on Windows the OS audio capture path, not the model, remains the dominant latency term, with WASAPI buffering adding 80-120 ms before samples reach the process.", "body_md": "llama.cpp v0.9.0 dropped on 4 November 2026. The release notes are short and concrete: OCP FP8 E4M3 quantization (saturating mode, new ftype `LLAMA_FTYPE_MOSTLY_F8_E4M3`), a `flash_attn_ext_rows` op that finally makes the unified KV cache competitive for multi-sequence work, ggml bumped to 0.29.0 with early NCCL multi-node tensor parallel, and first-class support for Chatterbox TTS plus Nemotron 3.5 Transcribe. Four new architectures land at the same time (Clover, Limite, Metis-1.6, Mistral Large 4). For anyone building local speech pipelines the audio bits are the interesting ones.\n\nThe practical surface is the new `llama-tts` tool and the multimodal side of the server. Chatterbox Turbo (English, ~350M) and Multilingual V3 (23 languages) are now convertible to GGUF. You feed a speaker reference (or use the predefined voice from `conds.pt`), set language on the multilingual checkpoint, and get a WAV out. Nemotron 3.5 Transcribe gets offline transcription in 40 languages with prompt context and speaker attribution. Predefined Pocket TTS voices are also listed via the server’s `/props` endpoint. On paper this is the cleanest path yet to a fully local TTS + STT stack that does not require a separate Python runtime or a cloud round-trip for every utterance.\n\nConversion still follows the usual GGUF path. The upstream Chatterbox checkpoints ship without a standard `config.json`, so you assemble the required files (talker, s3gen, ve, conds.pt, tokenizer bits) in one directory and run `convert_hf_to_gguf.py`. Once you have the GGUF + mmproj pair:\n\n```\nllama-tts -m chatterbox-mtl.gguf \\\n  -mm mmproj-chatterbox-mtl.gguf \\\n  -p \"The capture path is still the bottleneck.\" \\\n  --tts-lang en \\\n  --tts-speaker-file ref.wav \\\n  --output out.wav\n```\n\nThe multilingual variant runs classifier-free guidance with a second sequence. `llama-tts` wires that up for you; the server does not yet. FP8 E4M3 is exposed as a tensor type with conversion helpers, so you can quantize larger models (including the new Mistral Large 4) further if you are memory-constrained. The `flash_attn_ext_rows` change is the one that actually matters for concurrent sessions: previously the unified KV cache destroyed prefill performance when you tried to keep multiple short audio-related sequences alive. That tax is largely gone.\n\nOn Windows the pre-built binaries already embed an application manifest (new in this release). That removes one of the usual “why is my CUDA DLL load failing under the service account” surprises when you drop `llama-server` into a scheduled task or a local overlay process.\n\nNone of the above changes the dominant term in a real-time language tool. On Windows the capture path still owns the budget.\n\nWASAPI loopback or exclusive-mode capture, the audio engine’s mixing, any Bluetooth or USB class driver in the chain, and the process isolation around the overlay all sit in front of the model. A local TTS that can emit a sentence in 150–250 ms is useless if the microphone path has already burned 80–120 ms of buffering and the loopback path has added another 40–60 ms of latency before the first sample reaches your process. The model is no longer the long pole; the OS audio graph is.\n\nTypical measurements on a recent Windows 11 machine with an onboard Realtek and a USB headset still show:\n\nYou can shave the model side with FP8, smaller Chatterbox quant, and the improved multi-seq attention. You cannot shave the capture side without measuring the exact device and graph you are attached to. That is why tools that do live translation or interview overlays still spend more engineering time on `IAudioClient` initialization, `AUDCLNT_STREAMFLAGS_LOOPBACK`, and process-priority / MMCSS registration than they do on the GGUF loader.\n\nFor a local pipeline the useful split is:\n\nThe TTS step can now stay inside the same llama.cpp process that is already loaded for other local inference. That removes one network hop and one process boundary. It does not remove the need to keep the audio graph stable across Windows updates. A regression harness that dumps the current `IAudioClient` buffer sizes and the loopback endpoint after each monthly update is still more valuable than another 5 % on the TTS RTF.\n\n```\n# after each Windows update, re-measure the actual capture path\n# (example skeleton — adapt to your own measurement binary)\n.\\capture-probe.exe --device loopback --duration 5 --json > loopback-$(Get-Date -Format yyyyMMdd).json\n```\n\nFP8 helps when you want to keep a larger multimodal model resident alongside the TTS weights. The new ftype and conversion helpers mean you can re-quantize without leaving the llama.cpp toolchain. The tradeoff is the usual one: saturating FP8 can clip more aggressively on the audio projector or the final decoder layers; test the specific checkpoint on your target hardware before you ship it into a live path.\n\nIf you are already on llama.cpp for local models, 0.9.0 is a clean upgrade for speech. You get first-class Chatterbox support, Nemotron Transcribe, better multi-sequence efficiency, and FP8 without a new runtime. On Windows the pre-built binaries are less annoying to run as a service. The capture and mixing path underneath remains the part you still have to own, measure, and keep working after the next feature update.\n\nThat is the same constraint that shows up in any real-time language tool that has to stay out of the way of the call or the interview: the model can now live on the machine, the speech can be generated locally, and the latency budget is still decided by how cleanly you attach to the Windows audio engine.\n\nI'm building [Talkaside](https://talkaside.com/), a Windows app for job interviews and calls in another language.\n\n*Drafted with AI assistance from this project's code and logs.*", "url": "https://wpnews.pro/news/llama-cpp-0-9-0-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the", "canonical_source": "https://dev.to/madnessbrains_i/llamacpp-090-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the-windows-capture-10n0", "published_at": "2026-11-05 06:16:37+00:00", "updated_at": "2026-11-05 06:23:50.231306+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-infrastructure", "developer-tools", "machine-learning"], "entities": ["llama.cpp", "Chatterbox", "Nemotron 3.5 Transcribe", "ggml", "Mistral Large 4", "NVIDIA", "Windows 11", "WASAPI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llama-cpp-0-9-0-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the", "markdown": "https://wpnews.pro/news/llama-cpp-0-9-0-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the.md", "text": "https://wpnews.pro/news/llama-cpp-0-9-0-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the.txt", "jsonld": "https://wpnews.pro/news/llama-cpp-0-9-0-ships-chatterbox-tts-and-ocp-fp8-local-speech-still-dies-on-the.jsonld"}}