{"slug": "velum-cosyvoice-in-a-single-binary-for-deployment", "title": "Velum: CosyVoice in a Single Binary for Deployment", "summary": "HardenedLinux released Velum, a native C++ reimplementation of the CosyVoice3 text-to-speech pipeline that runs with zero Python at runtime, executing its LLM, Flow and HiFT neural networks on GGML with CUDA acceleration. The project ships as a single executable binary, with weight conversion and the acoustic frontend (campplus, speech tokenizer, matcha mel) computed once offline in Python and frozen into files the C++ binary loads verbatim. All four phases — DSP frontend, Flow decoder, HiFT vocoder and the Qwen2-0.5B LLM backbone — are implemented and numerically verified against the PyTorch reference, while the ONNX frontend and matcha 80-bin mel remain deferred and are offered only in a paid enterprise edition.", "body_md": "A native reimplementation of the [CosyVoice3](https://github.com/QwenAudio/CosyVoice) text-to-speech pipeline with **zero\nPython at runtime**. The neural networks (LLM / Flow / HiFT) run on\n[GGML](https://github.com/ggml-org/ggml) with CUDA acceleration. Weight\nconversion and the acoustic frontend (campplus / speech tokenizer / matcha mel)\nare computed **once, offline, in Python** and frozen into files the C++\nbinary loads verbatim.\n\n**It's designed for better deployment in product as a single executable binary file.**\n\nThis project is Human architectured and co-authored by AI.\n\n- LLM: deepseek-v4-pro\n- Coding Assistant: Claude Code\n\n``` js\nldd velum\n    linux-vdso.so.1 (0x00007ffceb3fd000)\n    libicuuc.so.74 => /lib/x86_64-linux-gnu/libicuuc.so.74 (0x00007aab5e400000)\n    libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007aab66b91000)\n    libcudart.so.12 => /lib/x86_64-linux-gnu/libcudart.so.12 (0x00007aab5e000000)\n    libcublas.so.12 => /lib/x86_64-linux-gnu/libcublas.so.12 (0x00007aab57600000)\n    libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007aab51e00000)\n    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007aab51a00000)\n    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007aab5e717000)\n    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007aab66b61000)\n    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007aab51600000)\n    libicudata.so.74 => /lib/x86_64-linux-gnu/libicudata.so.74 (0x00007aab4f800000)\n    /lib64/ld-linux-x86-64.so.2 (0x00007aab66c0a000)\n    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007aab66b5a000)\n    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007aab66b55000)\n    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007aab66b50000)\n    libcublasLt.so.12 => /lib/x86_64-linux-gnu/libcublasLt.so.12 (0x00007aab2e800000)\n```\n\nAll four phases are implemented and verified numerically against the PyTorch reference:\n\n1. DSP frontend (Whisper 128-bin log-mel + Kaldi 80-bin fbank) — `verify_dsp.py`\n2. Flow decoder (PreLookaheadLayer + DiT ×22 + CFM Euler) — `verify_flow.py`\n3. HiFT vocoder — `verify_hift.py`\n4. LLM backbone (Qwen2-0.5B + CosyVoice3LM heads + ras_sampling) — `verify_llm*.py`\n\nThe end-to-end CLI (LLM → Flow → HiFT) is wired and cross-checked by\n`tests/verify_e2e.py`. Still **deferred** (pre-extracted by Python): the ONNX\nfrontend (campplus + speech tokenizer) and the matcha 80-bin mel — the CLI reads\ntheir outputs as files instead of running them.\n\nPlease read [Missing Features](https://github.com/hardenedlinux/velum/blob/main/docs/missing-features.md), these missings will not be added in the community edition, we offer consulting service for enterprise edition.\nPlease contact [consulting@hardenedvault.com](mailto:consulting@hardenedvault.com).\n\n| dir | purpose | \n|---|---|\n| `src/dsp/` | Whisper 128-bin log-mel + Kaldi 80-bin fbank (verified) | \n| `src/llm/` | Qwen2 autoregressive decoder + CosyVoice3LM heads + ras_sampling (verified) | \n| `src/flow/` | DiT flow-matching estimator (verified) | \n| `src/hift/` | Causal HiFi-GAN vocoder (verified) | \n| `src/pipeline/` | orchestration: tokenizer → LLM → Flow → HiFT | \n| `src/cli/` | `velum` end-to-end entry point | \n| `src/frontend/` | reserved for the deferred ONNX frontend | \n| `tools/` | offline prep: `convert_weights.py` ,`export_tokenizer.py` ,`gguf.py` ,`gen_mel_filters.py` | \n| `tests/` | `verify_*.py` numerical checks +`export_*.py` /`extract_prompt_features.py` asset producers | \n| `docs/` | `ARCHITECTURE.md` , ADRs,`DSP.md` ,`FLOW.md` ,`HIFT.md` ,`LLM.md` ,`WEIGHT_FORMAT.md` | \n\n```\ncmake -S . -B build            # enables CUDA if a toolkit is detected\ncmake --build build -j\n```\n\nThis produces `velum` plus the `velum_*_dump` verification utilities. CUDA is\nauto-detected: `ggml`'s CUDA backend is compiled when `VELUM_ENABLE_CUDA=ON`\n(default) *and* a CUDA toolchain is found; otherwise it builds CPU-only. Both\nbackends are linked into `velum` — at runtime it picks CUDA when a device is\npresent and falls back to CPU. Force the CPU backend with `VELUM_BACKEND=cpu`\n(used by the numerical verify scripts so they don't depend on an idle GPU).\n\nNote: the LLM is large. `llm.gguf` is ~2.6 GB, so a CUDA run needs that much\nfree VRAM (plus the Flow graph). If `cudaMalloc` reports out-of-memory, either\nfree the GPU or prefix the run with `VELUM_BACKEND=cpu`.\n\nTwo Python environments are used:\n\n- **`.venv`** — repo-local,` torch` +`numpy` , for weight conversion.\n- **CosyVoice python3.10** — the reference environment that can import`cosyvoice` /`transformers` , for tokenizer/asset/prompt extraction.\n\n```\n# paths used below\nMODEL=\"$HOME/Project/CosyVoice/pretrained_models/Fun-CosyVoice3-0.5B\"\nPY310=\"$HOME/.local/share/uv/python/cpython-3.10-linux-x86_64-gnu/bin/python3.10\"\nPYTHONPATH=\"$HOME/Project/CosyVoice/.local/lib/python3.10/site-packages\"\n```\n\n**1. Convert weights** (`.venv`):\n\n```\n.venv/bin/python tools/convert_weights.py \\\n  --llm \"$MODEL/llm.pt\" --flow \"$MODEL/flow.pt\" --hift \"$MODEL/hift.pt\" \\\n  --out-dir build/\n```\n\nwrites `build/llm.gguf` / `build/flow.gguf` / `build/hift.gguf` (format-only\nconversion, no quantization). Use `--llm \"$MODEL/llm.rl.pt\"` for the RL-tuned\ncheckpoint.\n\n**2. Export the text tokenizer** (python3.10):\n\n```\nPYTHONPATH=\"$PYTHONPATH\" \"$PY310\" tools/export_tokenizer.py --out-dir build/tokenizer\n```\n\nwrites `vocab.tsv` / `merges.txt` / `added_tokens.tsv`.\n\n**3. Export the fixed RNG buffers** (python3.10):\n\n``` php\n\"$PY310\" tests/export_hift_source.py    # -> build/hift_source.bin  (HiFT SineGen2 rand_ini + sine_waves)\n\"$PY310\" tests/export_flow_noise.py     # -> build/flow_noise.bin   (Flow CFM seed noise)\n```\n\nThese are the model-internal buffers the reference samples once from PyTorch's RNG at construction; the C++ side loads the frozen values instead of reimplementing the RNG.\n\n**4. Extract the prompt-voice bundle** (python3.10):\n\n```\n\"$PY310\" tests/extract_prompt_features.py --out-dir wavs/flow_inputs\n```\n\nruns campplus + speech tokenizer + matcha mel on the prompt wav and writes\n`prompt_tokens.i32` / `prompt_feat.f32` / `spk_embedding.f32` (the deferred\nfrontend, \"temporarily handed to Python\"). Pass `--prompt-wav <wav>` to use a\ndifferent voice.\n\n```\n./build/velum \\\n  --text \"今天天气不错，我们一起去公园散步吧。\" \\\n  --prompt-dir wavs/flow_inputs \\\n  --out wavs/hello.wav\n```\n\nModel/asset paths default to `build/llm.gguf`, `build/flow.gguf`,\n`build/hift.gguf`, `build/hift_source.bin`, `build/flow_noise.bin` and\n`build/tokenizer`. `--text` is required; `--instruct` defaults to\n`\"You are a helpful assistant. 请用普通话表达。<|endofprompt|>\"` and must contain\n`<|endofprompt|>`. Optional dumps:\n\n```\n./build/velum --text ... --prompt-dir wavs/flow_inputs --out wavs/hello.wav \\\n  --seed 0 \\\n  --dump-tokens wavs/hello.tokens.i32 \\\n  --dump-mel    wavs/hello.mel.f32 \\\n  --dump-audio  wavs/hello.audio.f32\n```\n\n`--seed` drives the LLM sampling RNG; the speech-token sequence is stochastic,\nso different seeds (or no `--seed`) give different audio. `VELUM_BACKEND=cpu`\nforces CPU.\n\n```\nctest --test-dir build            # DSP / flow / hift / tokenizer / llm numerical checks\n\"$PY310\" tests/verify_e2e.py      # end-to-end CLI vs PyTorch (CPU, slower)\n```\n\nThe `ctest` suite needs the reference `.npz` dumps, which are regenerated by the\nmatching `tests/*_reference.py` scripts (see their docstrings). See\n`docs/DSP.md`, `docs/FLOW.md`, `docs/HIFT.md`, `docs/LLM.md` for the measured\nerror numbers.\n\nEvery cross-check against the PyTorch reference reports a scale-normalized\nrelative error `rel = max|C++ − ref| / max|ref|` and classifies each stage:\n\n| class | rel err | meaning | \n|---|---|---|\n| **GREEN** | ≤ 1e-2 (≤ 1%) | numerically correct — matches the reference within float32 accumulation | \n| **YELLOW** | 1e-2 … 1e-1 (1%–10%) | above the pass gate; warrants investigation, not yet a proven divergence | \n| **RED** | > 1e-1 (> 10%) | structural divergence (wrong op / layout / missing clip) — hard fail | \n\n`GREEN` is the same gate the verify scripts enforce (`fail if rel > 1e-2`);\n`YELLOW`/` RED` are escalation bands above it. A `RED` stage is never accepted.\n\nThe full CLI chain (LLM → Flow → HiFT) cross-checked against the PyTorch\nreference, seed 0, `--text \"今天天气不错，我们一起去公园散步吧。\"`:\n\n| backend | Flow mel (max abs / rel) | HiFT pcm (max abs / rel) | class | \n|---|---|---|---|\n| CPU | 1.142e-3 / 1.056e-4 | 5.630e-3 / 8.112e-3 | **GREEN** | \n| CUDA | 2.220e-3 / 2.052e-4 | 3.495e-3 / 5.036e-3 | **GREEN** | \n\nBoth backends are **GREEN**. The CPU HiFT pcm (8.112e-3) sits just inside the\n1% line (0.81%) — HiFT's nonlinear (exp/snake/phase) synthesis amplifies the\nFlow mel's float32 accumulation (see `docs/HIFT.md`). The CUDA path pins cuBLAS\nto `CUBLAS_DEFAULT_MATH` (TF32 disabled, `docs/adr/0002`) and releases the LLM\nweights after generation, since the resident LLM + Flow DiT graph (~4 GiB) do not\nfit an 8 GiB card together. Not a bug; a real divergence would land in `RED`.\n\nThe GREEN margin is **sequence-length dependent**: the HiFT max error is\nconcentrated on a few isolated onset samples and grows with mel length. On a\nlonger utterance — the ad-copy text with Pronunciation-Inpainting markers\n(`<strong>…</strong>`, `[j][ǐ]`), 286 mel frames / 5.72 s — the HiFT pcm max\nerror is 1.465e-2 (rel 2.427e-2, **YELLOW**), but the mean stays 6.9e-5 and only\n17 of 137280 samples (0.012%) exceed 1%. This is the same isolated-onset\naccumulation, **not** a marker effect: the Flow/HiFT stages never see the text\n(only the tokenizer does, and it is bit-exact on the PI markers).\n\nThe Flow decoder's CFM noise is **fixed at seed 0 by design**. The reference\n`CausalConditionalCFM.__init__` samples `rand_noise = torch.randn([1,80,50*300])`\nonce, under `set_all_random_seed(0)`, and every inference slices\n`z = rand_noise[:,:,:n]` from that single frozen buffer. The C++ decoder loads\nthat exact buffer from `build/flow_noise.bin` (exported by\n`tests/export_flow_noise.py`) and reuses it for **every** synthesis — this is a\ndeliberate, permanent design choice that keeps the decoder deterministic and\nreproducible, **not** a configurable option and **not** a per-run RNG. There is\nno seed flag for it (see `docs/FLOW.md`).\n\n- `docs/ARCHITECTURE.md` — authoritative design.\n- `docs/adr/0001-drop-onnx-runtime-for-compute.md` — why ONNX Runtime is frontend-only.\n- `docs/DSP.md` /`docs/FLOW.md` /`docs/HIFT.md` /`docs/LLM.md` — per-stage reference + validation numbers.\n- `docs/WEIGHT_FORMAT.md` — GGUF tensor organisation.", "url": "https://wpnews.pro/news/velum-cosyvoice-in-a-single-binary-for-deployment", "canonical_source": "https://github.com/hardenedlinux/velum", "published_at": "2026-09-26 02:16:15+00:00", "updated_at": "2026-09-26 02:31:13.800296+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Velum", "CosyVoice3", "HardenedLinux", "GGML", "Qwen2-0.5B", "CUDA", "Claude Code", "deepseek-v4-pro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/velum-cosyvoice-in-a-single-binary-for-deployment", "markdown": "https://wpnews.pro/news/velum-cosyvoice-in-a-single-binary-for-deployment.md", "text": "https://wpnews.pro/news/velum-cosyvoice-in-a-single-binary-for-deployment.txt", "jsonld": "https://wpnews.pro/news/velum-cosyvoice-in-a-single-binary-for-deployment.jsonld"}}