{"slug": "adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick", "title": "Adding GPU backends to a pure-C TTS engine: Metal, CUDA, and the rented-Mac trick", "summary": "A developer added opt-in GPU backends (Metal and CUDA) to a pure-C TTS inference engine for Qwen3-TTS, achieving faster-than-real-time performance on Apple Silicon and NVIDIA GPUs. The key optimization was making entire decode steps resident on the device, eliminating per-operation PCIe round trips. The project also implemented continuous request-batching, achieving up to 2.82× throughput improvement on an M2 Pro.", "body_md": "*Part of qwen3-tts — a pure C inference engine for Qwen3-TTS.*\n\nThe engine is pure C and **CPU by default**. We added two **opt-in** GPU backends that leave the CPU path untouched:\n\n`make metal`\n\n) — 0.6B model `make cuda`\n\n) — 1.7B model And two optimizations that *looked* obvious and that we **measured and threw away** (ICB and MMA). More on that at the end — it's the most useful part.\n\nRTF = processing time ÷ audio duration. **< 1.0 = faster than real time.**\n\nThe naive way to \"use the GPU\" is to offload one operation at a time — send the activation over, run a matmul, copy the result back. For a model that runs **16 sequential Code-Predictor passes per audio frame** plus a 28-layer Talker step per token, that's death by a thousand PCIe/round-trip cuts. We measured it: per-op offload was *slower* than the CPU.\n\nThe fix is the same on both backends: **make the whole step resident.** Weights, KV cache, and activations live on the device. A full decode step is encoded into **one command buffer / one kernel graph**, committed once, waited on once. The CPU orchestrates; it never babysits individual matmuls.\n\n`MTLCommandBuffer`\n\nper step + simdgroup matvec kernels.The biggest single win on Metal came from the **Code Predictor**. It was *sync-round-trip-bound* — 16 waits per frame. Moving the whole 16-pass RVQ loop (embed → 5 transformer layers → argmax, ×16) onto the device as **one command buffer with one sync/frame** took 0.6B from RTF 1.36 → 0.89. Direct kernel fusions (per-head RMSNorm+RoPE, residual-add+norm) took it to 0.72, and int4 weights to **0.60** — the practical floor on an M1, where the step is dispatch-bound.\n\nWe develop on an M1. To see what an M2 does, we rented a **Scaleway Mac mini M2 Pro by the hour**. Two things made this painless:\n\n**1. Metal shaders compile at runtime.** The MSL kernels are embedded as a string and compiled by the local Metal driver at startup (`newLibraryWithSource`\n\n). Consequence: a binary **built on an M1 drives an M2's GPU at full speed** — no rebuild needed, because the shaders are recompiled for whatever GPU is present. Only the CPU SIMD paths are baked in at build time.\n\n**2. One curl to bootstrap a bare box.** A fresh macOS box has `curl`\n\nbut not much else. A single script installs the Command Line Tools headlessly, clones the repo, pulls the models from HuggingFace, builds natively, and runs the benchmark:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/.../bootstrap_m2.sh | bash\n```\n\nThat gave us the first **cross-backend RTF matrix on real silicon** — CPU vs Metal (M1 & M2) vs CUDA (a real NVIDIA GPU, not a datacenter part). A gotcha worth writing down: **macOS has no timeout and no setsid** (both GNU-only). If your bench scripts wrap\n\n`curl`\n\nin `timeout`\n\n, they silently no-op on a Mac.Measured on the M2 Pro (0.6B, Metal, int8):\n\n| Mode | RTF | First audio (TTFA) |\n|---|---|---|\n| CLI / warm server (single) | 0.36–0.39 |\n— |\n| Streaming (single client) | 0.36 | 314 ms |\n\nint8 is the sweet spot on Apple Silicon: it's bandwidth-rich, so int4's nibble-unpacking doesn't pay off (that's the x86 lever).\n\nThis one trips people up, so it's worth being blunt: **batching does not make a single request faster.** In batch mode each step does the work of B slots, so per-request RTF *rises*. What you buy is **throughput** — you serve B concurrent users in roughly the wall-clock of one, because each weight row is read once (from DRAM) and reused for all B (matvec → matmat).\n\nWe wired continuous request-batching (`--serve --batch-size N`\n\n) into all three backends. On the M2 Pro, 0.6B:\n\n| Concurrent requests | Wall time | Batch speed-up |\n|---|---|---|\n| 1 | 19.5 s | — |\n| 2 | 21.9 s | 1.78× |\n| 4 | 27.7 s | 2.81× |\n\n4 requests served in 27.7 s instead of ~78 s serial. The 1.7B model scales identically (2.82× at B=4). CUDA does **3.35×** at B=8. And crucially, the **batch output is bit-identical to single-stream** — batching never changes what a user hears.\n\nOur first batched Metal matvec accumulated **scalar** (element by element). The single-stream kernel accumulated with ** float4 dot products**. Same math, different floating-point order → a ~1e-2 difference per step. In isolation this is benign (the argmax still matches). But TTS has a\n\nThe fix was three lines: vectorize the batched matvec to `float4`\n\nso it matches the reference kernel's FP order **exactly**. Result: bit-identical (RMS-rel 0.000), and the batch server produced mel-corr **1.00000** vs single-stream.\n\n**Lesson:** a numerically \"close enough\" kernel is not close enough inside a feedback loop. Match the reference's accumulation order.\n\nThe most valuable engineering this session was *not shipping* two things that looked obviously good.\n\n**ICB (Indirect Command Buffers) — the \"CUDA-graphs for Metal\" idea.** Pre-encode the step once, replay it. Before writing it, we profiled: the encode was only **12%** of the Talker step; the other 88% was GPU execution. ICB removes the 12% at best (~6–8% overall) for a large, invasive rewrite. And the Code Predictor — not the Talker — dominates the frame anyway. **Not built.**\n\n**MMA matmat (simdgroup_float8x8) for batched matvecs.** The matrix units are ~4.6× faster than a simdgroup matvec on a big B=32 tile. But at our batch sizes (B≤8) the 8×8 tile is half-empty and the kernel underutilizes the GPU. Measured on the M2: **better scaling ratio (3.27× vs 2.81×) but a 2× worse baseline** → net loss. It only wins at large B. **Kept opt-in, off by default.**\n\nBoth were killed by a five-minute measurement instead of a two-day build. That's the whole point.\n\nFull numbers, per-platform: [ docs/hardware-testing.md](https://github.com/gabriele-mastrapasqua/qwen3-tts) (Metal) and\n\n`docs/cuda-performance.md`", "url": "https://wpnews.pro/news/adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick", "canonical_source": "https://dev.to/gabrielemastrapasqua/adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick-16hh", "published_at": "2026-07-07 09:08:58+00:00", "updated_at": "2026-07-07 09:29:22.090354+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "large-language-models", "ai-infrastructure"], "entities": ["Qwen3-TTS", "Metal", "CUDA", "Scaleway", "Mac mini M2 Pro", "Apple Silicon", "HuggingFace", "NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick", "markdown": "https://wpnews.pro/news/adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick.md", "text": "https://wpnews.pro/news/adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick.txt", "jsonld": "https://wpnews.pro/news/adding-gpu-backends-to-a-pure-c-tts-engine-metal-cuda-and-the-rented-mac-trick.jsonld"}}