{"slug": "show-hn-turboquant-for-mlx-lm-apple-silicon", "title": "Show HN: TurboQuant for mlx-lm (Apple Silicon)", "summary": "A developer released TurboQuant, a pip-installable adapter for mlx-lm on Apple Silicon that uses a randomized Hadamard transform for data-oblivious, calibration-free quantization of weights and KV cache. The tool supports non-uniform quantization via a custom Metal kernel and can serve models with an OpenAI-compatible endpoint, achieving near-lossless compression at low bit widths.", "body_md": "A standalone, pip-installable **TurboQuant** adapter for\n[mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple silicon, with a custom\nMetal kernel for non-uniform quantization.\n\nTurboQuant ([Zandieh, Daliri, Hadian, Mirrokni, 2025](https://arxiv.org/pdf/2504.19874))\nis a *data-oblivious* (calibration-free) vector quantizer. Its core trick is a\n**random rotation** (a Randomized Hadamard Transform): rotating a vector spreads\noutliers across coordinates and turns the marginal into a concentrated,\nnear-Gaussian distribution that low-bit scalar quantizers handle gracefully.\nCrucially, an orthogonal rotation **preserves inner products**, so attention\nscores computed on rotated queries and keys are unchanged — which is what makes\nit so effective for KV-cache quantization.\n\nThis adapter implements both regimes:\n\n**Weights (MSE regime).** Rotate each weight matrix, then quantize. Rotation is the robust, always-on win; you can quantize with MLX's fast affine`quantized_matmul`\n\n(default) or with a custom**non-uniform Lloyd–Max LUT Metal kernel**(`--mode lut`\n\n).**KV cache (inner-product regime).** A drop-in`TurboQuantKVCache`\n\nthat stores keys in the rotated frame and rotates the query to match. This is where TurboQuant shines (see numbers below).\n\n```\npip install mlx-turboquant        # from PyPI\n# or, from source:\npip install -e .\n```\n\nRequires `mlx>=0.31.2`\n\nand `mlx-lm`\n\non macOS/Apple silicon.\n\n```\nturboquant convert --model mlx-community/Qwen3-0.6B-bf16 --out ./qwen3-tq4 --bits 4\n```\n\nProduces a standard mlx-lm model directory (safetensors + `config.json`\n\nwith a\n`quantization_config`\n\nof `quant_method: turboquant`\n\n).\n\n``` python\nimport mlx_turboquant as tq\ntq.register()                       # teach stock mlx-lm to load turboquant dirs\nfrom mlx_lm import load, generate\nmodel, tok = load(\"./qwen3-tq4\")\nprint(generate(model, tok, prompt=\"Why is the sky blue?\", max_tokens=128, verbose=True))\n```\n\nor the CLI:\n\n```\nturboquant generate --model ./qwen3-tq4 --prompt \"Why is the sky blue?\"\n```\n\n`turboquant serve`\n\nwraps `mlx_lm.server`\n\n: it installs the TurboQuant hooks (so a\nTurboQuant-quantized model dir loads through the stock server), optionally swaps\nin the rotated TurboQuant KV cache, then forwards every other flag straight to\n`mlx_lm.server`\n\n. The result is a drop-in OpenAI-compatible endpoint.\n\n```\n# Weight-quantized TurboQuant model (all mlx_lm.server flags pass through)\nturboquant serve --model ./qwen3-tq4 --port 8080\n\n# ...plus the rotated 4-bit KV cache, with the unbiased 1-bit QJL residual\nturboquant serve --model ./qwen3-tq4 --port 8080 --kv-bits 4 --qjl\n```\n\n**TurboQuant-specific flags** (everything else — `--host`\n\n, `--port`\n\n,\n`--adapter-path`\n\n, `--temp`\n\n, `--max-tokens`\n\n, `--trust-remote-code`\n\n, … — is parsed\nby `mlx_lm.server`\n\n; run `turboquant serve --help`\n\nfor the full list):\n\n| flag | default | effect |\n|---|---|---|\n`--kv-bits N` |\noff | enable the rotated TurboQuant KV cache at N bits (omit → normal fp16 cache) |\n`--kv-group-size N` |\n64 | KV quantization group size |\n`--qjl` |\noff | add the unbiased 1-bit QJL residual estimator (near-fp16 scores at low KV bits) |\n\n**Endpoints** (served by `mlx_lm.server`\n\n): `POST /v1/chat/completions`\n\n,\n`POST /v1/completions`\n\n, `GET /v1/models`\n\n.\n\nModel id gotcha:the`model`\n\nfield in each request must match the server's`--model`\n\n(the exact path or HF repo id). A short/renamed id makes the server try toload that as a new modelfrom the Hub (→ 401 \"Repository Not Found\"). Use the same string you launched with, or omit`model`\n\nto use the default.\n\n```\n# non-streaming\ncurl http://127.0.0.1:8080/v1/chat/completions -H \"Content-Type: application/json\" -d '{\n  \"model\": \"./qwen3-tq4\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Name two primary colors.\"}],\n  \"max_tokens\": 64, \"temperature\": 0.0\n}'\n\n# streaming (Server-Sent Events)\ncurl -N http://127.0.0.1:8080/v1/chat/completions -H \"Content-Type: application/json\" -d '{\n  \"model\": \"./qwen3-tq4\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Count to five.\"}],\n  \"stream\": true\n}'\n```\n\nOr with the official OpenAI Python client:\n\n``` python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://127.0.0.1:8080/v1\", api_key=\"not-needed\")\nresp = client.chat.completions.create(\n    model=\"./qwen3-tq4\",                       # match the server's --model\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\nprint(resp.choices[0].message.content)\n```\n\n**Requires mlx-lm>=0.31.3** — the server runs each request in a worker thread,\nand older mlx-lm uses a single global generation stream that fails there\n(\n\n`RuntimeError: no Stream(gpu, 0)`\n\n); 0.31.3+ uses thread-local streams. (`pip install mlx-turboquant`\n\npulls a compatible version automatically.) The stock\n`mlx_lm.server ...`\n\nalso works for TurboQuant **weight** models if you call\n\n`mlx_turboquant.register()`\n\nin the process first — `turboquant serve`\n\njust does\nthat for you and adds the KV-cache flags.The KV cache is applied at generation time to any (even unquantized) model:\n\n``` python\nimport mlx_turboquant as tq\nfrom mlx_lm import load, generate\nmodel, tok = load(\"mlx-community/Qwen3-1.7B-bf16\")\ncache = tq.make_prompt_cache(model, kv_bits=4, kv_group_size=64)   # rotated KV\n# or, for the unbiased 1-bit QJL residual estimator (+1 bit/channel):\ncache = tq.make_prompt_cache(model, kv_bits=3, kv_group_size=64, qjl=True)\ngenerate(model, tok, prompt=..., max_tokens=256, prompt_cache=cache, verbose=True)\n```\n\nThe rotated-MSE key `k̂`\n\ngives a *biased* attention score:\n`<Rq, k̂> = <Rq, Rk> − <Rq, r>`\n\n(it under-counts by the residual inner product\n`<Rq, r>`\n\n), and the bias is **largest for the high-similarity keys softmax\nweights most**. TurboQuant fixes this with a 1-bit **QJL** sketch of the residual\n`r = Rk − k̂`\n\n: store `sign(RHT₂(r))`\n\n(1 bit/channel) and `‖r‖`\n\n, then estimate\n`<Rq, r> ≈ √(π/2d)·‖r‖·⟨RHT₂(Rq), sign(RHT₂(r))⟩`\n\n. Adding it back yields an\n**unbiased** score. Verified numerically ([tests/test_qjl.py](/pythongiant/mlx_turboquant/blob/main/tests/test_qjl.py)):\nfor correlated (query, key) pairs the MSE-only bias of ≈ −0.05 is removed to\n≈ 0. Enable with `qjl=True`\n\n.\n\nTeacher-forced perplexity with a quantized KV cache (lower is better):\n\n**Qwen3-1.7B** (fp16 reference = 2.93):\n\n| KV bits | plain affine KV | TurboQuant KV |\nTurboQuant + QJL (+1 b/ch) |\n|---|---|---|---|\n| 8 | 2.91 | 2.93 | 2.91 |\n| 4 | 31.39 💥 | 3.16 |\n3.03 ✅ |\n| 3 | 4625 | 77.5 | 4.82 |\n| 2 | 2.1e6 | 28704 | 6937 |\n\n**Two effects, both the paper's claims, reproduced:**\n\n**Rotation**— at 4-bit, plain affine KV collapses (ppl ~31) while TurboQuant's rotated KV stays near-neutral (3.16). Rotation preserves inner products and removes the outliers that wreck low-bit affine KV.**QJL residual**— the +1-bit unbiased correction closes the last gap: 4-bit KV becomes fp16-neutral (2.93), and it rescues 3-bit KV from unusable (77.5) to usable (5.55). This matches the paper's \"quality-neutral at ~3.5 bits/channel\".\n\n(2-bit weight-*only* and ≤3-bit KV without QJL break these small models; the\nlarger the model, the lower the bits you can push.)\n\nMedian over **50 ShareGPT prompts** of varied length (13–2631 tokens, median\n224), 64 decode tokens each, on Apple silicon. `prepare_sharegpt.py`\n\nsamples the\nprompts; `throughput.py`\n\nruns the grid.\n\n| config | decode tok/s | TPOT (ms) | TTFT (ms) | weights | KV @ 2k ctx |\n|---|---|---|---|---|---|\n| MLX LM bf16 (baseline) | 26.6 | 37.6 | 327 | 3.44 GB | 0.23 GB |\nTurboQuant 4-bit + KV4 |\n56.6 |\n17.7 |\n330 | 1.42 GB |\n0.066 GB |\n| TurboQuant 4-bit + KV4 + QJL | 27.6 | 36.3 | 543 | 1.42 GB | 0.074 GB |\n\n**Recommended config — TurboQuant 4-bit weights + rotated 4-bit KV:**\n\n**~2.1× faster decode** than bf16 (56.6 vs 26.6 tok/s) and**~2.1× lower time-per-token**(17.7 vs 37.6 ms) — memory-bound decode loves 4-bit weights, and the rotated KV cache adds essentially no overhead.** 2.4× smaller weights**(3.44 → 1.42 GB) and**~3.5× smaller KV cache**(0.23 → 0.066 GB at 2k context) — so you fit far longer contexts in the same memory, the real constraint for on-device long-context inference.- All while staying\n**near fp16 quality** at 4-bit KV where plain affine KV collapses (see above).\n\n**QJL mode** (`qjl=True`\n\n) is the *maximum-quality / maximum-compression* option:\nit makes 3-bit KV usable and 4-bit KV fp16-neutral for only +1 bit/channel\n(0.066 → 0.074 GB). It trades decode speed for that quality (the unbiased\ncorrection adds an extra sketch dot-product per step), so reach for it when you\nare memory-bound at very low KV bits and want fp16-grade scores.\n\nThe KV cache — not the weights — is what grows with context and dominates memory\nfor long sequences. Since MLX's affine KV is unusable at 4-bit (ppl ~31), the\nhonest comparison is **iso-quality**: to stay near fp16, affine needs **8-bit**\nKV while TurboQuant is neutral at **4-bit**, so TurboQuant's KV cache is **~1.9×\nsmaller at matched quality** (and 3.5× smaller than fp16).\n\nMeasured on Qwen3-1.7B (`benchmarks/kv_memory.py`\n\n→ `plot_kv_memory.py`\n\n); the\nShareGPT prompt range is shaded, extrapolated to long context. **KB/token is\nmeasured directly** from the cache arrays (verified constant across 2k/4k/8k, and\nmatching the throughput benchmark's independent 0.066 GB @2k), so the GB columns\nare *exact* linear scaling — not estimates. **ppl** is the quality proxy from\n`kv_quality.py`\n\nat 509-token context (not re-measured at 32k/128k):\n\n| KV config | quality (ppl) | KB/token | KV @ 32k | KV @ 128k |\n|---|---|---|---|---|\n| fp16 | 2.93 | 112.0 | 3.76 GB | 15.0 GB |\n| MLX affine 8-bit | 2.91 | 59.5 | 2.00 GB | 8.0 GB |\n| MLX affine 4-bit | 31.4 ✗ | 31.5 | 1.06 GB | 4.2 GB |\nTurboQuant 4-bit |\n3.16 |\n31.5 | 1.06 GB |\n4.2 GB |\nTurboQuant 3-bit + QJL |\n4.82 |\n28.4 | 0.95 GB | 3.8 GB |\n\nTurboQuant 4-bit uses the *same* bytes as affine 4-bit but is actually usable\n(3.16 vs 31.4). At 128k context that's a **4.2 GB KV cache instead of 8 GB\n(affine, matched quality) or 15 GB (fp16)** — often the difference between\nfitting long context on-device or not.\n\nTurboQuant needs two operations MLX can't express with built-ins, so the adapter\nships two hand-written `mx.fast.metal_kernel`\n\ns:\n\n**Non-uniform LUT dequant + matmul**(`kernels/qmm.py`\n\n).`mx.quantized_matmul`\n\nonly supports uniform/affine codebooks; TurboQuant's MSE-optimal quantizer uses a**non-uniform Lloyd–Max codebook**. The kernel does LUT dequant + matmul directly on packed indices (one SIMD group reduces over`K`\n\nper output),`bits ∈ {2,4,8}`\n\n, and cuts weight-reconstruction MSE ~15% vs affine at 2-bit. Enable with`--mode lut`\n\n.**Packed-sign QJL inner product**(`kernels/qjl_dot.py`\n\n). The unbiased KV correction computes`Σ_d qproj[d]·sign_bit(d)`\n\nagainst the 1-bit residual sketch. The kernel reads the packed`uint32`\n\nsign words directly and accumulates`±qproj`\n\nper bit in fp32 — no dense unpack, no extra full matmul — powering the QJL KV path.\n\nRotations run on Metal via `mx.hadamard_transform`\n\n.\n\n**Weights:**`nn.Linear → TurboQuantLinear`\n\nmodule swap (same pattern as mlx-lm's`bitnet_quantize`\n\n).`register()`\n\nwraps`mlx_lm.utils.load_model`\n\nso turboquant dirs load through the stock`mlx_lm.load`\n\n/`generate`\n\n/`server`\n\n.**KV cache:**`TurboQuantKVCache`\n\nsubclasses mlx-lm's`QuantizedKVCache`\n\n(inherits all mask/state logic), rotates keys, and (with`qjl=True`\n\n) stores the 1-bit residual sketch;`register()`\n\npatches`scaled_dot_product_attention`\n\nto rotate the query and add the QJL correction.\n\n- A fused RHT Metal kernel (currently we reuse MLX's already-optimized\n`mx.hadamard_transform`\n\n). - 3-bit LUT packing (32 not divisible by 3).\n\n```\npip install -e \".[test]\"\npytest -q                 # rotation invariance, codebook MSE, kernel numerics, KV + QJL correctness\npython benchmarks/kv_quality.py  --model mlx-community/Qwen3-0.6B-bf16   # KV perplexity\npython benchmarks/throughput.py  --model mlx-community/Qwen3-1.7B-bf16   # TPOT/TTFT over ShareGPT\npython benchmarks/kv_memory.py && python benchmarks/plot_kv_memory.py    # memory chart\n```\n\nMIT licensed. Not affiliated with the TurboQuant authors or Apple.", "url": "https://wpnews.pro/news/show-hn-turboquant-for-mlx-lm-apple-silicon", "canonical_source": "https://github.com/pythongiant/mlx_turboquant", "published_at": "2026-07-07 11:52:04+00:00", "updated_at": "2026-07-07 11:59:16.079334+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["TurboQuant", "mlx-lm", "Apple Silicon", "Metal", "Lloyd–Max", "Qwen3", "PyPI", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-turboquant-for-mlx-lm-apple-silicon", "markdown": "https://wpnews.pro/news/show-hn-turboquant-for-mlx-lm-apple-silicon.md", "text": "https://wpnews.pro/news/show-hn-turboquant-for-mlx-lm-apple-silicon.txt", "jsonld": "https://wpnews.pro/news/show-hn-turboquant-for-mlx-lm-apple-silicon.jsonld"}}