TurboQuant, Four Months Later: Chasing Google's 6x VRAM Claim Into the Wild

Back in Q1, I read a headline about Google cutting AI memory use by 6x and filed TurboQuant under "watch and revisit" — no code, tested only up to 8B parameters, nothing to actually run against AI-NT-No-Problem

. Four months is a long time in this industry. I went back to see what actually happened, and the honest answer is: a lot, but not the thing I expected.

Quick recap for anyone who missed the original story. TurboQuant is a training-free algorithm suite — TurboQuant proper, plus PolarQuant and Quantized Johnson-Lindenstrauss — that compresses the KV cache specifically, not model weights, cutting memory by at least 6x with an 8x speedup in attention computation on H100s. The paper, "Online Vector Quantization with Near-optimal Distortion Rate," came out of Google Research and Google DeepMind and was accepted at ICLR 2026.

Here's the part that hasn't changed since March: as of the most recent status I could confirm, Google still hasn't shipped official code. The original "expected Q2 2026" timeline for an official release has quietly passed without one landing anywhere I can find.

What happened instead is the pattern anyone who's watched an ML paper drop before recognizes: two weeks after the ICLR paper, five independent implementations already existed, including one running a 104B parameter model on a MacBook. Four months later that's grown to at least eight or nine separate forks and packages, from a pip-installable HuggingFace wrapper to CUDA/Triton implementations to an AMD ROCm-specific fork.

The interesting shift is in tone, not just headcount. The maintainer of one of the more actively maintained forks is now openly walking back some of the original hype: the community has converged on a more nuanced picture than the initial hype suggested, currently recommending plain FP8 KV cache as the best default on Hopper/Blackwell hardware, and reaching for TurboQuant only when you need more than 2x compression and are willing to accept some throughput cost. That's a meaningfully more conservative position than "6x memory, zero accuracy loss" read in March.

For our actual stack, this is the part that matters most. There's an open PR proposing two new KV cache quantization types for llama.cpp, tbq3_0 (about 3.06 bits per element) and tbq4_0

(about 4.06 bits per element), adapting TurboQuant's two-stage recipe to GGML's block format. As of the last status check I could find, it's still open and under review, not merged into mainline.

What is available today: several community forks already ship it as a runtime flag, something like --cache-type-k turbo3 --cache-type-v turbo3 alongside -fa on

, and critically it applies only at the KV cache layer — no re-quantization or re-conversion of existing GGUF weights required. If we wanted to try this against our own llama-server

setup, we could point one of these forks at the exact same Qwen 3.5/3.6 GGUF we already run.

Independent evaluation is where the picture diverges hardest from the launch-week coverage. One evaluation from Red Hat AI and the vLLM team found meaningful accuracy drops on reasoning and very long context at 3-bit precision, particularly when the QJL residual-correction step is left enabled. Multiple independent teams reached the same conclusion from different directions: the paper's own "extra bit of correction" step often hurts more than it helps at low bit widths, with plain MSE-only quantization beating MSE+QJL across every model the community has tested.

There's also real hardware data putting a number on the actual problem this is trying to solve. One llama.cpp discussion thread includes measured DGX Spark GB10 results showing existing q4_0 KV cache quantization is already 36.8% slower than f16 at roughly 110K tokens of context, purely from per-token dequantization overhead — that's the exact bottleneck fused TurboQuant-style kernels are built to remove, and it's a legitimate, measured problem independent of how any individual fork's compression ratio claims hold up.

Two things line up well, and one thing is a real caveat worth testing before trusting any number.

The good news: at least one fork is explicitly tested on dense and MoE architectures across RTX 3090 and RTX 5090 GPUs with a vLLM/Triton integration, and a separate Windows build explicitly targets the CUDA 13.x runtime — both a much closer match to our actual hardware than the original paper's H100/A100-only validation.

The caveat: the original paper only validated on head_dim=128 models (Gemma, Mistral, Llama-3.1-8B). At least one fork found that head_dim=64 doesn't Gaussianize well enough for the core rotation-then-quantize math to hold, requiring a fallback to plain q8_0, and a separate community benchmark thread flagged that a Qwen3.6 model's GQA head_dim=256 configuration caused one implementation's K-cache to come out larger than plain q8_0 on that architecture specifically. Our daily driver is Qwen 3.5/3.6 35B-A3B. That's not a "probably fine" caveat — it's a "test this exact model before believing any compression number" caveat.

Same instinct as the LocalAI bakeoff plan: no YAML in this post, but the shape of the test is worth writing down now.

1. Head-dim compatibility | Does TurboQuant's math actually hold for Qwen 3.5/3.6's GQA config, or does it degrade like the community report suggests? | Direct KV-cache size and perplexity comparison, TurboQuant vs. our current q4_0 /q8_0 cache types, same model, same prompts |
2. Long-context throughput | Does it actually fix the dequantization slowdown, or just move the cost around? | Token/sec at 24K, 64K, and 128K+ context, mirroring the DGX Spark data above |
3. Known failure-mode check | Does the QJL step help or hurt at the bit widths we'd actually use? | Binary pass/fail against MSE-only vs. MSE+QJL, same test set |
4. Daily-drive soak test | Does the compression survive real agentic traffic, not just synthetic long-context benchmarks? | Run OpenClaw against the TurboQuant-patched llama-server fork for a few days |

If layer 1 fails outright, on our actual model, the rest doesn't matter, and that's worth knowing before writing a single line about a 6x win. Worth being upfront about: a fair amount of what's indexing for "TurboQuant" right now reads like SEO-optimized rewrites of the original launch coverage, plus a long tail of solo-developer GitHub forks with very confident, very polished README claims, versioned like production software, benchmark tables and all. The specific compression multipliers floating around (4.6x, 5.2x, 8.9x, 12x, take your pick) come from different implementations that haven't been reconciled against each other. None of that means the underlying idea is wrong. It means the number in any given headline, including the 6x one I originally filed this under, deserves a "reproduce it yourself" asterisk before it goes anywhere near a decision about this homelab.

tbq3_0

/ tbq4_0

), the actual path to mainline supportq4_0

KV cache quantization at ~110K context, the real problem being solved hereThe 6x headline was real, as far as the original paper's own benchmarks go. Whether it survives contact with our exact model, our exact GPU, and four months of community re-litigation is a different question, and it's the only one worth actually testing.