Two production systems ship TurboQuant, and neither ships all of it. Qdrant 1.18 uses the MSE variant plus extensions that are not in the paper, and vLLM’s TurboQuant module states that QJL is intentionally omitted.
That omission matters. QJL is the stage the paper adds to the rotation recipe to make inner products unbiased, and the two places that count have set it aside.
Disclosure: Claude, an AI assistant, wrote most of this draft. Each figure links to its source, and the code snippet is runnable.
TL;DR
Most quantizers store a scale, and often a zero point, for every small block of values. Google’s research post puts that overhead at 1 or 2 extra bits per number. As an illustrative case (assumption: blocks of 32 values, one fp16 scale and one fp16 zero point), the constants cost 32 bits per 32 values. A 4-bit code then costs 5 bits per value.
TurboQuant’s pitch is a quantizer with no per-block constants and no training pass. That fits two jobs: compressing a KV cache while tokens stream in, and indexing vectors without learning a codebook first. The paper calls this data-oblivious.
Claim: the random rotation is what lets a single fixed codebook work for any input.
The mechanism is short. Multiply each unit vector by one shared random orthogonal matrix. Each coordinate of the result follows a known Beta distribution, which approaches a Gaussian with variance 1/d at large dimension d, whatever the input looked like. Distinct coordinates also become nearly independent, so quantizing each one on its own loses little.
A concrete case: take a vector with most of its energy in 8 of 256 coordinates. A fixed grid fits it badly, because most coordinates sit near zero and a few sit far outside the grid. After rotation, every coordinate is a draw from the same bell curve.
import numpy as nprng = np.random.default_rng(0)d, n = 256, 2000# Anisotropic unit vectors: 8 dominant coordinatesX = rng.standard_normal((n, d)) * np.r_[np.full(8, 20.0), np.ones(d - 8)]X /= np.linalg.norm(X, axis=1, keepdims=True)# One shared random rotation (QR of a Gaussian matrix)Q, R = np.linalg.qr(rng.standard_normal((d, d)))Pi = Q * np.sign(np.diag(R))def lloyd_max(samples, k, iters=60): c = np.quantile(samples, (np.arange(k) + 0.5) / k) for _ in range(iters): idx = np.searchsorted((c[:-1] + c[1:]) / 2, samples) c = np.array([samples[idx == j].mean() for j in range(k)]) return cdef quant(v, c): return c[np.searchsorted((c[:-1] + c[1:]) / 2, v)]# Codebook fitted once on N(0, 1/d), the large-d limitg = rng.standard_normal(400_000) / np.sqrt(d)for b in (1, 2, 3, 4): c = lloyd_max(g, 2 ** b) rot = np.mean(np.sum((X - quant(X @ Pi.T, c) @ Pi) ** 2, axis=1)) raw = np.mean(np.sum((X - quant(X, c)) ** 2, axis=1)) print(f"b={b} rotated={rot:.4f} unrotated={raw:.4f} lower bound={4.0**-b:.4f}")
With NumPy 2.4 the snippet prints rotated MSE of 0.361, 0.116, 0.034 and 0.009 for 1 to 4 bits. The paper reports 0.36, 0.117, 0.03 and 0.009 for unit vectors. The unrotated run, with the same codebook and the same data, prints 1.05, 0.64, 0.46 and 0.37. The paper’s lower bound is 4^-b, which is 0.25 at one bit.
Condition: the bound assumes unit norm. Norms are stored separately in floating point, and that detail matters later.
Claim: a fixed codebook is only as good as the Gaussian assumption behind it.
Qdrant reports that the proof rests on vectors spread uniformly over the sphere and does not extend to anisotropic embeddings, where a few directions carry most of the variance. Its fix is a per-segment calibration pass. It estimates a shift and a scale per coordinate from a reservoir sample using streaming quantile estimates, and folds both into the query side so search pays nothing extra. On isotropic data the correction is the identity.
Qdrant also stores a per-vector length ratio, in four bytes it already reserves, that undoes the shrinkage plain MSE quantization causes. It credits that idea to RaBitQ, not to the TurboQuant paper.
My reading, labeled as such: the paper’s bound averages over the choice of rotation for a fixed input, while a deployed index draws one rotation and lives with it. Calibration targets the second situation. This is an interpretation of the two sources, not a claim from either.
Claim: QJL solves a genuine problem and is still the first thing production removes.
The problem is real. At one bit the MSE-optimal codebook is plus or minus the square root of 2/(pi d), and the paper shows the resulting inner product estimate carries a multiplicative bias of 2/pi. The bias shrinks as bits increase. TurboQuant-prod spends b minus 1 bits on the MSE quantizer, then applies QJL, the sign of a random Gaussian projection, to the residual. The result is unbiased.
Qdrant ships the MSE variant for three stated reasons: HNSW construction needs scores between two stored vectors, the correction costs bits that renormalization recovers for one scalar, and PROD adds a per-query random projection to every score. vLLM’s docs say community reports from five or more independent groups found QJL hurts attention quality by amplifying variance through softmax. I could not verify those five groups from the docs alone.
The mechanism is plausible on paper. Softmax exponentiates logits, so zero-mean error in a score becomes multiplicative error in an attention weight, and the largest weights dominate. That is a derivation, not a measurement. The condition matters too: unbiasedness pays when you average many independent estimates, and it pays less when you rank or exponentiate a single noisy score.
Qdrant’s benchmark uses ten datasets with HNSW (m=16, ef_construct=128). This is a vendor benchmark, so treat it as one setup.
Qdrant advises staying on binary quantization if throughput drops noticeably on your workload.
Perplexity hides the tail. An independent llama.cpp KV benchmark ran Qwen 3.6 27B on one RTX 3090 at 64k context with Q5_K_S weights. Through q4_0, the whole perplexity range stayed under 0.01. The 99.9th-percentile KL precision told a different story: 94.61% for q8_0, 89.84% for q4_0, 76.12% for turbo3 and 41.47% for turbo2. The caveats are real. It is a community fork, one model, one GPU, and the author lists his own limits.
Rotation was already in the box. That benchmark notes llama.cpp applies a random rotation before quantizing, so TurboQuant’s rotation is not new there. At 4 bits, turbo4 scored 89.13% against 89.84% for q4_0, with about 17% lower prefill throughput. The value showed up at 2 to 3 bits, mostly through the trellis-coded variants rather than the plain turbo modes, not at 4.
The headline numbers need a baseline. Google’s post claims at least 6x KV memory reduction at 3 bits and up to 8x faster attention logits. Check the arithmetic against a 16-bit cache (derived, ignoring norms and scales): 3.5 bits is about 4.6x, 3 bits is about 5.3x, and 6x implies roughly 2.7 bits. The paper itself says more than 4.5x. The post’s 8x compares 4-bit keys with 32-bit unquantized keys on an H100, which is a logits benchmark, not decode throughput.
Attribution and benchmark parity are disputed. The authors of RaBitQ published a technical note saying both methods start with a shared random rotation, that the paper’s RaBitQ timings came from a single CPU core with multithreading off against an A100 for TurboQuant, and that those setups were not disclosed. In their symmetric setup, RaBitQ on the same GPU ran about 1.2x to 1.8x faster, and they could not reproduce the paper’s reported times from the released code. In their KV harness, after fixing bugs in the released code, TurboQuant-mse scored 0.709 on needle-in-a-haystack at 2.5 bits against 0.951 for RaBitQ, and 0.962 against 0.977 at 3.5 bits. The authors of DRIVE and EDEN published a separate note arguing that TurboQuant-mse is EDEN with the scale fixed to 1.
Weigh those sources properly. They are arXiv notes from the authors of the prior work, not peer-reviewed, and I found no published rebuttal. A Milvus interview says a TurboQuant co-author agreed to revise the arXiv text on RaBitQ’s optimality. I could not confirm a revision. Qdrant, which has no stake in the dispute, describes RaBitQ as developing the same rotate-then-quantize foundation and uses RaBitQ’s renormalization.
What survives in production is the rotation, a fixed Lloyd-Max codebook, and a cheap per-vector scalar. What gets added is calibration for real embeddings. What gets dropped is the QJL correction. What stays contested is lineage and the head-to-head benchmarks.
Before you adopt it, run your own vectors at your own bit width, and measure a tail metric instead of perplexity alone. Compare against scalar quantization with rotation, since the rotation may account for much of the gain. That last point is my inference from the llama.cpp result, not a finding.
One question has no public answer yet. Nobody has published tail behavior at 2 to 3 bits on long agentic traces full of tool calls, where one dropped token in a structured argument breaks the run. The benchmark author names that as untested. Until someone measures it, the 3-bit KV cache is a claim about average quality.
TurboQuant in Production: Qdrant, vLLM, and What Survived was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.