AIArticle Cloudflare's production numbers for Kimi K2.6 and GLM 5.2 show where trillion-parameter inference actually gets cheaper.
Priya Nair The open-weight frontier now weighs a trillion parameters. Moonshot's Kimi K2.6 is a 1T-parameter mixture-of-experts model with 32B active per token; Z.ai's GLM 5.2 is 744B with 40B active. Both ship under MIT-style licenses, both sit on Hugging Face, and neither fact matters much if you can't afford to serve them. Access stopped being the bottleneck this year. Serving economics is the bottleneck now — and Cloudflare just published a set of production engineering notes, with real numbers, on how it runs both models behind Workers AI.
The techniques themselves — FP8 KV caches, INT4 weight-only quantization — are not new. What's rare is a provider showing its fleet-scale numbers, including the accuracy deltas and the trade-offs that cut against its own marketing. That transparency is the actual story, and it's worth pulling apart.
The KV cache is the real tenant #
At 256K-token context windows, the model weights aren't what eats your GPU — the KV cache is. Every concurrent request holds its own cache, and cache size scales with context length. For Moonshot's Kimi K2.6 on H200s, Cloudflare stores the decode-time cache in FP8 (e4m3) instead of BF16, halving it. Capacity jumps from roughly 686K to 1.37M cached tokens, which doubles peak concurrency from 32 requests to 64 and lifts throughput to 2,192 tokens/second — about 41% over the BF16 peak, at roughly 30% lower cost per token.
The honest part is the fine print: a single isolated request gets about 9% slower. This is the trade every serverless inference provider makes and few disclose — the fleet is tuned for batch throughput, and your individual request pays a small latency tax so the platform's economics work. Accuracy held up under their evals (GSM8K went from 94.24% to 94.09%, MMLU was flat), which is consistent with what the community has found about FP8 caches generally. But know what you're buying: if you're running a latency-critical single-stream workload, a throughput-optimized multi-tenant deployment is structurally not tuned for you.
Quantization is a per-phase decision, not a per-model one #
The most transferable idea in the post is architectural. Cloudflare runs prefill and decode as separate GPU pools — disaggregated serving, the pattern that's quietly become standard for large-model inference — and that split lets precision differ by phase.
Decode is memory-bandwidth-bound: every generated token re-reads the weights, so shrinking them pays directly. Compressing Z.ai's GLM 5.2 from its native FP8 down to INT4 cut the checkpoint from 705GB to 421GB — 52GB per GPU instead of 88GB across an 8-way tensor-parallel node — and lifted single-request decode 55% (60 to 92 tokens/second). Prefill is the opposite: compute-bound, and the INT4 dequantization overhead actually made it slower (8,660 vs. 10,160 tokens/second). So they serve INT4 for decode and FP8 for prefill.
If you self-host, this reframes the question you should be asking. Not "should I quantize GLM 5.2?" but "which phase am I bound on?" The levers are all in SGLang, the open-source framework Cloudflare benchmarked with — FP8 KV cache is a config flag, and INT4 weight-only quantization is well-trodden AWQ/GPTQ territory. Sobering caveat: an INT4 GLM 5.2 at 421GB still needs an 8×H200-class node. Quantization improves the economics of serving these models; it does not shrink the entry fee into single-GPU territory. Kimi K2.6 at 1T parameters is further out of reach still.
The failure mode nobody else writes about #
The "safer" third of the post is the most interesting, because it names a risk class the industry mostly doesn't discuss. Modern inference servers share KV cache aggressively — paged allocation, prefix caching, radix trees — across hundreds of concurrent requests from different customers. A bookkeeping bug in that machinery means your tokens get generated conditioned on fragments of someone else's context. That's simultaneously a correctness bug and a tenant-isolation leak.
Cloudflare's mitigation is unglamorous and correct: tag physical cache pages with the request they belong to and verify tags before decode reads them. Overhead measured under 1% on throughput and under 0.8% on p95 latency. The detail worth registering isn't the mechanism — it's that a major provider considered the risk real enough to build runtime integrity checking for it. If you operate your own multi-tenant inference cluster on vLLM or SGLang, cross-request cache corruption should be on your threat model, and today you mostly get no equivalent guardrail out of the box.
Trust, but verify the endpoint #
Here's the uncomfortable implication for everyone consuming open models through APIs: the "GLM 5.2" behind any given endpoint is not the checkpoint on Hugging Face. It's some provider-specific stack of quantization and serving choices, and quality varies. This isn't hypothetical — Moonshot published a vendor-verification suite for its K2 line precisely because tool-calling accuracy across third-party K2 providers turned out to be wildly inconsistent.
Cloudflare disclosing its benchmark deltas — even tiny unflattering ones like that GSM8K dip — is exactly the norm the ecosystem needs, and you should treat it as table stakes when picking a provider. If an inference vendor won't tell you the precision it serves at and the measured accuracy delta versus the reference checkpoint, assume the answer is unfavorable.
Where this goes #
None of the individual techniques here is research-grade novelty, and the HN crowd rightly noted the evals could be broader — a couple of benchmark suites is thin evidence for "no quality loss" on agentic workloads, where small per-step degradation compounds over hundreds of tool calls. Run your own task-level evals before assuming the deltas are free.
But the direction is clear. Serving efficiency is now where open-model competition happens: the weights are free, so margin lives entirely in tokens-per-GPU-hour. Expect the serving-time tricks to migrate into training time — GLM 5.2 already ships native FP8 weights, and DeepSeek's multi-head latent attention showed you can shrink the KV cache architecturally instead of quantizing it after the fact. The providers that win the open-model API market will be the ones that publish numbers like these and let you hold them to it.
Sources & further reading #
[Smaller, faster, safer: running Kimi and GLM at scale](https://blog.cloudflare.com/smaller-faster-safer-models/)— blog.cloudflare.com -
[Smaller, faster, safer: running Kimi and GLM at scale](https://news.ycombinator.com/item?id=49158581)— news.ycombinator.com -
[Moonshot AI Kimi K2.6 now available on Workers AI](https://developers.cloudflare.com/changelog/post/2026-04-20-kimi-k2-6-workers-ai/)— developers.cloudflare.com -
Moonshot AI releases Kimi K2.6 model with 1T parameters, attention optimizations— siliconangle.com -
[GLM-5.2 API](https://www.together.ai/models/glm-52)— together.ai
[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.