Most scaling discussions treat compute as a single knob: throw more FLOPs at the problem, get predictable returns. Tang's framing separates the training compute budget from the inference reality in a way that feels obvious once you hear it but rarely gets modeled explicitly. His team's argument: if you're optimizing for a fixed inference budget (which every real deployment is), the optimal training run looks different than what pure loss curves suggest.
The practical takeaway that stuck: they're advocating for smaller, deeper architectures trained longer rather than the wider/shallower trend we've seen since GPT-3. Their internal ablation shows a 7B model trained on 3T tokens beating a 13B model on 1.5T tokens across their target benchmarks — coding, reasoning, Chinese NLP. The depth helps with the multi-step reasoning chains that benchmarks like GSM8K and HumanEval actually reward.
wide_config = {
"n_layers": 32,
"d_model": 5120,
"n_heads": 40,
"params": "~13B"
}
deep_config = {
"n_layers": 48,
"d_model": 4096,
"n_heads": 32,
"params": "~7B"
}
The inference speed difference is non-trivial. At batch size 1, the 7B deep model runs ~2.3x faster on H100s because the smaller hidden dimension means less memory bandwidth pressure per token. That compounds when you're serving thousands of concurrent requests.
What's interesting is how this interacts with mixture-of-experts. Tang mentioned they experimented with MoE but found the routing overhead ate most gains at their scale — the expert specialization only pays off past ~30B active params. For sub-10B models, dense still wins. That contradicts a lot of the current open-source MoE hype.
Data curriculum matters more than people admit. Their pipeline does three passes: deduplication → quality filtering (they use a small classifier trained on human annotations) → domain reweighting (code/math/reasoning upweighted 3x). The reweighting alone bought them ~0.8% on MMLU compared to uniform sampling. Not huge, but it's free compute.
One detail that surprised me: they're not chasing context length aggressively. 32k is their ceiling for now. The argument: long-context attention scales quadratically, and their target workloads (coding agents, RAG) rarely need more than 8-16k effective context once you account for retrieval. They'd rather spend that compute on deeper reasoning at standard lengths.
If you're building a practical LLM system today — not a research artifact — the Z.ai playbook is worth studying. Small dense model, deep architecture, long training run, careful data curriculum, optimize for your actual inference constraints. The math works out.
ByteDance is pushing 10 trillion parameters into a single model 11d ago
DeepSeek-V3 just leaked and it is actually terrifyingly good 11d ago
Over-guardrailing AI might actually be helping hackers win 13d ago
Maple-Preview: 120 tok/s 20B MoE on iPhone Defies Expectations 15d ago
Maple-Preview: 20B MoE Hits 120 tok/s on iPhone 15d ago
Export controls get the headlines 18d ago
Next Google drops twelve billion on Marvell for next-gen TPU work →