Rethinking LLM scaling after Jie Tang's latest breakdown

Jie Tang's team at Z.ai argues that LLM scaling should optimize for fixed inference budgets, favoring smaller, deeper architectures trained longer over wider models. Their ablation shows a 7B model trained on 3T tokens outperforming a 13B model on 1.5T tokens across coding, reasoning, and Chinese NLP benchmarks, with the 7B deep model running ~2.3x faster on H100s at batch size 1. They also found dense models beat mixture-of-experts below ~30B active parameters, and domain reweighting (code/math/reasoning upweighted 3x) bought ~0.8% on MMLU.

Rethinking LLM scaling after Jie Tang's latest breakdown Most scaling discussions treat compute as a single knob: throw more FLOPs at the problem, get predictable returns. Tang's framing separates the training compute budget from the inference reality in a way that feels obvious once you hear it but rarely gets modeled explicitly. His team's argument: if you're optimizing for a fixed inference budget which every real deployment is , the optimal training run looks different than what pure loss curves suggest. The practical takeaway that stuck: they're advocating for smaller, deeper architectures trained longer rather than the wider/shallower trend we've seen since GPT-3. Their internal ablation shows a 7B model trained on 3T tokens beating a 13B model on 1.5T tokens across their target benchmarks — coding, reasoning, Chinese NLP. The depth helps with the multi-step reasoning chains that benchmarks like GSM8K and HumanEval actually reward. Rough architecture comparison they reference Wide-shallow traditional scaling wide config = { "n layers": 32, "d model": 5120, "n heads": 40, "params": "~13B" } Narrow-deep Z.ai preference deep config = { "n layers": 48, "d model": 4096, "n heads": 32, "params": "~7B" } The inference speed difference is non-trivial. At batch size 1, the 7B deep model runs ~2.3x faster on H100s because the smaller hidden dimension means less memory bandwidth pressure per token. That compounds when you're serving thousands of concurrent requests. What's interesting is how this interacts with mixture-of-experts. Tang mentioned they experimented with MoE but found the routing overhead ate most gains at their scale — the expert specialization only pays off past ~30B active params. For sub-10B models, dense still wins. That contradicts a lot of the current open-source MoE hype. Data curriculum matters more than people admit. Their pipeline does three passes: deduplication → quality filtering they use a small classifier trained on human annotations → domain reweighting code/math/reasoning upweighted 3x . The reweighting alone bought them ~0.8% on MMLU compared to uniform sampling. Not huge, but it's free compute. One detail that surprised me: they're not chasing context length aggressively. 32k is their ceiling for now. The argument: long-context attention scales quadratically, and their target workloads coding agents, RAG /en/tags/rag/ rarely need more than 8-16k effective context once you account for retrieval. They'd rather spend that compute on deeper reasoning at standard lengths. If you're building a practical LLM system today — not a research artifact — the Z.ai playbook is worth studying. Small dense model, deep architecture, long training run, careful data curriculum, optimize for your actual inference constraints. The math works out. ByteDance is pushing 10 trillion parameters into a single model 11d ago /en/news/5656/ DeepSeek-V3 just leaked and it is actually terrifyingly good 11d ago /en/news/5653/ Over-guardrailing AI might actually be helping hackers win 13d ago /en/news/5424/ Maple-Preview: 120 tok/s 20B MoE on iPhone Defies Expectations 15d ago /en/news/5085/ Maple-Preview: 20B MoE Hits 120 tok/s on iPhone 15d ago /en/news/5019/ Export controls get the headlines 18d ago /en/news/4714/ Next Google drops twelve billion on Marvell for next-gen TPU work → /en/news/7054/