{"slug": "acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-model-an", "title": "ACL Anthology +2 Proceedings of Machine Learning Research +2 + 2 + 2 Proposal: Progressive Model Growth and an Open Hardware Profile for Transformers", "summary": "A proposal for progressive model growth and an open hardware profile for Transformers notes that function-preserving model expansion is an established research line dating to Net2Net in 2016, with bert2BERT extending it to Transformers at roughly 47% FLOPs reduction and later work including LLaMA Pro, SOLAR, LESA, and MSG pushing depth-scaling. The proposal argues depth growth outperforms width growth for training speedups of 30-50% less compute, and that a Transformer-specific hardware profile covering RoPE, GQA, KV-cache, and MoE routing would fill a gap left by ONNX Execution Providers and OpenXLA/StableHLO. It suggests checking whether that gap explains why ASIC vendors Groq, Etched, and Cerebras build for one architecture generation at a time.", "body_md": "Substantial post, worth engaging point by point rather than a generic reply.\n\nOn model growth specifically, this isn’t hypothetical, it’s an active research line with a decade of results behind it: Net2Net (2016) started function-preserving width/depth expansion, bert2BERT extended it to Transformers with ~47% FLOPs reduction, and more recent work (LLaMA Pro, SOLAR, LESA, MSG) has pushed depth-scaling specifically since it composes better with parallel training than width expansion. The consistent finding across this literature: depth growth (duplicating/stacking layers) gives the best speedup, width growth is less effective and harder to keep function-preserving cleanly. So your Stage 1→2→3 idea already has a name and a body of evidence, the honest answer to “why isn’t it a first-class Transformers workflow” is probably that the training-efficiency gain (30-50% less compute) hasn’t been compelling enough yet to outweigh the simplicity of just training each size independently with well-understood scaling laws, not that nobody’s tried it.\n\nOn the hardware profile idea, this also already exists in a different form: ONNX + Execution Providers, and OpenXLA/StableHLO are both attempts at exactly “stable target so hardware vendors don’t need N special cases,” and they’ve had real adoption (every major chip vendor ships an ONNX Runtime execution provider). Where your proposal differs is scoping it specifically to LLM/Transformer primitives (RoPE, GQA, KV-cache, MoE routing) rather than general neural-net ops, that’s a fair gap, ONNX’s op set is architecture-agnostic and doesn’t give an ASIC vendor a “here’s exactly what to hardwire for Transformers” target the way a narrower profile would. Worth checking whether that gap is actually why ASIC vendors (Groq, Etched, Cerebras) keep building for one architecture generation at a time instead of a stable common profile, that’d tell you if the standardization problem is real or if hardware vendors already work around it some other way.", "url": "https://wpnews.pro/news/acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-model-an", "canonical_source": "https://discuss.huggingface.co/t/acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-progressive-model-growth-and-an-open-hardware-profile-for-transformers/180643#post_2", "published_at": "2026-09-20 13:53:26+00:00", "updated_at": "2026-09-20 14:23:25.697985+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure", "ai-chips", "machine-learning"], "entities": ["Net2Net", "bert2BERT", "LLaMA Pro", "SOLAR", "LESA", "MSG", "ONNX", "OpenXLA"], "alternates": {"html": "https://wpnews.pro/news/acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-model-an", "markdown": "https://wpnews.pro/news/acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-model-an.md", "text": "https://wpnews.pro/news/acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-model-an.txt", "jsonld": "https://wpnews.pro/news/acl-anthology-2-proceedings-of-machine-learning-research-2-2-2-proposal-model-an.jsonld"}}