ACL Anthology +2 Proceedings of Machine Learning Research +2 + 2 + 2 Proposal: Progressive Model Growth and an Open Hardware Profile for Transformers

A proposal for progressive model growth and an open hardware profile for Transformers notes that function-preserving model expansion is an established research line dating to Net2Net in 2016, with bert2BERT extending it to Transformers at roughly 47% FLOPs reduction and later work including LLaMA Pro, SOLAR, LESA, and MSG pushing depth-scaling. The proposal argues depth growth outperforms width growth for training speedups of 30-50% less compute, and that a Transformer-specific hardware profile covering RoPE, GQA, KV-cache, and MoE routing would fill a gap left by ONNX Execution Providers and OpenXLA/StableHLO. It suggests checking whether that gap explains why ASIC vendors Groq, Etched, and Cerebras build for one architecture generation at a time.

Substantial post, worth engaging point by point rather than a generic reply. On model growth specifically, this isn’t hypothetical, it’s an active research line with a decade of results behind it: Net2Net 2016 started function-preserving width/depth expansion, bert2BERT extended it to Transformers with ~47% FLOPs reduction, and more recent work LLaMA Pro, SOLAR, LESA, MSG has pushed depth-scaling specifically since it composes better with parallel training than width expansion. The consistent finding across this literature: depth growth duplicating/stacking layers gives the best speedup, width growth is less effective and harder to keep function-preserving cleanly. So your Stage 1→2→3 idea already has a name and a body of evidence, the honest answer to “why isn’t it a first-class Transformers workflow” is probably that the training-efficiency gain 30-50% less compute hasn’t been compelling enough yet to outweigh the simplicity of just training each size independently with well-understood scaling laws, not that nobody’s tried it. On the hardware profile idea, this also already exists in a different form: ONNX + Execution Providers, and OpenXLA/StableHLO are both attempts at exactly “stable target so hardware vendors don’t need N special cases,” and they’ve had real adoption every major chip vendor ships an ONNX Runtime execution provider . Where your proposal differs is scoping it specifically to LLM/Transformer primitives RoPE, GQA, KV-cache, MoE routing rather than general neural-net ops, that’s a fair gap, ONNX’s op set is architecture-agnostic and doesn’t give an ASIC vendor a “here’s exactly what to hardwire for Transformers” target the way a narrower profile would. Worth checking whether that gap is actually why ASIC vendors Groq, Etched, Cerebras keep building for one architecture generation at a time instead of a stable common profile, that’d tell you if the standardization problem is real or if hardware vendors already work around it some other way.