# ACL Anthology +2 Proceedings of Machine Learning Research +2 + 2 + 2 Proposal: Progressive Model Growth and an Open Hardware Profile for Transformers > Source: > Published: 2026-09-20 13:53:26+00:00 Substantial post, worth engaging point by point rather than a generic reply. On model growth specifically, this isn’t hypothetical, it’s an active research line with a decade of results behind it: Net2Net (2016) started function-preserving width/depth expansion, bert2BERT extended it to Transformers with ~47% FLOPs reduction, and more recent work (LLaMA Pro, SOLAR, LESA, MSG) has pushed depth-scaling specifically since it composes better with parallel training than width expansion. The consistent finding across this literature: depth growth (duplicating/stacking layers) gives the best speedup, width growth is less effective and harder to keep function-preserving cleanly. So your Stage 1→2→3 idea already has a name and a body of evidence, the honest answer to “why isn’t it a first-class Transformers workflow” is probably that the training-efficiency gain (30-50% less compute) hasn’t been compelling enough yet to outweigh the simplicity of just training each size independently with well-understood scaling laws, not that nobody’s tried it. On the hardware profile idea, this also already exists in a different form: ONNX + Execution Providers, and OpenXLA/StableHLO are both attempts at exactly “stable target so hardware vendors don’t need N special cases,” and they’ve had real adoption (every major chip vendor ships an ONNX Runtime execution provider). Where your proposal differs is scoping it specifically to LLM/Transformer primitives (RoPE, GQA, KV-cache, MoE routing) rather than general neural-net ops, that’s a fair gap, ONNX’s op set is architecture-agnostic and doesn’t give an ASIC vendor a “here’s exactly what to hardwire for Transformers” target the way a narrower profile would. Worth checking whether that gap is actually why ASIC vendors (Groq, Etched, Cerebras) keep building for one architecture generation at a time instead of a stable common profile, that’d tell you if the standardization problem is real or if hardware vendors already work around it some other way.