Architecture Research as Addressing Constraints to Scaling
Architecture research aims to remove bottlenecks to scaling rather than merely tweaking loss, according to a post by Beren on beren.io. The transformer's attention mechanism and residual stream enable…