GPT2

Researchers introduced Stiefel-AdamW, a near drop-in replacement for the AdamW optimizer that constrains one factor of linear factorization blocks to the Stiefel manifold, according to arXiv paper 260…