GPT-2 Small

A controlled study of GPT-2 Small found that representational simplicity and causal circuit size dissociate in a threshold-dependent way, with adversarially trained models needing substantially fewer …

A new 306M-parameter language model architecture, the Cognitive Categorical Transformer (CCT), achieved 21.27 validation perplexity on WikiText-103, a 12% relative improvement over a fine-tuned GPT-2 …