Recurrent Depth in LLMs: The Arithmetic Behind Looped Transformers
NVIDIA's Dory looped transformer stores 36 layers but executes 96 per token by running a 20-layer core four times, according to an arithmetic breakdown of recurrent-depth models published on piszczek.pl. Fastino's 10 Nov…