Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

A scaling study of looped transformers finds that truncating backpropagation through time (BPTT) to the last few loops, a common method for saving training compute, yields biased gradients. The research examines how truncated BPTT affects the training of looped transformers, which reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters.

Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time BPTT . To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased g