A scaling study of looped transformers finds that truncating backpropagation through time (BPTT) to the last few loops, a common method for saving training compute, yields biased gradients. The research examines how truncated BPTT affects the training of looped transformers, which reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters.
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased g
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.