04:39
2026-06-24
lesswrong.com
machine-learning
Toy transformers may represent belief-state geometry optimally but not minimally
A toy transformer retains provably predictively defunct belief state data in its residual stream, shedding it only under sufficient capacity pressure. Experiments extending Shai et al. 2024 show that …