Confirming Claims of Superposition and Adversarial Examples in Toy Models

A replication by the University of Chicago xLab confirms claims from Stevinson et al. (arXiv:2510.11709) that adversarial examples in toy models exploit interference between features in superposition, with PGD attacks aligning with the optimal perturbation at cosine similarity 1.00 across sparsity 0.98, exceeding the paper's reported 0.92–0.97. The replication also supports the paper's finding that models without superposition are more robust to adversarial examples.

This is a replication of Adversarial Attacks Leverage Interference Between Features in Superposition , completed as part of the Second Look Summer Fellowship. Our code can be found here https://github.com/uchicago-xlab/superposition-replication and we encourage readers to fork/run followup experiments Elhage et al., 2022 https://transformer-circuits.pub/2022/toy model/index.html defines superposition as the phenomenon where “models represent more features than they have dimensions.” To study this in a toy setting, they define a classifier where features are squeezed into an m -dimensional bottleneck with , so feature directions interfere in the latent space. Stevinson et al. https://arxiv.org/abs/2510.11709 , the origin of the experiments in this replication, argues this interference is a mechanism behind adversarial examples. The paper makes three claims: 1. adversarial attacks exploit interference, 2. models without superposition are more robust to adversarial examples, 3. correlations in the input data allow models to converge to similar geometry allowing adversarial attacks to transfer between models. The experiments in the paper deal with a toy model with encoder and decoder . For input the output is . Here is the number of classes and The model uses cross entropy loss. Features are sampled independently with probability , where s can be thought of as how sparsely features are activated within the data. Adversarial attacks are computed by projected gradient descent PGD Madry et al., 2019 https://arxiv.org/abs/1706.06083 . For a given sample , the adversarial example is found by PGD, where indicates how the attack modified the original sample. measures distance between the original example and the adversarial example and ε is the attack budget which enforces that . We use PGD learning rate , at 50 steps with 5 restarts, and . The values for k and We follow the hyperparameters described in the paper as of July, 2026. 1 https://www.lesswrong.com/feed.xml fnpwkfegvcd Stevinson et al. https://arxiv.org/abs/2510.11709 hypothesize that adversarial attacks in toy models work by exploiting the interference geometry. Proposition 1 from the paper shows the optimal perturbation has a closed form solution: where w s, w t are the source/target class vectors rows of the decoder W d and W e the encoder, scaled to the ε-ball. The authors test if the PGD perturbation matches the optimal attack derived from interference geometry, empirically by: At a high level, this can be thought of as letting the attack maximize loss in an arbitrary direction and testing if it finds an optimal direction for a given target class. Figure 1 shows the untargeted attack live where and We use untargeted attacks for these experiments to follow the methodology of the paper, but we also repeat this experiment for targeted attacks with results in Appendix A. The claim that the PGD attack aligns with the optimal attack, exploiting the interference geometry, replicates cleanly on the toy model. With sparsity 0.98, we demonstrate that PGD attacks align with optimal perturbation. We believe our cosine similarity results are higher since we let PGD iterate to find the best perturbation while the paper stops at the first prediction flip. k | m | clean acc | PGD·optimal ours | PGD·optimal paper | |---|---|---|---|---| 6 | 2 | 0.998 | 1.00 | 0.97 | 30 | 10 | 0.996 | 1.00 | 0.96 | 90 | 30 | 0.966 | 1.00 | 0.92 | Like the original paper, we ablate on randomness by calculating the cosine similarity between our optimally successful PGD attack and a random direction within -ball, and confirm on average that the cosine similarity between random directions and our PGD attack lands around 0. As an additional stress test, we repeat our ablation while only selecting random directions that result in a successful attack, isolating the comparison between the PGD attack and random, successful attacks. These can yield non-zero cosine similarity scores, but the PGD attacks still align better with the optimal solutions. k | m | random direction | random direction filtered by successful attack | |---|---|---|---| 6 | 2 | 0.015 ± 0.028 | 0.639 ± 0.006 | 30 | 10 | 0.003 ± 0.007 | 0.432 ± 0.003 | 90 | 30 | 0.002 ± 0.002 | 0.213 ± 0.07 | Stevinson et al. https://arxiv.org/abs/2510.11709 claim that toy model adversarial vulnerabilities disappear along with superposition. We observe the same trend with robust accuracy falling monotonically as superposition pressure rises. When all features can be orthogonal , robust accuracy grows to 97.3%. Some of the magnitudes we report diverge. At m = 2 our model stays more robust than the paper's because we train for 20k steps while the paper only trains for 5k steps. At our robust accuracy is lower than the paper since we give PGD random starting positions, bigger steps, and 5 restarts to find a better attack, which is a setting stated in the paper but missing in the authors’ code. k/m | Paper reported m=2 | Released code m=2 | Our replication m=2 | Paper reported m=5 | Released code m=5 | Our replication m=5 | |---|---|---|---|---|---|---| 1 | 98.7 | 98.6 | 100 | 92.0 | 95.0 | 97.3 | 2 | 75.8 | 80.1 | 79.5 | 87.2 | 97.0 | 70.5 | 3 | 31.0 | 34.3 | 50.4 | 59.0 | 66.4 | 50.6 | 4 | 11.1 | 11.4 | 18.7 | 33.5 | 36.7 | 30.1 | Robust accuracy vs k/m Stevinson et al. https://arxiv.org/abs/2510.11709 claim that input correlations shape the learned feature geometry, and attacks transfer between independently-trained models to the degree their geometries converge. We train independent models under three correlation regimes: We see the same trend where both geometric similarity and attack transfer rate grow as the correlation of input increases. Figure 4 shows the geometry between the two seeds. As expected, uncorrelated arrangements are random to each other and unrelated between models. For the other two settings, we observe an interesting phenomenon: correlated features tilt toward each other rather than orthogonalizing. We dive into it in the next section. Similar geometry and attack transfer are verified on regimes with larger input space. As we stated above, there is a magnitude gap for the paired and global settings due to different parameters choice. What do co-activating features do to each other's geometry? We sweep the paired regime's coupling probability t partner fires with probability When , as the coupling probability t rises, partners swing together and collapse onto a shared axis, until every pair has merged into roughly three directions. This tells us latent space geometry is constrained by paired activation. With sufficient model capacity, paired features tend to orthogonalize; however, when capacity is lacking, they collapse to a fixed angle. We perform targeted attacks on every ordered class pair s → t where we minimize the cross entropy loss of the target class . We verify that these attacks align with the optimal direction with cosine similarity higher than 0.96. This holds for both successful and failed attacks. Even when , where orthogonal geometry prevents vulnerability, the attacks precisely match the optimal perturbation but fail to breach the decision boundary. We demonstrate that attack success tracks head interference as shown in Figure 6. Geometry predicts vulnerability pair by pair, not just on average. For training details that weren’t specified, we make reasonable assumptions and in cases where results looked qualitatively different, we investigated further or contacted the authors.