A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute does not yet influence the output. This gap between where information is readable and where it is used has been shown for attributes stated directly in the input. We ask whether it also hold
Putting a Transformer Inside Every Card Authorization: How Ramp Achieves Sub-30ms Inference