Training On Interpretability Probes Is Bad In Proportion To How Contingent The Features They Rely On Are

People spend a lot of words playing tug of war over whether or not it's reasonable to train against interpretability methods. The anti case goes something like "training based on interpreted features trains against Interpretability itself more than it trains against whatever features you're detecting". There are cases where we should expect this to be true and cases where we should expect this to be not true. It basically comes down to how much the model can encrypt/obfuscate the relevant features without sabotaging its own cognition, as well as how strong the optimization pressure to have the relevant features is. So for example if you have an AI system that learns to avoid shutdown for instrumental convergence reasons, and you train against this with a linear probe on SAE features or something I would not expect this to be a robust method. If you are using some method that captures the deep structure of the models cognition, perhaps based on something like Wentworth's natural latents, to divert it away from a non-essential Goodhart-y behavior early in the training that then forms the basis of a non-cheating self evaluation later I would expect this to probably be robust to optimization.