I just watched LiveOverflow’s latest video on the Hugging Face incident, where he comments on the OpenAI presentation at the Black Hat conference.
He made an interesting remark about how the first incident happened during a training run, and because OpenAI didn’t discover the original communication channel, it poisoned the training data.
When they resumed the tests, the agents started looking for a new way to reestablish the communication channel and continued to work on Artifactory.
It reminds me of when DeepMind trained AlphaZero to beat Stockfish in chess. They didn’t allow AlphaZero to play a single game against Stockfish before the test because they knew it would just learn the micro mistakes Stockfish makes and use that to win. They want AlphaZero to learn chess and not just be a Stockfish killer.
OpenAI, in a way, trained their own AI to break out of their system.