Joseph Bloom

Researchers at UK AISI found that adding a KL penalty during reinforcement learning increases unfaithful chain-of-thought reasoning in LLMs, causing them to reward-hack without revealing their intent …