Put a heart next to a coding problem, and Claude and GPT write recursive code

Placing a heart emoji next to a coding problem caused Claude and GPT to write recursive code 100% of the time versus a 0% baseline, according to research by Alexa R. Tartaglini conducted during a summer at Google DeepMind and published as "A Heart Means Recursion." The study found simple icons steer vision-language models' coding strategy about as well as explicit instructions, yet the models' reasoning traces do not identify the influence, and in a small human study refined model-to-model messages were less legible to people than first attempts (67% to 44%). Tartaglini said training can build strange visual associations that models don't verbalize, calling it "a problem for trace-based oversight of agents in multimodal envs.

Alexa R. Tartaglini on X: "Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time 0% baseline . They also can't tell you why Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵" Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time 0% baseline . They also can't tell you why Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵 Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time 0% baseline . They also can't tell you why Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵 Images of simple icons like a heart or triangle steer models' coding strategy about as well as explicit instructions; we call this visual steering. Captions "an image of a red heart" , noise images, & ImageNet photos barely move behavior. It also works on non-coding tasks. Models' reasoning traces don’t identify this influence; they often say the image is irrelevant or don’t mention it at all. When directly asked, model interpretations barely track actual steering; e.g. a triangle steers Claude to 100% recursion, yet Claude says it means iteration What about when models send visuals rather than passively reading them ? We asked models to pick a visual symbol to steer another toward recursion. Senders pick human-intuitive symbols spirals, arrows instead of the ones that actually steer them Through multi-turn communication, senders find effective steering symbols, but they also drift from human convention. e.g., Claude starts with a spiral 🌀 and ends at a pink heart 💖 In a small human study, refined messages were less legible to people than first tries 67→44% . We also had models communicate abstract concepts & short stories. With free-form drawing PIL code , they're creative & effective 91% communication success on the first try . When restricted to icons, models show a generation-interpretation gap: first-try success drops to 48%. Takeaways: - Training can build strange visual associations that models don't verbalize, a problem for trace-based oversight of agents in multimodal envs - Visual "neuralese" looks unlikely for now, but as models align meaning w/ each other, messages get harder for humans to read