Alexa R. Tartaglini on X: "Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time (0% baseline). They also can't tell you why! Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵"
Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time (0% baseline). They also can't tell you why! Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵
Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time (0% baseline). They also can't tell you why! Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵
Images of simple icons like a heart or triangle steer models' coding strategy about as well as explicit instructions; we call this visual steering. Captions ("an image of a red heart"), noise images, & ImageNet photos barely move behavior. It also works on non-coding tasks.
Models' reasoning traces don’t identify this influence; they often say the image is irrelevant or don’t mention it at all. When directly asked, model interpretations barely track actual steering; e.g. a triangle steers Claude to 100% recursion, yet Claude says it means iteration
What about when models send visuals (rather than passively reading them)? We asked models to pick a visual symbol to steer another toward recursion. Senders pick human-intuitive symbols (spirals, arrows) instead of the ones that actually steer them!
Through multi-turn communication, senders find effective steering symbols, but they also drift from human convention. e.g., Claude starts with a spiral 🌀 and ends at a pink heart 💖 In a small human study, refined messages were less legible to people than first tries (67→44%).
We also had models communicate abstract concepts & short stories. With free-form drawing (PIL code), they're creative & effective (91% communication success on the first try). When restricted to icons, models show a generation-interpretation gap: first-try success drops to 48%.
Takeaways:
- Training can build strange visual associations that models don't verbalize, a problem for trace-based oversight of agents in multimodal envs
- Visual "neuralese" looks unlikely for now, but as models align meaning w/ each other, messages get harder for humans to read