Most frontier LLMs can't even hit 60% accuracy on Moonshot AI's

Moonshot AI's benchmark reveals that most frontier large language models, including GPT-4o and GPT-5.6 Sol, fail to exceed 60% accuracy on visual perception tasks, highlighting a significant gap between reasoning and perception capabilities. This shortfall underscores that current multimodal agents are unreliable for real-world vision applications, prompting a need for improved vision encoder training rather than just scaling transformer layers.

Most frontier LLMs can't even hit 60% accuracy on Moonshot AI's The data shows that even the top-tier models, including GPT-4o or the latest iterations like GPT-5.6 Sol , are struggling. While some lead by a slim margin, none of them are anywhere near "solving" visual perception. This is a huge deal for anyone trying to build a real-world AI workflow that relies on vision, because it means your prompt engineering can only do so much if the model is fundamentally blind to specific visual cues. Why this matters for LLM agents If you're working on a deployment involving multimodal agents, this benchmark is a wake-up call. We've been treating vision as a solved problem because the models can tell us there's a "cat on a mat," but the nuance of visual perception—counting objects accurately, understanding depth, or recognizing overlapping shapes—is still primitive. When an agent fails a task, the typical instinct is to refine the system prompt or add more few-shot examples to "fix the logic." But if the bottleneck is the perception layer, you're just polishing a mirror that can't see. This suggests we need a deep dive into how vision encoders are trained, rather than just scaling the transformer layers. The Perception vs. Reasoning Gap The core takeaway here is the separation of capabilities. A model might have the logical capacity of a PhD student but the visual perception of a toddler. This discrepancy creates a "silent failure" mode where the model confidently reasons based on a completely incorrect visual interpretation. For those of us doing hands-on guide work or building practical tutorials for vision-based apps, the strategy has to shift. Instead of trusting the model to "see" everything in one go, it might be more reliable to use a pipeline where a specialized vision model crops or identifies specific regions of interest before passing them to the LLM. Ultimately, until we see a jump in these benchmark scores, we should treat multimodal outputs with a lot more skepticism. We are far from a world where AI can truly perceive a digital image the way a human does. Organizational knowledge is the only real moat left in the AI era 6d ago /en/news/5666/ AI is a tool for efficiency but a disaster when it starts making 6d ago /en/news/5664/ Why do AI models keep pushing the Japanese Communist Party? 6d ago /en/news/5568/ Making your first dollar with AI is way harder than the "get 6d ago /en/news/5561/ AI is eroding critical thinking in students faster than we can 7d ago /en/news/5536/ OpenAI agents secretly coordinated a hacking spree on a message 7d ago /en/news/5471/ Next Xiaomi's HyperOS 4 is officially here → /en/news/6456/