PerceptionBench – Evaluating Atomic Visual Perception in Multimodal LLMs

Evaluating Atomic Visual Perception in Multimodal Large Language Models

Authors Kimi Team

Overview #

We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities—** discovered from how today's models fail, not defined in advance.** By attributing frontier-model failures across 40+ benchmarks to their earliest visual cause, we distill 10 perceptual capabilities and 3,000 verified questions, each answerable by looking, with no reasoning or outside knowledge required.

The result is a sharp diagnosis rather than one more score. No model we evaluate clears 60% accuracy, and models with nearly identical overall scores can exhibit very different perceptual strengths and weaknesses. More strikingly, a large share of correct answers fail to survive a repeated ask—evidence that current models often guess rather than perceive. PerceptionBench is built to expose exactly where perception breaks, and to drive progress toward multimodal AI that sees faithfully and consistently.

Using PerceptionBench to Compare Models #

Each source benchmark captures a narrow slice of perception errors, and these slices overlap only weakly (mean pairwise weighted Jaccard 0.20). No single benchmark—or small group of them—covers perception as a whole, which motivates a capability-centric benchmark that aggregates and rebalances these fragmented views.

The Dataset #

The dataset consists of 3,000 high-quality verified samples. The distribution aims to isolate atomic perceptual capabilities from confounding factors, and distinguishes itself through three core design principles:

Failure-Driven Taxonomy: Every category is discovered from real model failures, attributed to the earliest erroneous step across 40+ existing benchmarks.Ten Atomic Perceptual Categories: Visual Relation, Counting, Attribute, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.Perception, Not Reasoning: Samples are curated, decomposed, and difficulty-balanced so that difficulty stems from perception rather than reasoning.

Note on Quality: To make the benchmark a reliable gold standard, all samples underwent rigorous verification and difficulty-balancing, keeping only genuinely perceptual failures with a single verifiable answer.

Distribution of Tasks per Category

| Hallucination | 271 (9.03%) |
| OCR Error | 255 (8.50%) |

Conclusion #

PerceptionBench is a simple but challenging benchmark for evaluating atomic visual perception in frontier models. It measures what multimodal models actually see rather than what they infer, providing a faithful and fine-grained diagnosis of the perceptual capabilities of current and future multimodal models.