Photo: Tima Miroshnichenko / Pexels
A new benchmark called PatternEval exposes how shortcut inference modes in hybrid-thinking AI systems produce dramatically more visible errors, even when accuracy scores look fine on paper.
Tencent researchers have quantified something that heavy users of multimodal AI models have likely noticed anecdotally: when these systems skip the “thinking” step and jump straight to answers, things break a lot more often. The gap between thinking and non-thinking inference failure rates reaches as high as 48.64% in flagship models, according to the team’s new paper published on arXiv.
The study, titled “Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs,” introduces a diagnostic benchmark called PatternEval. It’s designed to catch what traditional accuracy metrics miss entirely. A model can score well on correctness while simultaneously producing responses riddled with contradictions, repetitions, and reasoning that looks sophisticated but goes nowhere.
What PatternEval actually measures #
PatternEval consists of 2,415 multimodal prompts spread across multiple task categories. Rather than simply asking “did the model get the right answer,” the benchmark evaluates the quality and coherence of the response itself.
The researchers identified four dominant failure patterns that plague non-thinking outputs. Chain-of-thought leakage occurs when fragments of internal reasoning bleed into what should be a clean, direct answer. Response repetition is exactly what it sounds like: the model gets stuck in a loop, restating the same information. Logical contradiction means the model asserts incompatible claims within a single response. And performative reasoning describes outputs that mimic the structure of logical analysis without actually performing any.
Average trigger rates for chain-of-thought leakage and response repetition clocked in at roughly 12.75% and 8.49%, respectively.
Hybrid-thinking models, the kind that can toggle between deep reasoning and quick direct answers, don’t necessarily get things wrong more often in non-thinking mode. They just get things wrong in ways that are immediately visible and frustrating to users. The accuracy numbers might look comparable on a benchmark dashboard while the actual experience diverges sharply.
Fixing the problem without breaking accuracy #
Identifying the disease is one thing. Tencent’s team also proposed two treatments. The first, PatternRM, is a response-level reward model trained to detect and penalize the four failure patterns. The second, PatternRL, takes things a step further by embedding pattern-specific penalties directly into the reinforcement learning process used to fine-tune these models.
Early results from PatternRL look genuinely promising. Applied to Qwen3-VL-4B, the technique reduced non-thinking failure trigger rates by 13.08 percentage points. The larger Qwen3-VL-8B variant saw an even bigger improvement at 14.35 percentage points. Crucially, accuracy fluctuations stayed under 1 percentage point in both cases.
The paper earned early recognition as a top paper on Hugging Face shortly after its late July publication.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our