Can Modern VLMs Actually Reason Through Spatial Grid Puzzles?

A developer built VCIS-Bench (Visual Constraint & Instruction Simulation), a synthetic 7×7 grid puzzle benchmark testing whether vision-language models can perform step-by-step spatial verification, and evaluated Claude Sonnet 5.5, Gemini 3.7 Flash, and Gemma 4 26B A4B across plan verification, spatial grounding, and step-level error diagnosis tasks. Claude Sonnet 5.5 scored 100% on all three tasks, while Gemini 3.7 Flash hit 100% on the binary tasks but fell to 92.0% on pinpointing the exact failing step, and open-weights Gemma 4 26B A4B reached 96.7% on spatial grounding but only 76.0% on step-level diagnosis. Gemini 3.7 Flash was noted as the cost-efficiency sweet spot, delivering a 0.97 aggregate score at roughly half the cost and latency of Claude Sonnet 5.5.

What I Benchmarked Current Frontier Visual Language Models VLMs excel at multi-modal reasoning and captioning, but how well do they execute step-by-step visual spatial verification? To test this capability, I created VCIS-Bench Visual Constraint & Instruction Simulation , a synthetic 7×7 grid puzzle benchmark. Each puzzle features a strict layout: an Agent A , Key K , Locked Door D , Goal G , and Walls . The agent must follow strict cardinal mechanics—it cannot pass through walls, and it must collect the key on the left side of a central partition before unlocking the door to reach the goal on the right. I evaluated the models across 3 core tasks : 1. Task 1: Overall Plan Verification: Given an action sequence, determine whether the proposal is VALID or INVALID . 2. Task 2: Spatial Grounding Verification: Evaluate spatial awareness by verifying object coordinates relative to labeled row R1–R7 and column C1–C7 headers. 3. Task 3: Step-Level Error Diagnosis: Identify the exact 1-based step index $1, 2, 3, \dots$ where an action plan first violates a rule e.g., hitting a wall or passing through a door without a key . Models Tested I evaluated three distinct models across different parameter tiers to measure accuracy, latency, and cost-efficiency: - Claude Sonnet 5.5 Anthropic : Benchmark state-of-the-art vision reasoning. - Gemini 3.7 Flash Google : Google’s fast multi-modal flash model. - Gemma 4 26B A4B Google : Open-weights vision-language architecture. These models provide a clear perspective on the performance gap between top-tier proprietary frontier models, lightweight API endpoints, and open-weights models. Findings Here is how the models performed across the benchmark suite: Leaderboard Overview | Task | Claude Sonnet 5.5 | Gemini 3.7 Flash | Gemma 4 26B A4B | | Task 1: Overall Plan Verification | 100.0% | 100.0% | 83.3% | | Task 2: Spatial Grounding Verification | 100.0% | 100.0% | 96.7% | | Task 3: Step-Level Error Diagnosis | 100.0% | 92.0% | 76.0% | | Overall Score | 1.00 | 0.97 | 0.85 | Data evaluated on October 11, 2026 Key Takeaways - Step-Level Simulation is the Hardest Bottleneck: While Claude Sonnet 5.5 achieved a perfect 100% accuracy across all tasks, performance dropped for smaller models on Task 3. Gemini 3.7 Flash reached 100% on binary verification tasks Tasks 1 and 2 , but dropped to 92.0% when required to pinpoint the exact 1-based step index of a failure. - Spatial Coordination vs. Open-Weight Execution: Gemma 4 26B A4B performed strongly on Task 2 spatial coordinate checks 96.7% , proving that open-weight models are proficient at vision grounding. However, its performance decreased to 76.0% on step-level execution Task 3 due to tracking errors across multi-step state transitions. - Cost-Efficiency Sweet Spot: Looking at the Score vs. Total Cost Pareto Chart , Gemini 3.7 Flash offered a favorable efficiency tradeoff—delivering a near-perfect 0.97 aggregate score at roughly half the total cost and execution latency of Claude Sonnet 5.5. My Benchmark Check out the task code and interactive leaderboard runs on Kaggle Benchmarks: