#
What I Benchmarked
Current Frontier Visual Language Models (VLMs) excel at multi-modal reasoning and captioning, but how well do they execute step-by-step visual spatial verification?
To test this capability, I created VCIS-Bench (Visual Constraint & Instruction Simulation), a synthetic 7×7 grid puzzle benchmark. Each puzzle features a strict layout: an Agent (** A**), Key (** K**), Locked Door (** D**), Goal (** G**), and Walls (#). The agent must follow strict cardinal mechanics—it cannot pass through walls, and it must collect the key on the left side of a central partition before unlocking the door to reach the goal on the right.
I evaluated the models across 3 core tasks:
Task 1: Overall Plan Verification: Given an action sequence, determine whether the proposal isVALID orINVALID .
2.
Task 2: Spatial Grounding Verification: Evaluate spatial awareness by verifying object coordinates relative to labeled row (R1–R7 ) and column (C1–C7 ) headers.
3.
Task 3: Step-Level Error Diagnosis: Identify the exact 1-based step index ($1, 2, 3, \dots$) where an action plan first violates a rule (e.g., hitting a wall or passing through a door without a key).
#
Models Tested
I evaluated three distinct models across different parameter tiers to measure accuracy, latency, and cost-efficiency:
Claude Sonnet 5.5 (Anthropic): Benchmark state-of-the-art vision reasoning. #
Gemini 3.7 Flash (Google): Google’s fast multi-modal flash model. #
Gemma 4 26B A4B (Google): Open-weights vision-language architecture. These models provide a clear perspective on the performance gap between top-tier proprietary frontier models, lightweight API endpoints, and open-weights models.
#
Findings
Here is how the models performed across the benchmark suite:
Leaderboard Overview
| Task | Claude Sonnet 5.5 | Gemini 3.7 Flash | Gemma 4 26B A4B | | Task 1: Overall Plan Verification | 100.0% | 100.0% | 83.3% | | Task 2: Spatial Grounding Verification | 100.0% | 100.0% | 96.7% | | Task 3: Step-Level Error Diagnosis | 100.0% | 92.0% | 76.0% | | Overall Score | 1.00 | 0.97 | 0.85 |
(Data evaluated on October 11, 2026)
Key Takeaways
Step-Level Simulation is the Hardest Bottleneck: WhileClaude Sonnet 5.5 achieved a perfect100% accuracy across all tasks, performance dropped for smaller models on Task 3.Gemini 3.7 Flash reached 100% on binary verification tasks (Tasks 1 and 2), but dropped to92.0% when required to pinpoint the exact 1-based step index of a failure. #
Spatial Coordination vs. Open-Weight Execution:**Gemma 4 26B A4B performed strongly on Task 2 spatial coordinate checks (96.7% ), proving that open-weight models are proficient at vision grounding. However, its performance decreased to76.0%** on step-level execution (Task 3) due to tracking errors across multi-step state transitions. #
Cost-Efficiency Sweet Spot: Looking at theScore vs. Total Cost Pareto Chart ,Gemini 3.7 Flash offered a favorable efficiency tradeoff—delivering a near-perfect0.97 aggregate score at roughly half the total cost and execution latency of Claude Sonnet 5.5.
#
My Benchmark
Check out the task code and interactive leaderboard runs on Kaggle Benchmarks: