I Tested Three Vision Models on Catalog Images: OCR Was the Easy Part

A developer benchmarked Mistral OCR, DeepSeek V4 Flash Vision Exp, and Qwen3-VL-32B-Instruct on extracting structured data from 138-page image-only product catalogs, finding that OCR accuracy was less critical than preserving relationships between SKUs, prices, and shared-price blocks. The team selected Qwen3-VL-32B-Instruct for its cost-to-performance ratio and implemented a human-in-the-loop review workflow, also discovering that an evaluator schema mismatch caused a false 0% score for Qwen.

Converting a visual product catalog into structured, queryable data sounds simple on paper: Our hands-on benchmark with Mistral OCR , DeepSeek V4 Flash Vision Exp , and Qwen3-VL-32B-Instruct proved why that mental model breaks down in real life. The hard problem was never character recognition. The real bottleneck was relationship preservation : linking the right 5-digit SKU to the correct variant, distinguishing campaign offers from list prices, and resolving multi-product shared-price blocks without human intervention. Here is what we learned building an extraction pipeline for 138-page PDF catalogs, why raw OCR metrics are misleading, and how we architected a human-in-the-loop workflow that prevents business incidents. Our catalog consisted of 138 image-only pages, each embedded as a full-page JPEG capped at 150 DPI. Our first architectural decision was straightforward: rendering the PDF at 300+ DPI was a waste of compute. Upscaling low-resolution assets doesn't create new signal; it just burns CPU cycles and expands token payloads. Instead, our pipeline extracts the raw JPEG byte stream directly and feeds the original images to model endpoints. The real challenge lay in the layout patterns: If a pipeline transcribes every digit correctly but pairs a campaign price with the wrong SKU, it has generated corrupt catalog data. We evaluated three candidates across the same catalog fixtures: mistral-ocr-latest :All requests ran with deterministic settings temperature=0 . We captured raw payloads, latency, normalized JSON schemas, and per-page cost. We tested all three candidates against a tricky baseline: Page 94 , featuring three SKUs 41367 , 41368 , 41369 sharing a single campaign price C$339.00 and list price C$570.00 . | Model | SKU Exact Match | Campaign Price Exact Match | Shared-Price Resolution | Cost / Page | |---|---|---|---|---| Mistral OCR | 0% 0/3 | 100% 1/1 | 0% 0/1 | ~$0.00350 Est. | Qwen3-VL-32B | 0% 0/3 | 100% 1/1 | 0% 0/1 | $0.00024 Actual | DeepSeek V4 Flash | 0% 0/3 | 100% 1/1 | 0% 0/1 | $0.00269 Actual | Note on Qwen3-VL below: the 0% score was caused by an evaluator schema mismatch, not an inference failure. The takeaway: Every model caught the campaign price. But isolating rotated product codes and mapping them back to a shared parent price failed across automated runs. DeepSeek hallucinated nearby numbers 41347 , 41348 , 41349 , while Mistral misclassified the list price as a second campaign price. Our biggest engineering takeaway wasn't model accuracy—it was an evaluator contract bug . During manual verification, Qwen3-VL had successfully identified product codes and prices. However, our automated test harness reported a 0% match rate because the scorer expected a JSON field named code , while Qwen returned product code . A single key mismatch turned a working extraction into a synthetic 0% metric on the dashboard. Rule of thumb: In LLM/VLM evaluation pipelines, parser contracts are production code: Because a wrong price or mismatched SKU causes actual financial errors downstream, we rejected a 100% zero-touch ingestion pipeline. Instead, we established operating confidence bands: We selected Qwen3-VL-32B-Instruct for our pipeline due to its exceptional cost-to-performance ratio $0.00024/page and clean structured output, backing it with a human-in-the-loop review interface: List Price - Campaign Price / List Price matches the printed badge discount within 1% .