{"slug": "i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part", "title": "I Tested Three Vision Models on Catalog Images: OCR Was the Easy Part", "summary": "A developer benchmarked Mistral OCR, DeepSeek V4 Flash Vision Exp, and Qwen3-VL-32B-Instruct on extracting structured data from 138-page image-only product catalogs, finding that OCR accuracy was less critical than preserving relationships between SKUs, prices, and shared-price blocks. The team selected Qwen3-VL-32B-Instruct for its cost-to-performance ratio and implemented a human-in-the-loop review workflow, also discovering that an evaluator schema mismatch caused a false 0% score for Qwen.", "body_md": "Converting a visual product catalog into structured, queryable data sounds simple on paper:\n\nOur hands-on benchmark with **Mistral OCR**, **DeepSeek V4 Flash Vision Exp**, and **Qwen3-VL-32B-Instruct** proved why that mental model breaks down in real life.\n\nThe hard problem was never character recognition. The real bottleneck was **relationship preservation**: linking the right 5-digit SKU to the correct variant, distinguishing campaign offers from list prices, and resolving multi-product shared-price blocks without human intervention.\n\nHere is what we learned building an extraction pipeline for 138-page PDF catalogs, why raw OCR metrics are misleading, and how we architected a human-in-the-loop workflow that prevents business incidents.\n\nOur catalog consisted of 138 image-only pages, each embedded as a full-page JPEG capped at 150 DPI.\n\nOur first architectural decision was straightforward: **rendering the PDF at 300+ DPI was a waste of compute.** Upscaling low-resolution assets doesn't create new signal; it just burns CPU cycles and expands token payloads.\n\nInstead, our pipeline extracts the raw JPEG byte stream directly and feeds the original images to model endpoints.\n\nThe real challenge lay in the layout patterns:\n\nIf a pipeline transcribes every digit correctly but pairs a campaign price with the wrong SKU, **it has generated corrupt catalog data.**\n\nWe evaluated three candidates across the same catalog fixtures:\n\n`mistral-ocr-latest`\n\n):All requests ran with deterministic settings (`temperature=0`\n\n). We captured raw payloads, latency, normalized JSON schemas, and per-page cost.\n\nWe tested all three candidates against a tricky baseline: **Page 94**, featuring three SKUs (`41367`\n\n, `41368`\n\n, `41369`\n\n) sharing a single campaign price (`C$339.00`\n\n) and list price (`C$570.00`\n\n).\n\n| Model | SKU Exact Match | Campaign Price Exact Match | Shared-Price Resolution | Cost / Page |\n|---|---|---|---|---|\nMistral OCR |\n0% (0/3) | 100% (1/1) | 0% (0/1) | ~$0.00350 (Est.) |\nQwen3-VL-32B |\n0% (0/3)* | 100% (1/1) | 0% (0/1) |\n$0.00024 (Actual) |\nDeepSeek V4 Flash |\n0% (0/3) | 100% (1/1) | 0% (0/1) | $0.00269 (Actual) |\n\n**Note on Qwen3-VL below: the 0% score was caused by an evaluator schema mismatch, not an inference failure.*\n\n**The takeaway:** Every model caught the campaign price. But isolating rotated product codes and mapping them back to a shared parent price failed across automated runs. DeepSeek hallucinated nearby numbers (`41347`\n\n, `41348`\n\n, `41349`\n\n), while Mistral misclassified the list price as a second campaign price.\n\nOur biggest engineering takeaway wasn't model accuracy—it was an **evaluator contract bug**.\n\nDuring manual verification, Qwen3-VL had successfully identified product codes and prices. However, our automated test harness reported a `0%`\n\nmatch rate because the scorer expected a JSON field named `code`\n\n, while Qwen returned `product_code`\n\n.\n\nA single key mismatch turned a working extraction into a synthetic 0% metric on the dashboard.\n\n**Rule of thumb:** In LLM/VLM evaluation pipelines, parser contracts are production code:\n\nBecause a wrong price or mismatched SKU causes actual financial errors downstream, we rejected a 100% zero-touch ingestion pipeline. Instead, we established operating confidence bands:\n\nWe selected **Qwen3-VL-32B-Instruct** for our pipeline due to its exceptional cost-to-performance ratio ($0.00024/page) and clean structured output, backing it with a human-in-the-loop review interface:\n\n`(List Price - Campaign Price) / List Price`\n\nmatches the printed badge discount within 1%).", "url": "https://wpnews.pro/news/i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part", "canonical_source": "https://dev.to/alfchee/i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part-4ilj", "published_at": "2026-09-01 00:10:45+00:00", "updated_at": "2026-09-01 00:52:40.240468+00:00", "lang": "en", "topics": ["computer-vision", "large-language-models", "ai-tools", "mlops"], "entities": ["Mistral OCR", "DeepSeek V4 Flash Vision Exp", "Qwen3-VL-32B-Instruct"], "alternates": {"html": "https://wpnews.pro/news/i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part", "markdown": "https://wpnews.pro/news/i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part.md", "text": "https://wpnews.pro/news/i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part.txt", "jsonld": "https://wpnews.pro/news/i-tested-three-vision-models-on-catalog-images-ocr-was-the-easy-part.jsonld"}}