Benchmarking Frontier AI on Curved Tire Sidewall OCR

This is a submission for the Kaggle Benchmarking Challenge

In computer vision pipelines for automotive manufacturing, fleet maintenance, and tire recycling, reading tire specifications is a notorious challenge.

Tire sidewalls feature embossed, black-on-black rubber text following the wheel's circular arc. When training custom object detection models (such as YOLO) to detect individual character boxes (/, 0-9, R), traditional post-processing relies on sorting bounding boxes by horizontal X-coordinates to reconstruct the tire code (e.g., 205/55R16).

However, this naive approach fails on real-world tires:

225/60R18 is read as 81R06/522). Can frontier general-purpose Vision-Language Models (VLMs) perform zero-shot OCR and structured extraction of ISO metric tire size specifications ([Width]/[Aspect]R[Rim]) directly from raw tire sidewall photos without any fine-tuning, and act as automated auditors for human-annotated YOLO datasets?

To measure this, I created a curated test suite of 51 tire sidewall images across 22 distinct tire dimensions (covering standard horizontal text, steep curved arcs, and rotated positions), backed by verified ground-truth labels.

The live Kaggle Benchmark evaluates 11 frontier models across 5 AI providers and open-weight architectures:

(Note: During task exploration, Gemini 3.7 Flash [46/51], Gemini 3 Flash Preview [45/51], and Claude Sonnet 5 [44/51] were also evaluated on the underlying task. DeepSeek-R1 was correctly rejected by the benchmark runner as a text-only reasoning model lacking a visual encoder, and Grok 4.6 encountered upstream proxy connection timeouts).

Each model was prompted with an isolated context per image:

"Inspect the tire sidewall in this image carefully. Locate the standard ISO metric tire size specification (format: WWW/AARDI, e.g. 205/55R16, 225/60R18, 235/65R17). Return ONLY the exact 9-character tire size code."

The benchmark recorded:

The table below reflects the official leaderboard from the live Kaggle Benchmark:

Note on Kaggle UI Score Display: On the Kaggle Benchmarks leaderboard UI, raw numeric scores appear with an automatic × 100% display badge (e.g. 46.00 renders as 4600.0% and 41.00 renders as 4100.0%, representing 46 and 41 correct items out of 51 total samples, or 90.20% and 80.39%). The table above provides the normalized, exact percentage rates.

When deploying vision models in production—such as scanning thousands of tire sidewalls daily across an automotive fleet or recycling facility—raw accuracy must be balanced against inference cost. Kaggle Benchmarks plotted the Score vs. Total Cost Pareto Frontier across all evaluated models:

In our YOLO dataset, human annotations on rotated or curved tire arcs were often scrambled by bounding-box sorting algorithms (e.g. producing 81R06/522 or 71R05/522).

All top-tier frontier VLMs effortlessly read these rotated, curved tires in the correct semantic human reading order, outputting 225/60R18 and 225/50R17 flawlessly. This demonstrates that multimodal LLMs perceive continuous text streams gestalt-style rather than through brittle geometric axis projections.

Google's Gemini 3.8 Flash and Anthropic's flagship Claude Opus 5.5 shared top honors with identical scores of 46 / 51 (90.20%). While Claude Opus 5.5 demonstrated immense precision in distinguishing faint embossed characters, Gemini 3.8 Flash delivered that same accuracy with lightning-fast inference times.

Right behind them, Google's lightweight Gemini 3.5 Flash-Lite scored 45 / 51 (88.24%), outperforming heavyweights like Gemini 2.5 Pro (44 / 51) and GPT-6.1 Sol (39 / 51)—proving that specialized vision distillation can outperform pure model parameter scale.

One of the most exciting results is Gemma 4 31B scoring 80.39% (41 / 51). It matched proprietary frontier flagships Claude Sonnet 5.5 and Claude Haiku 5.5, while beating OpenAI GPT-6.1 Sol (76.47%). For manufacturing plants and tire depots requiring local, on-premise execution without cloud API dependency, Gemma 4 31B offers production-grade zero-shot OCR capability.

Anthropic's Claude Haiku 5.5 tied Claude Sonnet 5.5 exactly at 41 / 51 (80.39%). Getting flagship-grade multimodal OCR accuracy at Haiku's speed and cost profile makes it an exceptional candidate for high-throughput batch auditing.

A fascinating divergence occurred with Qwen 3 Next 80B Thinking (3 / 51, 5.88%):

While reasoning and "thinking" architectures excel at multi-step mathematics and coding, they frequently failed this pure extraction benchmark because their outputs included internal reasoning preambles, chain-of-thought commentary, or Markdown wrappers (e.g.,

``` text\n205/55R16\n```

) rather than adhering strictly to the required bare 9-character string.

Across the ~10% of cases where top models failed, the errors were not random hallucinations:

235/45R18), models predicted 225/45R18 (confusing a scuffed embossed 3 with a 2). 245/50R20), models predicted 265/50R20 (confusing 4 with 6). 255/60R18), models misread the width and rim on a heavily worn sidewall. Notice that in almost every failure case, the Aspect Ratio and Construction ('R') remained 100% correct. The error was almost exclusively an off-by-ten millimeter width confusion caused by low-contrast black rubber embossments.

During benchmark preparation, we discovered a human labeling typo in our original dataset where an annotator typed 3 instead of R (255/60318). During inference, the models naturally output 255/60R18 based on semantic domain awareness of tire codes, proving that VLMs can serve as effective automated validation auditors for industrial labeling pipelines.

You can inspect the full benchmark, code, and live evaluation runs directly on Kaggle: