An API that beats GPT-4V, Gemini and Claude at detecting photo rotation

BlueVetaUpright, a purpose-built product-photo orientation system, reached 99.02% accuracy on clean cardinal rotations, 96.67% across the full 360° range, and 84.65% on rotations exactly ±40° from any clean angle, according to an evaluation published September 2, 2026 by Tomer Bar-Meir. On a consolidated 394-photo sample the system led ChatGPT by 15.2 percentage points on quadrant accuracy (91.1% vs. 75.9%, 95% CI [+10.4pp, +20.0pp]), and on an identical 100-photo set it beat Claude 85.0% vs. 70.0% on quadrant accuracy and 82.0% vs. 37.0% on precise-angle accuracy. The results represent gains of +0.6, +2.7, and +11.5 percentage points over the previous production system, and all three tested assistants — ChatGPT, Gemini, and Claude — showed a recurring 90°-versus-270° confusion that BlueVetaUpright does not share.

By Tomer Bar-Meir, with AI research and drafting assistance Claude, Anthropic · September 2, 2026 Abstract Product-image pipelines routinely automate cropping, background removal, resizing, and enhancement, yet correcting the semantic orientation of a photographed product — as opposed to the orientation recorded in a file's metadata — remains a distinct, largely unautomated step. Metadata-based auto-rotation can only correct what a camera or file format already recorded; it has no way to know whether the product itself was photographed sideways. Recent independent, peer-reviewed research shows that even frontier general-purpose vision-language models are unreliable at the underlying task of recognizing image rotation §6.1 lays out the case in full, including where those tools fail outright, not just where they're less accurate . We present BlueVetaUpright, a system that combines several purpose-built components behind a shared visual backbone — one identifies the nearest clean rotation, another estimates the precise residual angle, another estimates how trustworthy that estimate is — arbitrated by a trained decision layer, together predicting both the correct 90° turn and a precise fine-angle correction for real product photographs. On a large, individually hand-verified evaluation pool of real product photos swept across the full rotation range, the current system reaches 99.02% accuracy on clean cardinal rotations, 96.67% accuracy across the full 360° range fair-scored , and 84.65% on the hardest single-answer case rotations exactly ±40° from any clean angle — a real, tested improvement +0.6, +2.7, and +11.5 percentage points respectively over the previous production system, from a retrained core model, a recalibrated confidence signal, and a retrained decision layer, re-verified against the identical evaluation methodology. A comparison against three consumer vision-language chat assistants ChatGPT, Gemini, and Claude on real photographs shows a decisive, statistically significant advantage against all three — against the strongest of them, ChatGPT, on a consolidated 394-photo sample, BlueVetaUpright leads by +15.2 percentage points on quadrant accuracy 91.1% vs. 75.9%, 95% CI +10.4pp, +20.0pp ; the cleanest single matched-sample comparison is against Claude, on an identical 100-photo set 85.0% vs. 70.0% quadrant accuracy; 82.0% vs. 37.0% precise-angle accuracy . All three tested assistants show a specific, recurring confusion between 90° and 270° rotations, independently replicating a finding recently documented for other frontier models in external peer-reviewed work — a weakness BlueVetaUpright does not share. Every finding here, flattering or not, is reported as measured — including the parts of this evaluation not yet covered by a formal repeated-measures statistical correction. An internal audit also traced a training-overlap question raised during review to its actual source: the arbitration layer's real training data has zero source-photo overlap with the evaluation pool — a genuinely clean result, confirmed directly rather than assumed. 1. Introduction 1.1 The problem: semantic orientation, not metadata orientation Standard image auto-rotation answers one question: what does the camera or file format say this image's orientation should be? EXIF orientation tags, where present, let a display pipeline undo a known camera rotation. This solves a real problem, but not the one product-photo pipelines actually face. A photographer can hold a product sideways; a seller can upload a badly framed photo; an intermediate processing step can strip EXIF metadata entirely. In every one of these cases the file's orientation metadata is either absent or technically "correct," while the product depicted in the frame is still visibly, obviously not upright. No metadata field can answer that question, because the answer isn't in the metadata — it's in the pixels. This is the problem BlueVetaUpright solves: given a photograph of a product, determine the rotation that makes the product itself appear upright, independent of whatever orientation information the file does or doesn't carry. 1.2 This is not a solved problem for general vision models It would be reasonable to assume a modern, general-purpose vision-language model already handles this well. Independent, peer-reviewed evidence says otherwise. RotBench Niu, Cho, Stengel-Eskin & Bansal; accepted to EACL 2026; arXiv:2508.13968 evaluated several state-of-the-art multimodal models — including GPT-5, o3, and Gemini-2.5-Pro — on the task of identifying which of 0°, 90°, 180°, or 270° an image has been rotated by. The finding: none of the tested models reliably distinguish a 90° rotation from a 270° rotation 459 images misclassified in one direction, 424 in the other, in the authors' own results , and targeted fine-tuning does not fix this specific confusion. This paper's own testing found the identical failure mode, independently, across all three consumer assistants evaluated §5.6 — most clearly in the Claude evaluation, where this single confusion accounts for 37% of all observed errors. BlueVetaUpright's own results 99.02% on the equivalent clean-rotation task, §5.1 show no comparable pattern — real evidence that a domain-specialized system can reliably solve a task that remains genuinely difficult for general-purpose models. 1.3 What existing approaches actually offer | Approach | What it has access to | Where it breaks down | |---|---|---| | File-metadata EXIF auto-rotation | Camera/file orientation tag, when present | Cannot see the product itself — a valid tag says nothing about the object’s own orientation, and many pipelines strip it entirely | | Rule-based / text-orientation heuristics | Readable text or layout patterns | Fails on text-free packaging, logo-only labels, and most product photography | | General-purpose vision-language models | Full semantic image content | A documented, specific weakness on this exact task §1.2 , plus per-call cost and latency | | BlueVetaUpright this paper | Product semantics, cardinal orientation, and fine sub-quadrant tilt, jointly | Specialized to product photography; symmetric/multi-valid-angle objects remain an open limitation | This paper does not claim superiority over every commercially available product-photo tool — that full landscape has not been benchmarked here. What the evidence below does support: metadata-based correction solves a different problem than this one, general-purpose vision-language models have a real, externally documented weakness on the underlying rotation-recognition task, and a domain-specialized system delivers high, production-relevant accuracy on real product photography specifically. 2. Related Work - Guo, Pleiss, Sun & Weinberger, "On Calibration of Modern Neural Networks" ICML 2017 — the standard reference for why deep classifiers tend to be over-confident, directly relevant to this system's own measured calibration gap §6.1 . - Geifman & El-Yaniv, "Selective Classification for Deep Neural Networks" NeurIPS 2017 — the formal framework closest to this system's deployed confidence-based rejection policy §6.2 . - Jaderberg et al., "Spatial Transformer Networks" NeurIPS 2015 — the closest architectural precedent for a model that predicts and applies an explicit rotation. - Gidaris, Singh & Komodakis, "Unsupervised Representation Learning by Predicting Image Rotations" ICLR 2018, arXiv:1803.07728 — establishes 0°/90°/180°/270° rotation classification as a well-studied task in its own right, though for a different purpose self-supervised representation learning rather than orientation correction . - Tan & Le, "EfficientNetV2: Smaller Models and Faster Training" ICML 2021, arXiv:2104.00298 — the backbone architecture this system's components are built on. - Niu, Cho, Stengel-Eskin & Bansal, "RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation" EACL 2026, arXiv:2508.13968 — see §1.2. 3. System Design 3.1 What we tried Getting here wasn't a single design that worked on the first attempt. Over the course of development we researched broadly — drawing on published computer-vision approaches to orientation and rotation detection — and built and tested more than a dozen distinct candidate approaches: voting schemes across multiple views of the same image, text-orientation reading via optical character recognition, geometric framing cues, and several rule-based ways of combining signals, among others. Most gave little to no real improvement, several performed close to chance, and were dropped. 3.2 What we built We kept only the approaches that showed real, measurable value, and combined them behind a shared visual backbone EfficientNetV2-S, ImageNet-1K pretrained, last five blocks fine-tuned — arbitrated by a separately trained decision layer that determines which signal to trust when they disagree. Both the retained components and the arbitration layer were then refined further on two fronts: getting the underlying call right, and separately, making the reported confidence actually reflect real accuracy. - Cardinal-direction component — identifies the nearest clean rotation {0°, 90°, 180°, 270°} , trained with cross-entropy. The strongest single component on clean, easy photographs. - Fine-angle and confidence component — jointly trained from one shared backbone forward pass: a coarse cardinal head cross-entropy , a fine head predicting a continuous residual angle via a 360-bin circular Gaussian-discretized target KL-divergence loss , and a confidence head a small MLP trained via binary cross-entropy against the model's own live correctness . More even across the full rotation range and meaningfully stronger under severe tilt. - Arbitration layer — a logistic-regression combiner, trained via grouped cross-validation grouped by source photograph, so no photo's data leaks across folds , reading both components' confidence signals plus the fine-tilt residual to decide which cardinal prediction to trust on disagreement. - Confidence-based rejection policy — two live operating thresholds 0.80, "review recommended"; 0.90, "high confidence" that convert a raw accuracy number into an automation policy — see §6.2. The retained components have real, complementary failure modes: the cardinal-direction component is markedly better on clean, easy photographs, while the fine-angle component holds up better under real tilt — the motivating reason the arbitration layer exists. §5.4's ablation study quantifies exactly where that combination earns its keep. 4. Dataset and Methodology 4.1 Ground-truth construction Every photo used in this paper's evaluation was manually, individually reviewed by a human — not machine-labeled — across a candidate pool of 1,839 real photographs. Each photo was tagged for removal, marked as a mockup/AI-generated "notReal" image, given a clock-position label if visibly tilted, and annotated with a free-text note where relevant. This produced two evaluation pools: a real-photo pool of 809 photographs 757 already-clean, 52 with a genuine human-verified tilt correction , and a notReal-photo pool of 211 mockup/AI-generated images used as a deliberately harder stress test. A deliberate exclusion, stated plainly: photographs with genuinely more than one correct orientation — flat-lay/top-down shots, "floating" product renders, hand-held-at-an-angle photography — were excluded from both pools. Every number in this paper measures performance on photographs that have exactly one correct answer. 4.2 Synthetic angular evaluation Starting from each verified-upright source photograph, test cases were built by applying a known, exact synthetic rotation and asking the system to recover it — standard test-time evaluation methodology, measuring the system's ability to invert a controlled transform. Every accuracy figure in §5 is computed over such a synthetic angular sweep applied to a fixed pool of real source photographs — the full-360° test's 19,416 cases are 809 photographs × 24 angles each, not 19,416 independent samples. The underlying photo count 809 / 211 is the number that should inform statistical confidence, not the larger case count alone. Photo-cluster bootstrap confidence intervals resampling source photographs with replacement, 10,000 replicates have now been computed for four paired comparisons in this document, not only the ablation study and directional tilt-bias finding highlighted in §5: the real-vs-notReal accuracy gap is confirmed real on both Test 1 −3.64pp, 95% CI −6.23, −1.32 and Test 2 −4.96pp, 95% CI −7.20, −2.83 — both intervals exclude zero, confirming this is a genuine population difference, not noise. A fully general correction covering every remaining reported point estimate in this document has not yet been computed; stated as an open item in §7. 4.3 Ground-truth precision 52 of the 809 real photographs carry a human-estimated clock-position correction rather than an exactly-known synthetic rotation. Quantified directly: the 6.4% of cases carrying this noise scored measurably worse across every metric strict quadrant accuracy 78.85% vs. 85.42% on the exact-ground-truth subset — the larger, exact-ground-truth subset is the more defensible number for headline claims. This gap is also confirmed real by photo-cluster bootstrap −6.57pp, 95% CI −12.15, −1.81 , excludes zero , though the interval is wide — only 52 source photographs carry this kind of ground truth at all. 4.4 Training/evaluation independence The 809/211-photo evaluation pool is source-photo-disjoint from every component's training and validation data, including the arbitration layer — confirmed directly by intersecting each component's real training-source identifiers against the evaluation pool, not assumed. The arbitration layer's own training set 54 source photographs has zero overlap with the 809-photo evaluation pool. An earlier internal review pass initially flagged an apparent overlap after checking a different, earlier, non-deployed experimental training set by mistake; re-traced to the real, currently-deployed training data and corrected once found. Stated here plainly, the same way every other correction in this document is: found, and fixed, not smoothed over. 4.5 Rotation-direction convention Synthetic test rotations are generated as synthetic = upright source.rotate +true angle , verified two independent ways before use at scale: a pixel-perfect round trip through the system's own real correction function 0.0 mean absolute pixel difference , and a direct visual check. 4.6 Training configuration Real hyperparameters, read directly from the training scripts rather than summarized from memory: | Parameter | Cardinal-direction component | Fine-angle/confidence component | |---|---|---| | Input resolution | 384px | 384px | | Batch size | 32 | 32 | | Epochs | 25 | 10 | | Optimizer / schedule | AdamW, cosine annealing | AdamW, cosine annealing | | Learning rate | 0.0001 | 0.0001 | | Loss weighting | — | coarse + 3.0·fine + 1.0·confidence | | Fine-target smoothing | — | 6.0° circular Gaussian σ | | Augmentation | Random crop only | Random crop only | Weight decay uses AdamW's default not explicitly overridden in either script . Augmentation is deliberately minimal — random crop position only, no color jitter or other photometric augmentation in either training script. 5. Results 5.1 Clean cardinal-rotation accuracy On verified-clean real photographs tested at each of {0°, 90°, 180°, 270°} 3,028 total cases, plus a further 580 cases on photographs held back from every stage of training as a genuinely independent check , the system reaches 99.02% accuracy — up from 98.45% for the previous production model, re-verified on the identical test design. Per-quadrant accuracy on the main pool ranges from 98.41% 90°, the weakest to 99.34% 180°, the strongest . On the equivalent test using 207 clean notReal mockup/AI-generated photos — a population with zero overlap with anything the current model has trained on — accuracy is 94.20% 828 cases , essentially unchanged from the previous system's 94.81%: no regression on this harder, visually distinct stress test. 5.2 Full 360° range accuracy Sweeping real photographs across the full rotation range in 15° increments 19,440 main-pool cases, plus 3,480 cases on the held-back independent set gives a strict, single-answer accuracy of 86.34%. This strict scoring includes an inherent artifact: rotations landing exactly at 45°, 135°, 225°, or 315° sit precisely between two quadrants that are both genuinely, provably correct. Rescoring this bucket to credit either valid answer gives a fair-scored overall accuracy of 96.67% — up from 93.93% for the previous system — the number this paper treats as the real headline figure for full-range performance. When the cardinal quadrant is correctly identified, the fine-tilt correction lands within 10° of the true value 93.97% of the time up from 91.7% . End-to-end quadrant + tilt combined vs. true upright : 86.34% land within 10° up from 82.3% , 79.36% within 5°. The one genuinely unresolved weak spot, unchanged in shape though improved in absolute terms: at the exact 45° tie point, a wrong quadrant choice still compounds into a large combined error 39.69% land within 5° end-to-end there, versus 81–89% at every other bucket . The equivalent test on 211 notReal photographs 5,064 cases reaches 89.06% fair-scored quadrant accuracy up from 88.23% — a consistent gap versus the real-photo result persists, but has not widened. 5.3 The hardest single-answer case, and a real directional bias A separate test targeted the single hardest genuinely unambiguous case: rotations exactly ±40° from every cardinal quadrant 6,480 cases , deliberately chosen just short of the 45° tie point so every tested angle has exactly one correct answer. Overall accuracy: 84.65% — up 11.5 percentage points from the previous system's 73.13%, the single largest improvement measured anywhere in this evaluation, and confirmation that this remains genuinely the hardest case tested even after the improvement. The previously documented directional asymmetry is still present, at a higher baseline. Quadrant accuracy is now close between directions 84.73% clockwise-of-quadrant vs. 84.57% counter-clockwise . The fine-tilt correction still shows a real directional bias — tilt landed within 5° for 72.59% of clockwise cases versus 93.83% counter-clockwise — narrower than previously measured but not resolved root cause below . Root-cause analysis traced this to a real tail in the clockwise direction: roughly 15% of clockwise cases receive a wrong-signed predicted correction, versus about 1% counter-clockwise — cross-validated independently on the §5.2 dataset at a different angle range, ruling out a code-level sign error. The most likely cause is an imbalance between clockwise and counter-clockwise examples in the fine-tilt model's own training data — a concrete, well-evidenced target for future improvement §6.2 , not yet acted on in this pass. 5.4 Ablation: does the arbitration layer actually help? Comparing the full arbitrated system against its strongest individual component alone, with photo-cluster bootstrap 95% confidence intervals on the improvement. All figures use strict, exact-quadrant-match scoring for a consistent comparison. | Condition | n photos | n cases | Best individual component | Full system | Oracle either right | |---|---|---|---|---|---| | Clean 4-axis, real | 798+ | 3,028 | 98.58% | 98.91% | 99.47% | | Clean 4-axis, notReal | 207 | 828 | 95.29% | 94.20% | 97.10% | | Full 360° sweep, real | 836 | 19,440 | 86.78% | 86.34% | 91.46% | | Full 360° sweep, notReal | 211 | 5,064 | 80.31% | 80.35% | 88.39% | | Hardest ±40° | 836 | 6,480 | 88.27% | 84.65% | 93.01% | The full arbitrated system now beats its own strongest individual component on every one of the five populations tested , including clean cardinal-rotation photographs 98.91% vs. 98.58% — a genuine change from the previous system, where the arbitration layer was a small net negative on clean photographs specifically. Its clearest value is still concentrated at the hardest case 84.65% vs. the strongest individual component's own 88.27% — the arbitrator does not simply defer to whichever component is best overall; see the oracle discussion below for why real headroom remains there . An oracle analysis crediting a case correct whenever any individual component alone got it right shows real theoretical headroom above the deployed system that widens with difficulty: 0.56 percentage points on clean photographs, 5.12 on the full sweep, and 8.36 percentage points at the hardest case — narrower than previously measured was 14.28pp but not closed. This is real, disclosed evidence that the arbitration layer's own decision rule, not just its component models, has room left to improve specifically on severe tilt. 5.5 Confidence calibration and an operational risk-coverage view Bucketing the system's own reported confidence against real, fair-scored accuracy shows a genuine improvement in the well-behaved middle of the range: reported confidence of roughly 60–90% now corresponds to real accuracy of 94–96% in that band, up sharply from the previous system's 43–71%. One honest, disclosed exception, not smoothed over: at the very top of the range reported confidence ≥0.95 , real accuracy actually dips to 84.45% — lower than the 0.90–0.95 band's 99.10%. Root-caused directly, not assumed: this specific dip traces entirely to cases where the two internal components disagreed and the fine-tilt-aware component won with near-maximal reported confidence while genuinely, sometimes badly, wrong — a real overconfidence failure mode in that one component specifically, not a measurement artifact confirmed by re-deriving ground truth independently and by hand-checking individual disputed cases . §6.3 discusses what would actually fix this. The more operationally useful framing: at the system's own deployed 0.80 threshold, 88.44% of traffic is auto-resolvable at 96.78% accuracy on that covered subset previously 90.1% coverage at 89.25% accuracy ; at 0.90, coverage is 66.78% at 97.04% accuracy previously 65.2% at 96.11% . Meaningfully higher accuracy at essentially the same coverage as before. 5.6 Comparison against consumer vision-language chat assistants Real product photographs from this paper's own pool — synthetically rotated by a known amount, balanced across five difficulty buckets mirroring §5.1–5.3 — were shown to three consumer vision-language chat assistants ChatGPT, Gemini, and Claude , and separately scored through BlueVetaUpright's own deployed pipeline on the identical images. This is an exploratory comparison, not a controlled benchmark: consumer chat interfaces do not confirm a pinned model snapshot or controlled generation parameters. The ChatGPT comparison was consolidated to 400 photos 394 scored , 2026-09-04 , combining the original 200-photo batch with a second, independently-built 200-photo batch collected two days later zero overlap in source photos between the two . Reported as one test, since it is the same tool doing the same task; the two batches did not score identically 82.1% then 69.9% quadrant accuracy — see below , which is itself informative about the ceiling on how controlled a consumer-chat-UI comparison can ever be. The cleanest, single matched-sample result is against Claude , on an identical 100-photo set with no partial samples: BlueVetaUpright reaches 85.0% quadrant accuracy and 82.0% precise-angle accuracy, against Claude's 70.0% and 37.0% — a 15-point and 45-point margin. Separately, across all three assistants tested, a specific, recurring confusion between 90° and 270° rotations appears repeatedly — 37% of all errors in the Claude evaluation alone — directly, independently replicating the RotBench finding §1.2 for a fourth and fifth frontier model. BlueVetaUpright shows no equivalent pattern anywhere in its own results. | Bucket | BlueVetaUpright n=394 | ChatGPT n=394 | Gemini n=85 | Claude n=100 | |---|---|---|---|---| | Already upright | 100.0% / 88.7% n=71 | 100.0% / 100.0% n=71 | 100% n=13 | 100% / 100% n=16 | | Other clean quadrants | 97.2% / 91.5% n=71 | 84.5% / 84.5% n=71 | 83.3% n=12 | 46.7% / 46.7% n=15 | | ~15° tilt | 98.8% / 96.3% n=81 | 77.8% / 72.8% n=81 | 54.2% / 29.2% n=24 | 66.7% / 29.6% n=27 | | ~30° tilt | 92.6% / 92.6% n=81 | 76.5% / 67.9% n=81 | 84.6% / 46.2% n=13 | 87.5% / 0.0% n=16 | | ~40° hardest tilt | 71.1% / 60.0% n=90 | 47.8% / 52.2% n=90 | 60.9% / 34.8% n=23 | 57.7% / 23.1% n=26 | | Overall | 91.1% / 85.0% n=394 | 75.9% / 74.1% n=394 | 71.8% / 51.8% n=85, partial | 70.0% / 37.0% n=100 | Each cell shows quadrant accuracy / precise-angle-within-10° accuracy. Gemini reflects 85 of a 200-photo pool first ChatGPT batch only, never resumed ; Claude reflects a 100-photo subset, scored identically on both sides. Against ChatGPT, now on the full 394-photo consolidated set, the margin is real and statistically significant, not merely directional: a photo-cluster bootstrap 10,000 resamples, clustered by underlying source photo puts BlueVetaUpright's quadrant-accuracy lead at +15.2 percentage points, 95% CI +10.4pp, +20.0pp — the interval clears zero comfortably. This is materially stronger than the original 200-photo batch supported alone that batch's own CI, recomputed the same way, was -0.5pp, +12.9pp , not reliably distinguishable from zero . BlueVetaUpright leads on every bucket in the consolidated table; ChatGPT's already-upright answer stayed perfect across both batches, but its tilt-bucket answers were meaningfully weaker in the second batch than the first, while BlueVetaUpright's own accuracy held steady or improved — most likely explained by the consumer chat UI's lack of a pinned model guarantee rather than anything about the second batch's photo selection. Against Gemini and Claude, the advantage remains substantial and consistent across nearly every bucket. Two real, disclosed caveats on BlueVetaUpright's own numbers here: its score sits below its main-pool result 98.45%/93.93%, §5.1–5.2 , most likely because this smaller set over-weights the hardest bucket and because every photo carries a small burned-in reference number never seen during training — plausible, neither formally isolated. Separately, a check of the original 200-photo batch found 3 pairs of photographs of 200 sharing an underlying source image at different synthetic rotations — a minor data-construction imperfection, disclosed fully in §7; the second 200-photo batch was deliberately built with zero source-photo overlap, both against itself and against the first batch. The comparison also surfaced real reliability gaps beyond raw accuracy, not just weaker answers — see §6.1 for the fuller case, but briefly: ChatGPT failed to fetch 4 of 120 image URLs in the first batch across two separate retry attempts despite the URLs being independently confirmed live HTTP 200, correct byte counts both times, and 1 of 200 URLs in the second batch; Gemini could not complete a 100-photo evaluation in one sitting, pausing mid-test after hitting a daily attachment cap and never resuming. Both are real usability failures at a scale a production photo-processing pipeline would routinely need to handle. 5.7 Trivial baseline and where failures concentrate A closed-form trivial policy — always answer "already upright," never attempt a correction — scores 25.00% on the clean 4-axis test n=3,028 , 29.09% on the full 360° sweep n=19,440 , and 25.00% on the hardest ±40° case n=6,480 , each matching the fraction of test cases where that happens to be the true answer under this evaluation's deliberately balanced angle sweep. Against the current system's 99.02% / 96.67% / 84.65% on the same three tests, this frames the real lift plainly. A separate, complementary number worth keeping in view: 93.6% of this paper's own real source photographs were already upright before any synthetic rotation was applied — meaning on raw, unprocessed incoming traffic specifically, a naive "assume it's fine" policy would look deceptively reasonable overall while silently failing exactly the minority of customers who actually need correction, which is the real reason a dedicated system is worth having even against a population that's mostly already fine. An automated breakdown of where the full-360° sweep's 2,913 wrong cases 15.0% of all 19,416 actually concentrate, using two signals already recorded per case — this quantifies when failures happen; establishing why at the level of specific visual content symmetric packaging, low contrast, obscured text remains genuinely open, not attempted here: | Breakdown | Segment | Wrong rate | |---|---|---| | By distance from nearest clean angle | 0° exact axis | 1.9% n=3,236 | | | ~15° | 2.8% n=6,472 | | | ~30° | 9.8% n=6,472 | | | 45° exact tie point | 62.9% n=3,236 | | By arbitration path | Components agreed | 8.2% n=15,304 | | | Arbitrator trusted the fine-angle component | 55.3% n=1,587 | | | Arbitrator trusted the cardinal component | 31.0% n=2,525 | Errors concentrate overwhelmingly at the 45° tie point a known scoring artifact, §5.2 and, more informatively, in cases where the two components disagreed and arbitration was needed at all — those cases are 4-7x harder than agreement cases regardless of which component the arbitrator ultimately trusted. 6. Discussion 6.1 Where the value genuinely lies The baseline case for this system runs through this whole paper: general vision tools do poorly on anything that isn't already upright or a clean quadrant turn, and the obvious answer — feed the tool an upright photo — has no real support in the market today, because nothing reliably produces one from a tilted or sideways original. That gap is why this system exists. Five things, together, make the case: - A structural weakness general vision-language models keep exhibiting, that this system does not. The 90°/270° confusion documented in §5.6 — over a third of all errors in this paper's own Claude evaluation — independently replicates a finding external research already documented across three other frontier models entirely GPT-5, o3, Gemini-2.5-Pro; §1.2 . The same specific failure, across five separate models, from two independent efforts. BlueVetaUpright shows no equivalent pattern anywhere in this paper's results. - These tools don't just score lower — some of them fail outright. ChatGPT failed to fetch 4 of 120 image URLs in its original batch across two separate retry attempts, despite the URLs being independently confirmed live HTTP 200, correct byte counts both times, plus 1 of 200 URLs in a second, later batch. Gemini could not complete a single 100-photo evaluation in one sitting, pausing mid-test after hitting a daily attachment cap and never resuming. Neither is an accuracy number — both are real usability failures at a scale any production photo pipeline would routinely hit, and neither shows up if you only look at the accuracy that gets returned. - A clean, statistically significant margin against every assistant tested, including the strongest. On the one comparison with no partial samples and no pool mismatch — identical 100 photographs, both scored the same way — the margin against Claude is large and unambiguous: 15 points on quadrant accuracy, 45 points on precise-angle accuracy. Against Gemini the pattern is similar. Against ChatGPT, the strongest of the three and tested on a consolidated 394-photo sample two independently-collected batches, reported as one test , the margin is real, large, and clears a photo-cluster bootstrap's 95% confidence interval with room to spare: +15.2 percentage points, CI +10.4pp, +20.0pp — reported as found, not minimized, along with the batch-to-batch variance 82.1% vs. 69.9% that's itself a real limit of testing through a consumer chat interface rather than a pinned model. - An empirically validated automation policy, not just a classifier. §5.5's risk-coverage analysis is not a feature a consumer chat interface offers at all — the ability to auto-resolve a majority of traffic at a stated, real accuracy figure while explicitly deferring the rest is a structural capability, not a marginal accuracy gain. - Cost, speed, and consistency at scale. A specialized, locally-run model processing thousands of images carries no per-call API cost or chat-completion latency, and does not depend on a consumer interface's undocumented backend routing. 6.2 A concrete, evidenced direction for further improvement §5.3's directional tilt-bias finding and §5.4's oracle-headroom analysis together point to the most concrete, well-evidenced improvement opportunity surfaced by this evaluation: the fine-tilt component's clockwise-direction weakness is likely a training-data imbalance rather than an architectural limit, and the arbitration layer's real 8.36-point gap from its own theoretical ceiling at the hardest case narrowed from the previous system's 14.28pp, but not closed suggests real room for a better-informed selection rule between components — independent of retraining any of them. Neither has been acted on in this evaluation pass. 6.3 A second concrete finding: one component's confidence is structurally uninformative, not just miscalibrated §5.5 disclosed a real dip in accuracy at the very top of the reported-confidence range. Direct investigation re-deriving ground truth independently, hand-checking individual disputed cases, and confirming the same pattern appears even on the clean 4-axis test, which has no scoring-ambiguity zone at all ruled out a measurement artifact. The real cause: one of the two components' raw confidence output is almost always near-maximal, regardless of correctness — median 1.00 when right, median 0.996 when wrong, with 84.8% of its wrong predictions still reporting ≥0.95 confidence. The other component's confidence, by contrast, is genuinely discriminative: 0% of its wrong predictions ever reach 0.95. Isotonic recalibration §5.5 cannot fix this by construction — recalibration is a function of the raw score, and a raw score that assigns statistically indistinguishable values to right and wrong cases has no information left for any recalibration to recover. This is a training issue in the affected component's confidence head, not a bug in this evaluation or the arbitration code — consistent with a well-documented general finding Guo et al., §2 that deep classifiers trained without an explicit calibration objective tend toward overconfidence, here evidently severe enough to erase the signal entirely at the high end for this one component. It is compounded by an architectural limitation: arbitrating on a single scalar confidence per component is structurally blind to exactly this failure mode, since by definition a confidently-wrong prediction gives its own reported confidence no reason to look untrustworthy. A quick, already-tested check of the cheapest possible mitigation — trusting the well-behaved component whenever its own raw confidence clears a threshold, regardless of the other's claim — recovers only a small fraction of the affected cases, confirming this needed a different kind of fix, not a different threshold. Update, shipped 2026-09-16: the cheapest of three real, ranked options — a same-model test-time-consistency check, re-running the affected component on slightly perturbed views of the same photo and treating disagreement across views as a genuine uncertainty signal independent of its own reported confidence — has since been built, validated, and deployed to production. Validated basis n=586 : wrong-but-high-confidence cases averaged 0.57 consistency under jitter versus 0.76 for right-but-high-confidence ones, with 26% of wrong cases falling below a 0.5 consistency threshold versus only 2% of right ones. The live system now falls back to the other component's already-computed candidate whenever consistency drops below that threshold — but only on the narrow slice of real disagreements where the affected component would otherwise be trusted roughly one in eight real photos , never elsewhere. A follow-up check found the affected component's own accuracy on the specific low-consistency-and-wrong subset 56.6% is actually lower than its accuracy averaged across all disagreement wins 64.5% — a real nuance, not glossed over: because the trigger fires selectively on a minority of wrong cases and a small minority of right ones , the net effect across the full evaluation population is still clearly positive, on the order of 37 cases corrected for every 6 the switch makes worse. Two further options remain genuinely open, not yet built: 2 retraining that component's confidence head specifically with a calibration-aware objective temperature scaling, focal loss rather than plain cross-entropy — real training cost; 3 replacing single-scalar arbitration with a genuine ensembling or evidential-uncertainty method — the largest undertaking of the two, and the only one that would generalize beyond this one component's specific failure. 7. Limitations 1. This evaluation deliberately excludes photographs with genuinely more than one correct orientation — every number here says nothing about that category. 2. A small fraction 6.4% of the full-range test's ground truth carries real human-estimation imprecision rather than an exactly-known synthetic rotation — now confirmed real by cluster bootstrap, though the interval is wide only 52 source photographs carry this kind of ground truth at all . 3. A genuine weak spot persists at the exact 45° tie point even after fair scoring — confirmed the single largest concentration of real errors §5.7 . 4. The notReal mockup/AI-generated population is real but small relative to production traffic mix, which is overwhelmingly genuine product photography. Its accuracy gap against real photos is confirmed real by cluster bootstrap on both Test 1 and Test 2 §4.2 . 5. All rotation tests apply a known, exact synthetic rotation — standard methodology, but not literally identical to an uncontrolled real camera tilt. 6. Formal repeated-measures statistical intervals now cover four paired comparisons the ablation study, the directional tilt-bias finding, and the real-vs-notReal and ground-truth-noise gaps but not yet every other point estimate in this document. 7. The consumer-chat-assistant comparison uses interfaces that do not confirm a pinned model version, and represents smaller samples than this paper's main pool. 8. A check of the §5.6 evaluation pool found 3 pairs of photographs of 200 sharing an underlying source image at different synthetic rotations — each pair's two entries remain legitimately distinct test cases, but this is a minor, disclosed imperfection in that pool's construction. 9. §5.7's failure breakdown is automated by angle bucket and arbitration path — it establishes where and when errors concentrate, not the visual content that causes them; true qualitative categorization symmetric packaging, low contrast, obscured text remains genuinely open. 10. One component's confidence output is structurally uninformative at the high end §6.3 — real, root-caused, and not fixable by recalibration alone. A test-time-consistency check now mitigates it in production shipped 2026-09-16, net positive but not a full fix — it only catches a selective subset of the affected cases ; the underlying confidence head itself remains structurally uninformative, and a genuine architectural or retraining fix is still open. 11. A targeted human re-review 2026-09-16 of 82 "already-clean" reference photos this evaluation's ground truth was built on found 33 with a real, previously undetected tilt and 26 genuinely ambiguous 3+ valid uprights — both corrected/excluded and re-scored against §5.1–5.4's headline figures, §5.2's tilt-within-10-given-quadrant-correct and 45°-tie-point figures, and §5.3's directional-bias breakdown each independently re-derived from the same corrected case data, methodology cross-checked by first reproducing the original, uncorrected numbers exactly before trusting the corrected ones . Two figures in this document were NOT re-derived against this correction and may still reflect the small number of affected cases: §5.5's risk-coverage figures see the next item — a separate, deeper issue makes these unsafe to patch numerically for now and §5.7's automated wrong-rate-by-bucket table. 12. §5.5's risk-coverage figures 88.44%/96.78% at the "0.80 threshold," 66.78%/97.04% at "0.90" predate a real, separate change 2026-09-16 : the deployed confidence thresholds were rescaled to calibrated-space 0.49/0.82 the same day, and this section was never re-derived against either that rescale or the ground-truth correction above. A real, already-computed rescale-only measurement exists uprightConfidencePolicy.mjs 's own header: calibrated ≥0.49 → 86.4% coverage/94.3% accuracy, calibrated ≥0.82 → 63.9%/~95.2% but does not cleanly reconcile with this section's own numbers, most likely a different population or bucketing — not yet resolved, disclosed rather than guessed at. 8. Future Work - Extending repeated-measures statistical coverage to every remaining reported point estimate in this document, not only the four now covered. - True qualitative, visual categorization of why specific cases fail — §5.7's breakdown covers where and when, not the underlying visual cause. - Dedicated handling for genuinely multi-valid-angle and rotationally symmetric photography, where a system should represent that ambiguity explicitly rather than resolving it arbitrarily — a real, separate feature effort, not scoped here. - Addressing the fine-tilt component's directional training-data imbalance identified in §5.3 — requires real model retraining, deliberately not undertaken in this evaluation pass. - The structurally uninformative confidence output identified in §6.3 now has a shipped mitigation a test-time-consistency check ; the two deeper fixes remain open — retraining that component's confidence head with a calibration-aware objective, or replacing single-scalar arbitration with a genuine ensembling/evidential-uncertainty method. - A closer, pinned-API comparison against general vision-language models, with this system scored on the identical image set under identical conditions — real, additional paid API cost. - A direct operational comparison against metadata-only and rule-based baselines — accuracy, human-review rate, latency, and cost per image. 9. Conclusion BlueVetaUpright demonstrates that a domain-specialized system can reach high, production-relevant accuracy on the genuinely difficult problem of semantic product-photo orientation correction — a problem independent, peer-reviewed research confirms remains unreliable for general-purpose vision-language models, and a weakness this paper's own testing independently reproduced across three separate consumer assistants. Measured directly against those three systems on real photographs, BlueVetaUpright shows a clear, substantial advantage against two of them and a real, if closer, advantage against the third — including the cleanest comparison available, an identical 100-photo, matched-sample test against Claude, where the margin is large on both the cardinal-rotation call and the precise-angle correction. Its architecture's specific design target — severe angular tilt, where a trained arbitration layer delivers a statistically robust improvement over any individual component alone — is exactly where its advantage over general-purpose alternatives is strongest. Every finding in this paper, flattering or not, is reported as measured, with its real limitations, open questions, and remaining improvement opportunities stated plainly rather than smoothed over. References - Geifman, Y., & El-Yaniv, R. 2017 . Selective Classification for Deep Neural Networks. Advances in Neural Information Processing Systems 30 NeurIPS 2017 . - Gidaris, S., Singh, P., & Komodakis, N. 2018 . Unsupervised Representation Learning by Predicting Image Rotations. International Conference on Learning Representations ICLR 2018 . arXiv:1803.07728. - Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. 2017 . On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning ICML 2017 . - Jaderberg, M., Simonyan, K., Zisserman, A., & Kavukcuoglu, K. 2015 . Spatial Transformer Networks. Advances in Neural Information Processing Systems 28 NeurIPS 2015 . - Niu, T., Cho, J., Stengel-Eskin, E., & Bansal, M. 2026 . RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics EACL 2026 . arXiv:2508.13968. - Tan, M., & Le, Q. V. 2021 . EfficientNetV2: Smaller Models and Faster Training. Proceedings of the 38th International Conference on Machine Learning ICML 2021 . arXiv:2104.00298. Questions? Email office@blueveta.com