{"slug": "can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels", "title": "Can an LLM measure UI thresholds from screenshots? 164 crops vs DOM gold labels", "summary": "A developer benchmarked whether large language models can judge UI style-rule compliance from screenshots by comparing GPT and Gemini verdicts against DOM-derived gold labels across 60 contrastive rule pairs. GPT scored 54/60 (90%) while Gemini scored 41/60 (68%), with both models performing far better on visual rules like contrast than on numeric thresholds such as font size and line height. The developer concluded that image review is reliable for heading sizes, line counts, and line length, but pixel-adjacent threshold checks should stay in the DOM.", "body_md": "People paste screenshots into an LLM and ask \"does this page follow the rules?\". I wanted a number for how far that works, with the answer key coming from somewhere the model can't see: the DOM.\n\nTen rules, frozen from my own site's style rules (the thresholds are mine, not a standard):\n\nI built the site at the commit *before* the fixes (`7bec67f`), opened 12 pages at 1280 and 375 in headless Chrome via CDP, and measured every rule with `getComputedStyle` / `getBoundingClientRect`. That gives 12 × 11 = **132 judgments** (R2 counts twice), of which 36 are \"element not present on this page\". 96 pairs remain.\n\nFrom the **same render**, the script clips 1:1 crops of the elements in question — 164 PNGs, SHA-256 of each in a manifest. That matters: my first attempt measured positions in an iframe and shot with headless Chrome, and at 375px the two disagreed by 300–700px, so six footer crops were blank white.\n\n```\n// R1 = smallest font-size among text nodes inside <main>, excluding chrome\nconst EXCL = 'nav, .crumbs, form, button, .works-nav, .toc-block, footer, .site-header, .site-footer';\nconst textEls = [...document.querySelectorAll('main *')].filter((e) =>\n  !['SCRIPT','STYLE','SVG','PATH','NOSCRIPT'].includes(e.tagName) && !e.closest(EXCL) && vis(e) &&\n  [...e.childNodes].some((n) => n.nodeType === 3 && n.textContent.trim()));\n```\n\nEach judge got, per page, a frozen instruction (\"for each rule answer YES / NO / unknown / not-applicable with one line of reasoning; a rule with no image is not-applicable; do not guess; at most three free remarks\") plus that page's crops.\n\nTwo different delivery paths, so this is **not** a model-vs-model comparison. Scoring is a regex over the four verbatim values against the gold — no human in the loop.\n\nMy first summary said \"GPT 94% on 96 pairs, and it got all 35 of the 44px pairs right, telling 40px from 45px\". An external reviewer looked at the gold and pointed out that in the *before* version, every breadcrumb is 40px and every footer link is 45px: within each 44px rule there are **only violations or only non-violations**. A judge that answers \"small links violate, the footer doesn't\" scores 35/35 without measuring anything. The same holds for R5.\n\nSo the main metric is now the **60 pairs whose rule contains both labels** (R1–R4). The 44px rules were scored (GPT 36/36) but that score cannot test threshold discrimination, so it is reported and not counted.\n\nAccuracy = correct YES/NO including true negatives, divided by **all** pairs (unknowns count against). GPT answered 60/60, so the two bases coincide; Gemini answered 57/60, so I also give the answered-only rate.\n\n| Judge | Group | Pairs | Answered | Correct | FP rate | FN rate | \n|---|---|---|---|---|---|---|\n| GPT | contrastive R1–R4 | 60 | 100% | **54/60 = 90%** | 7% | 21% | \n| GPT | visual (R2, R4) | 36 | 100% | **36/36** | 0% | 0% | \n| GPT | numeric (R1, R3) | 24 | 100% | 18/24 | 16% | 60% | \n| Gemini | contrastive R1–R4 | 60 | 95% | **41/60 = 68%** (41/57 = 72% of answered) | 26% | 36% | \n| Gemini | visual | 36 | 97% | 28/36 | 12% | 44% | \n| Gemini | numeric | 24 | 92% | 13/24 | 47% | 20% | \n\nWhat I'd hand to image review today: mixed heading sizes, heading line counts, line length with a multi-em gap. What stays in the DOM: anything within a couple of pixels of its threshold. What I can't say yet: whether image review discriminates the 44px tap-target threshold — the fixed version of the site needs to be shot to create contrast.\n\n*Part 3 of the series \"Measure the Outcome, Not the Proxy\" (part 1: transfer size ≠ speed; part 2: shorter ≠ requirements preserved — both on this account).*\n\n[Verification record](https://sumitsuke.jp/via/devto/lab/screenshot-ai-vs-dom-ten-rules/) (environment, verdict, evidence: gold JSON, both score sheets, the 164 crops as a ZIP, all 24 verbatim answers) and the canonical Japanese write-up. [The DOM measurer that fixed the pages](https://sumitsuke.jp/via/devto/lab/readability-dom-eight-items/). [Reproduction kit](https://github.com/sumitsuke/screenshot-ai-vs-dom-ten-rules) with the images, the gold, the judge script for Gemini and the scorer (code MIT, data CC BY 4.0). I verify AI-generated and outsourced work against pre-registered pass/fail criteria — text-only, no calls: [Sumitsuke / verification](https://sumitsuke.jp/works/verification/).", "url": "https://wpnews.pro/news/can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels", "canonical_source": "https://dev.to/tauridev/can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels-4gbb", "published_at": "2026-09-21 08:31:50+00:00", "updated_at": "2026-09-21 08:54:28.034581+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools", "computer-vision"], "entities": ["GPT", "Gemini", "Chrome DevTools Protocol", "headless Chrome"], "alternates": {"html": "https://wpnews.pro/news/can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels", "markdown": "https://wpnews.pro/news/can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels.md", "text": "https://wpnews.pro/news/can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels.txt", "jsonld": "https://wpnews.pro/news/can-an-llm-measure-ui-thresholds-from-screenshots-164-crops-vs-dom-gold-labels.jsonld"}}