{"slug": "show-hn-layoutlens-ai-powered-visual-ui-testing", "title": "Show HN: LayoutLens: AI-Powered Visual UI Testing", "summary": "LayoutLens, an AI-powered visual UI testing tool, catches layout and accessibility bugs using deterministic axe-core and geometry checks that run keyless and free in CI, with an optional vision-LLM tier that the deterministic layer can overrule. The tool's LLM tier measures 81.1% accuracy on its bundled benchmark (60/74 labeled queries with gpt-4o-mini, measured 2026-07-21). LayoutLens is available via pip and supports three tiers: deterministic, hybrid, and LLM-only.", "body_md": "LayoutLens catches the layout and accessibility bugs your pixel baseline can't\nsee and your LLM can't be trusted about — **deterministic axe-core and\ngeometry checks that run keyless and free in CI, with an optional vision-LLM\ntier that the deterministic layer is allowed to overrule.**\n\nThree tiers, use what you need:\n\n| Tier | What runs | Needs | Reliability |\n|---|---|---|---|\nDeterministic |\naxe-core WCAG A/AA + geometry/contrast/occlusion scorers | no API key or model call | measured facts, reproducible; this is the CI gate |\nHybrid (default) |\ndeterministic scan grounds the vision LLM; measured violations force the verdict |\nan LLM API key | precision-preserving: the model can add findings, never erase measured ones |\nLLM |\nnatural-language questions answered from a screenshot | an LLM API key (any LiteLLM provider, incl. Ollama/vLLM via `api_base` ) |\nhonest numbers below |\n\n```\n# Keyless, deterministic — safe as a required check on any fork\nresult = await lens.check_accessibility(\"page.html\", mode=\"axe\")\nresult = await lens.check_layout(\"page.html\", viewport=\"mobile\", mode=\"deterministic\")\n\n# Natural-language, grounded by the deterministic scan (hybrid)\nresult = await lens.analyze(\"https://example.com\", \"Is the navigation user-friendly?\")\n```\n\nOr from pytest — the deterministic assertions need no key, and `assert_ui`\n\nskips (never fails) without one:\n\n``` python\ndef test_landing_page(layoutlens):\n    layoutlens.assert_a11y(\"landing.html\")  # axe, keyless\n    layoutlens.assert_layout(\"landing.html\", viewport=\"mobile\")  # keyless\n    layoutlens.assert_ui(\"landing.html\", \"Is the CTA above the fold?\")\n```\n\n**Honest numbers:** the LLM tier measures **81.1%** on the bundled benchmark\n(60/74 labeled queries, `gpt-4o-mini`\n\n, measured 2026-07-21 —\n[artifact](/gojiplus/layoutlens/blob/main/benchmarks/results/2026-07-21_gpt-4o-mini.json)). See\n[Limitations](#limitations) for what vision models can and cannot reliably\njudge — the deterministic tier exists precisely because of those limits.\n\n```\npip install layoutlens\nplaywright install chromium  # For screenshot capture\n```\n\nLayoutLens's API is async — run it with `asyncio.run(...)`\n\n, or `await`\n\nit\ndirectly if you're already inside an `async def`\n\n(e.g. pytest-asyncio,\nFastAPI, a notebook cell). Every snippet below assumes one of those two\ncontexts; only the first one spells out the `asyncio.run(...)`\n\nwrapper.\n\n``` python\nimport asyncio\nfrom layoutlens import LayoutLens\n\nasync def main():\n    # Initialize (uses OPENAI_API_KEY env var)\n    lens = LayoutLens()\n\n    # Test any website or local HTML\n    result = await lens.analyze(\n        \"https://your-site.com\", \"Is the header properly aligned?\"\n    )\n    print(f\"Answer: {result.answer}\")\n    print(f\"Confidence: {result.confidence:.1%}\")\n\nasyncio.run(main())\n```\n\nThat's it! No selectors, no complex setup, just natural language questions.\n\nLayoutLens vendors [axe-core](https://github.com/dequelabs/axe-core) 4.10.3 and runs it against a real\nPlaywright-rendered page to catch actual WCAG 2.1 A/AA violations — not an LLM guess. This mode is fully\nkeyless: no `OPENAI_API_KEY`\n\n, no network call to an AI provider, just deterministic, reproducible results.\n\n```\n# Deterministic axe-core scan only — no API key needed\nlayoutlens page.html --a11y axe\n\n# Hybrid: axe-core + LLM vision, axe overrides the verdict on violations (needs an API key)\nlayoutlens https://example.com --a11y hybrid\n\n# Legacy vision-only accessibility check (needs an API key)\nlayoutlens page.html --a11y llm\n```\n\n`--a11y`\n\nrequires one of `hybrid`\n\n/`axe`\n\n/`llm`\n\nand is mutually exclusive with `--query`\n\n— accessibility mode\nalways uses the built-in WCAG checks instead of a free-form question.\n\n``` python\nfrom layoutlens import LayoutLens, AxeAuditor\n\n# Raw axe-core report — no LayoutLens instance or API key needed at all\nreport = await AxeAuditor().audit(\"page.html\")\nprint(report.summary())\nprint(report.ok)  # True if there are zero violations\nprint(report.violations)  # list[A11yFinding]: rule_id, impact, wcag_refs, nodes, ...\n\n# Via the LayoutLens API, restricted to WCAG A/AA tags, still keyless\nlens = LayoutLens()  # no API key required at construction\nresult = await lens.check_accessibility(\"page.html\", mode=\"axe\")\nprint(\n    result.answer\n)  # \"Yes — axe-core found no WCAG A/AA violations\" (or lists violated rules)\n```\n\n— deterministic axe-core only. No API key, no LLM call.`mode=\"axe\"`\n\n`confidence`\n\nis always`1.0`\n\n.(default for`mode=\"hybrid\"`\n\n`check_accessibility`\n\n/`check_accessibility`\n\n) — runs axe-core*and*the LLM vision analysis, injecting the axe findings into the LLM's prompt as grounding context. If axe finds any violation, the final verdict is deterministically forced to \"no\" (confidence`1.0`\n\n), regardless of what the LLM says — axe overrides the model, not the other way around. If axe finds nothing, the LLM's own answer/confidence are kept (it can still flag issues axe's automated rules can't catch, like poor color choices that pass contrast math or confusing visual hierarchy).— legacy vision-only analysis, no axe-core involved. Requires an API key.`mode=\"llm\"`\n\n```\n# Hybrid: axe grounds the LLM and can force the verdict\nresult = await lens.check_accessibility(\"page.html\", mode=\"hybrid\")\nprint(result.metadata[\"a11y\"])  # full axe report dict\nprint(result.metadata[\"engine\"])  # \"axe-core 4.10.3\"\n```\n\nAlongside axe-core, LayoutLens ships `LayoutScorer`\n\n— a keyless, LLM-free detector for\ngeometric and contrast defects, measured directly off the rendered page with the browser's\nown layout engine. Foundational contrast and geometry measurements were ported from\n[UIJudgeBench](https://github.com/gojiplus/uijudge-bench); newer WCAG and text-occlusion\nchecks are independent LayoutLens implementations evaluated by that benchmark. It finds:\n\n**contrast**— text below the WCAG AA ratio (4.5:1 normal, 3.0:1 large), with the measured ratio** overlap**— sibling elements whose bounding boxes collide** clipping**— content cut off by a fixed-size box with hidden overflow** viewport-protrusion**— elements extending past the viewport width (horizontal-scroll bugs)** target-size**— undersized targets that also fail the machine-measurable WCAG 2.5.8 spacing, inline, and unmodified user-agent-control exceptions**focus-obscured**— keyboard-focused components entirely hidden by author DOM content (the automatable geometric core of WCAG 2.4.11)** text-occlusion**— rendered text, including chart labels, covered by another painted DOM element; this is a visual-quality finding, not a WCAG criterion\n\n``` python\nfrom layoutlens.layout import LayoutScorer, contrast_ratio, read_computed_styles\n\n# Scan a page — no LayoutLens instance, no API key, deterministic.\nreport = await LayoutScorer().scan(\"page.html\", viewport=\"mobile\")\nprint(report.ok)  # True if no defects found\nprint(report.summary())  # findings grouped by class, with measured receipts\nfor f in report.findings:\n    print(\n        f.defect_class, f.selector, f.measured\n    )  # each finding carries the numbers behind it\n\n# Or use the pure WCAG contrast math directly (no browser):\ncontrast_ratio((0x76, 0x76, 0x76), (0xFF, 0xFF, 0xFF))  # -> 4.54\n```\n\nEvery finding is a receipt: the offending selector, its bounding box, the measured value, and\nthe threshold it violated. `scan(viewport=...)`\n\nre-runs the geometry at any viewport, so\nprotrusion/overlap that only appear on mobile are caught. Automated findings are not a\nsite-wide WCAG conformance claim. In particular, target-size equivalent/essential exceptions\nand focus-obscuration interaction-history exceptions remain explicit manual-review fields.\n\nInstalling layoutlens registers a pytest plugin (entry point `layoutlens`\n\n).\nThe `layoutlens`\n\nfixture gives you three assertions:\n\n``` python\ndef test_checkout(layoutlens):\n    layoutlens.assert_a11y(\"checkout.html\")  # keyless axe gate\n    layoutlens.assert_layout(\n        \"checkout.html\", viewport=\"mobile\"\n    )  # keyless geometry gate\n    layoutlens.assert_ui(\n        \"checkout.html\", \"Is the pay button the most prominent element?\"\n    )\n```\n\n`assert_a11y`\n\n/`assert_layout`\n\nare**keyless and deterministic**— they run on every fork and PR with no secrets, and failure messages carry the rule id, selector, and measured numbers.`assert_ui`\n\n(vision LLM)**skips instead of failing** when no API key is configured, or always with`--layoutlens-no-llm`\n\n— so one suite serves both the free deterministic lane and the LLM lane.`--layoutlens-model`\n\npicks the model for`assert_ui`\n\n.\n\n`layoutlens-mcp`\n\nexposes the checks as [MCP](https://modelcontextprotocol.io)\ntools for Claude Code, Cursor, and friends:\n\n```\npip install \"layoutlens[mcp]\"\n# register the stdio server in your agent config:\n#   command: layoutlens-mcp\n```\n\nTools: `audit_accessibility`\n\nand `scan_layout`\n\n(keyless, deterministic —\nthey return **measured numbers, not model opinions**, in compact summaries of\na few hundred tokens), plus `check_ui`\n\nand `compare_ui`\n\n(vision LLM). The\ndeterministic tools cover visual facts accessibility-tree snapshots cannot\nsee: contrast, geometry, target spacing, complete focus obscuration, and text\nocclusion such as a chart line painted over its label.\n\nBoth deterministic engines emit [SARIF 2.1.0](https://sarifweb.azurewebsites.net/):\n\n```\nlayoutlens page.html --layout deterministic --output sarif > layout.sarif\nlayoutlens page.html --a11y axe --output sarif > a11y.sarif\n```\n\nUpload with `github/codeql-action/upload-sarif`\n\nand findings appear as PR\nannotations with stable rule ids (`layout/page-overflow`\n\n, `axe/color-contrast`\n\n,\n...) tracked over time — keyless, so it works on every fork.\n\nOr use the packaged action —\n[ gojiplus/layoutlens-action](https://github.com/gojiplus/layoutlens-action)\n— which bundles install, scan, job summary, PR annotations, a sticky results\ncomment, and the SARIF upload into one step:\n\n```\n- uses: gojiplus/layoutlens-action@v1\n  with:\n    sources: \"dist/*.html\"\n```\n\nTest single pages with custom questions:\n\n```\n# Test local HTML files\nresult = await lens.analyze(\"checkout.html\", \"Is the payment form user-friendly?\")\n\n# Test with expert context\nfrom layoutlens.prompts import Instructions, UserContext\n\ninstructions = Instructions(\n    expert_persona=\"conversion_expert\",\n    user_context=UserContext(\n        business_goals=[\"reduce_cart_abandonment\"], target_audience=\"mobile_shoppers\"\n    ),\n)\n\nresult = await lens.analyze(\n    \"checkout.html\",\n    \"How can we optimize this checkout flow?\",\n    instructions=instructions,\n)\n```\n\nPerfect for A/B testing and redesign validation. `compare()`\n\naccepts URLs,\nlocal HTML files, or screenshot images — every source is rendered and every\nscreenshot is sent to the model:\n\n```\nresult = await lens.compare(\n    [\"https://old-design.example.com\", \"https://new-design.example.com\"],\n    \"Which design is more accessible?\",\n)\nprint(f\"Winner: {result.answer}\")\n```\n\nDomain expert knowledge with one line of code:\n\n```\n# Professional accessibility audit (WCAG expert)\nresult = await lens.check_accessibility(\"product-page.html\", compliance_level=\"AA\")\n\n# Conversion rate optimization (CRO expert)\nresult = await lens.optimize_conversions(\n    \"landing.html\", business_goals=[\"increase_signups\"], industry=\"saas\"\n)\n\n# Mobile UX analysis (Mobile expert)\nresult = await lens.analyze_mobile_ux(\"app.html\", performance_focus=True)\n\n# E-commerce audit (Retail expert)\nresult = await lens.audit_ecommerce(\"checkout.html\", page_type=\"checkout\")\n\n# Legacy methods still work\nresult = await lens.check_accessibility(\"product-page.html\")  # Backward compatible\n```\n\n`analyze()`\n\nhandles single or multiple sources/queries — pass lists to either\n`source`\n\nor `query`\n\nand it fans out every combination concurrently:\n\n```\nresults = await lens.analyze(\n    source=[\"home.html\", \"about.html\", \"contact.html\"],\n    query=[\"Is it accessible?\", \"Is it mobile-friendly?\"],\n)\n# Returns a BatchResult; processes 6 combinations concurrently\nprint(f\"{results.successful_queries}/{results.total_queries} succeeded\")\n# Cap concurrent API calls with max_concurrent\nresult = await lens.analyze(\n    source=[\"page1.html\", \"page2.html\", \"page3.html\"],\n    query=\"Is it accessible?\",\n    max_concurrent=5,\n)\n```\n\nAll results provide clean, typed JSON for automation:\n\n```\nresult = await lens.analyze(\"page.html\", \"Is it accessible?\")\n\n# Export to clean JSON\njson_data = result.to_json()  # Returns typed JSON string\nprint(json_data)\n# {\n#   \"source\": \"page.html\",\n#   \"query\": \"Is it accessible?\",\n#   \"answer\": \"Yes, the page follows accessibility standards...\",\n#   \"confidence\": 0.85,\n#   \"reasoning\": \"The page has proper heading structure...\",\n#   \"screenshot_path\": \"/path/to/screenshot.png\",\n#   \"viewport\": \"desktop\",\n#   \"timestamp\": \"2024-01-15 10:30:00\",\n#   \"execution_time\": 2.3,\n#   \"metadata\": {}\n# }\n\n# Type-safe structured access\nfrom layoutlens.types import AnalysisResultJSON\nimport json\n\ndata: AnalysisResultJSON = json.loads(result.to_json())\nconfidence = data[\"confidence\"]  # Fully typed: float\n```\n\nChoose from 6 built-in domain experts with specialized knowledge:\n\n```\n# Available experts: accessibility_expert, conversion_expert, mobile_expert,\n# ecommerce_expert, healthcare_expert, finance_expert\n\n# Use any expert with custom analysis\nresult = await lens.analyze_with_expert(\n    source=\"healthcare-portal.html\",\n    query=\"How can we improve patient experience?\",\n    expert_persona=\"healthcare_expert\",\n    focus_areas=[\"patient_privacy\", \"health_literacy\"],\n    user_context={\n        \"target_audience\": \"elderly_patients\",\n        \"accessibility_needs\": [\"large_text\", \"simple_navigation\"],\n        \"industry\": \"healthcare\",\n    },\n)\n\n# Expert comparison analysis (URLs, local HTML files, or screenshots)\nresult = await lens.compare_with_expert(\n    sources=[\"https://old.example.com\", \"https://new.example.com\"],\n    query=\"Which design converts better?\",\n    expert_persona=\"conversion_expert\",\n    focus_areas=[\"cta_prominence\", \"trust_signals\"],\n)\n```\n\nTest suites are declared in YAML/JSON and loaded into a `UITestSuite`\n\n. **Breaking\nchange (v1.7.0):** every test case must declare `expected_results`\n\n— an `answer`\n\n(\"yes\"/\"no\", matched against the parsed leading yes/no token of the analysis\nanswer) and/or a `contains`\n\nlist (terms that must appear, case-insensitively, in\nthe answer + reasoning). A case with no `expected_results`\n\nnow raises\n`ValidationError`\n\nat load time instead of silently grading on confidence alone.\n\n```\n# test_suite.yaml\nname: \"Homepage Suite\"\ndescription: \"Accessibility and layout checks\"\ntest_cases:\n  - name: \"Navigation Alignment\"\n    html_path: \"pages/home.html\"\n    queries:\n      - \"Is the navigation menu properly centered?\"\n    viewports: [\"desktop\"]\n    expected_results:\n      answer: \"yes\"\n      contains: [\"centered\"]\n    expected_confidence: 0.7   # optional, defaults to 0.7\npython\nimport yaml\nfrom layoutlens import LayoutLens, UITestSuite\n\nwith open(\"test_suite.yaml\") as f:\n    suite = UITestSuite.from_dict(yaml.safe_load(f))\n\nlens = LayoutLens()\nresults = await lens.run_test_suite(suite)  # list[UITestResult], one per test case\nfor r in results:\n    print(f\"{r.test_case_name}: {r.passed_tests}/{r.total_tests} passed\")\n    print(r.to_json())  # includes per-assertion \"assertion_detail\"\n```\n\nThere is no CLI subcommand for suites — `run_test_suite`\n\nis a Python API only.\nSee [ examples/sample_test_suite.yaml](/gojiplus/layoutlens/blob/main/examples/sample_test_suite.yaml) for a\ncomplete, runnable example.\n\nFor external evaluation harnesses (e.g. UIJudgeBench), `judge()`\n\nsends your\nprompt **verbatim** — no persona, no scaffolding, no appended JSON contract —\nalongside a single image, and returns a parsed, structured verdict. Your harness\nowns the entire prompt, including its own response contract and prompt versioning.\n\n``` python\nfrom layoutlens import LayoutLens\n\nlens = LayoutLens(model=\"gpt-4o\")  # or any vision model via provider/api_base\n\nprompt = (\n    \"You are a UI evaluation judge. Compare the layout in the image against the \"\n    \"criteria below and respond ONLY as JSON: \"\n    '{\"answer\": \"A\" | \"B\", \"confidence\": 0.0-1.0, \"rationale\": \"...\"}.\\n'\n    \"Criteria: which layout has clearer visual hierarchy?\"\n)\n\nresult = await lens.judge(\"candidate.png\", prompt, max_tokens=300)\n\nresult.answer  # parsed \"answer\" field, or \"unknown\" if unparseable\nresult.confidence  # parsed 0-1, else 0.0\nresult.rationale  # parsed \"rationale\"/\"reasoning\", else \"\"\nresult.raw  # full raw model text\nresult.refused  # True if the model declined\nresult.usage  # {\"prompt_tokens\": ..., \"completion_tokens\": ..., \"total_tokens\": ...}\nresult.parse_mode  # \"json\" | \"fallback\" | \"none\"\n```\n\nFor bulk evaluation, `judge_batch()`\n\nuses provider-native asynchronous Batch\nAPIs. Native OpenAI uses the official Responses Batch API, `gemini/*`\n\nmodels use\nthe Google Gen AI inline Batch API, and other supported providers use LiteLLM's\nfile-based Batch API. For example, a localization benchmark can preserve the\ninput coordinate frame and explicitly cap reasoning:\n\n``` python\nfrom layoutlens import BatchRequest, LayoutLens\n\nlens = LayoutLens(provider=\"openai\", model=\"gpt-5.6-luna\")\nresults = await lens.judge_batch(\n    [BatchRequest(\"item-1\", \"target.jpg\", prompt)],\n    max_tokens=256,\n    reasoning_effort=\"low\",\n    image_detail=\"original\",\n)\n```\n\nResume manifests are content-addressed by the exact prompts, images, model,\nbackend, endpoint, token budget, reasoning effort, and image detail, so a changed\nrequest cannot reuse a stale response. A per-manifest lock prevents two\nprocesses from submitting the same exact batch concurrently. Manifests created\nbefore 2.1.1 fail closed with explicit migration details because they cannot\nattest their original prompts, images, or token budget. Changing an input creates\na new fingerprint; if any prior same-model manifest records an overlapping\nsubmitted id, resume fails closed until the user explicitly migrates the job or\nauthorizes a fresh billed run. An ungraceful process stop can leave\na `.json.lock`\n\nfile: confirm no matching run is active, then remove only that\nlock file to resume from the preserved manifest.\n\nKey guarantees:\n\n-\n**Verbatim prompt**— LayoutLens adds nothing to the text you provide. -\n**No caching**— every judge call hits the model, so a benchmark controls its own determinism. -\n**Per-model parameter policy**— models that reject non-default sampling params (Claude Sonnet 5, Opus 4.6+) omit`temperature`\n\nautomatically; others send`temperature=0.0`\n\n. -\n**Self-hosted endpoints**— point at Ollama/vLLM via`api_base`\n\n:\n\n```\nlens = LayoutLens(\n    provider=\"litellm\",\n    model=\"ollama/qwen2.5vl\",\n    api_base=\"http://localhost:11434\",\n)\n# Analyze a single page\nlayoutlens https://example.com \"Is this accessible?\"\n\n# Analyze local files\nlayoutlens page.html \"Is the design professional?\"\n\n# Compare two designs (URLs, local HTML files, or screenshot images)\nlayoutlens https://old.example.com https://new.example.com --compare\n\n# Analyze with different viewport\nlayoutlens site.com \"Is it mobile-friendly?\" --viewport mobile\n\n# JSON output for automation\nlayoutlens page.html \"Is it accessible?\" --output json\n\n# Deterministic WCAG accessibility scan — no API key required\n# (see \"Deterministic Accessibility Checks\" above for hybrid/llm modes)\nlayoutlens page.html --a11y axe\n\n# Choose model / pass an API key explicitly\nlayoutlens page.html \"Is it accessible?\" --model gpt-4o --api-key sk-...\n```\n\nRun `layoutlens`\n\nwith no arguments (or `--help`\n\n) to see the full flag reference:\n`--query/-q`\n\n, `--compare/-c`\n\n, `--viewport/-v {desktop,mobile,tablet}`\n\n,\n`--output/-o {text,json}`\n\n, `--api-key`\n\n, `--model/-m`\n\n, `--a11y {hybrid,axe,llm}`\n\n.\n\n```\n- name: Visual UI Test\n  run: |\n    pip install layoutlens\n    playwright install chromium\n    layoutlens ${{ env.PREVIEW_URL }} \"Is it accessible and mobile-friendly?\"\npython\nimport pytest\nfrom layoutlens import LayoutLens\n\n@pytest.mark.asyncio\nasync def test_homepage_quality():\n    lens = LayoutLens()\n    result = await lens.analyze(\"homepage.html\", \"Is this production-ready?\")\n    assert result.confidence > 0.8\n    assert \"yes\" in result.answer.lower()\n```\n\nLayoutLens bundles a compact benchmark suite (18 fixtures / 74 labeled queries) for smoke-testing\nAI performance. For a larger, paper-rigor benchmark of AI judges of web UI quality — 4,000+\nmachine-verified items across accessibility, layout, and referring tasks, built on LayoutLens's\nown axe/browser machinery — see ** UIJudgeBench**\n(\n\n[dataset on Hugging Face](https://huggingface.co/datasets/gojiberries/uijudge-bench)). LayoutLens is a planned judge baseline there.\n\n```\n# Run LayoutLens against test data\npython benchmarks/run_benchmark.py --api-key sk-your-key\n\n# With custom settings\npython benchmarks/run_benchmark.py \\\n  --api-key sk-your-key \\\n  --output benchmarks/my_results \\\n  --no-batch \\\n  --filename custom_results.json\n# Evaluate results against ground truth\npython benchmarks/evaluation/evaluator.py \\\n  --answer-keys benchmarks/answer_keys \\\n  --results benchmarks/layoutlens_output \\\n  --output evaluation_report.json\n```\n\nThe evaluator scores every answer deterministically (leading yes/no token vs the\nanswer key; ambiguous answers count as incorrect) and writes an artifact with\nper-category and overall accuracy. The committed\n[ benchmarks/results/2026-07-21_gpt-4o-mini.json](/gojiplus/layoutlens/blob/main/benchmarks/results/2026-07-21_gpt-4o-mini.json)\nis a real measured run:\n\n```\n{\n  \"evaluation_summary\": {\n    \"date\": \"2026-07-21\",\n    \"model\": \"gpt-4o-mini\",\n    \"total_queries\": 74,\n    \"total_correct\": 60,\n    \"ambiguous_answers\": 7,\n    \"overall_accuracy\": 0.811,\n    \"evaluator_version\": \"2.0\",\n    \"evaluator_method\": \"Deterministic structured yes/no; ambiguous answers count as incorrect.\"\n  },\n  \"category_results\": {\n    \"responsive_design\": {\"total_queries\": 21, \"correct_predictions\": 20, \"accuracy\": 0.952},\n    \"layout_alignment\":  {\"total_queries\": 24, \"correct_predictions\": 19, \"accuracy\": 0.792},\n    \"accessibility\":     {\"total_queries\": 21, \"correct_predictions\": 16, \"accuracy\": 0.762},\n    \"ui_components\":      {\"total_queries\": 8,  \"correct_predictions\": 5,  \"accuracy\": 0.625}\n  }\n}\n```\n\nCreate your own test data and answer keys:\n\n``` python\n# Use the async API for custom benchmark workflows\nfrom layoutlens import LayoutLens\n\nasync def run_custom_benchmark():\n    lens = LayoutLens()\n\n    test_cases = [\n        {\"source\": \"page1.html\", \"query\": \"Is it accessible?\"},\n        {\"source\": \"page2.html\", \"query\": \"Is it mobile-friendly?\"},\n    ]\n\n    results = []\n    for case in test_cases:\n        result = await lens.analyze(case[\"source\"], case[\"query\"])\n        results.append(\n            {\n                \"test\": case,\n                \"result\": result.to_json(),  # Clean JSON output\n                \"passed\": result.confidence > 0.7,\n            }\n        )\n\n    return results\n```\n\nSimple configuration options:\n\n```\n# Via environment\nexport OPENAI_API_KEY=\"sk-...\"\n\n# Via code\nlens = LayoutLens(\n    api_key=\"sk-...\",\n    model=\"gpt-4o-mini\",  # or \"gpt-4o\" for higher accuracy\n    cache_enabled=True,   # Reduce API costs\n    cache_type=\"memory\",  # \"memory\" or \"file\"\n)\n```\n\nCalibrate your trust to the tier you use:\n\n**Vision LLMs miss fine-grained UI differences.** On DiffSpot ([arXiv 2605.29615](https://arxiv.org/abs/2605.29615)), a 2026 benchmark of fine-grained web-UI changes, the best frontier model scored 47.2% overall and**under 23% recall on the hard tier**; open models hallucinated differences on 18–24% of identical pairs. Do not use the LLM tier as a sole gate for subtle visual regressions — that is what the deterministic scorers are for.**Passing axe-core is not WCAG conformance.** Automated rules cover only a subset of WCAG; Microsoft's a11y LLM evaluation makes the same disclaimer for its own checks. axe passing means \"no automated rule failed\", not \"accessible\".**The deterministic scorers measure rendered facts, not full intent.** The WCAG 2.5.8 spacing, inline, and unmodified-user-agent-control exceptions are modeled. Equivalent-control and essential-presentation exceptions still require review, as do interaction-history cases under WCAG 2.4.11. General text occlusion is a visual-quality signal, not a WCAG conformance claim. Findings carry their measured numbers so you can judge.**Our own benchmark is small**(74 labeled queries) and easier than DiffSpot-class tasks; the 81.1% figure is honest but narrow. The harness is model-agnostic (`benchmarks/run_benchmark.py --model ...`\n\n) — re-run it rather than trusting ours.\n\n- 📖\n- Comprehensive guides and API reference[Full Documentation](https://gojiplus.github.io/layoutlens/) - 🎯\n- Real-world usage patterns[Examples](https://github.com/gojiplus/layoutlens/tree/main/examples) - 🐛\n- Report bugs, request features, get help[Issues](https://github.com/gojiplus/layoutlens/issues)\n\n**Natural Language**- Write tests like you'd describe the UI to a colleague** Domain Expert Knowledge**- Built-in expertise in accessibility, CRO, mobile UX, and more** Rich Context Support**- Business goals, user personas, compliance standards, and technical constraints** Zero Selectors**- No more fragile XPath or CSS selectors** Visual Understanding**- AI sees what users see, not just code** Async-by-Default**- Concurrent processing for optimal performance** Simple API**- One analyze method handles single pages, batches, and comparisons** Structured JSON Output**- TypedDict schemas for full type safety in automation** Honest Benchmarking**- Compact built-in suite (81.1% measured accuracy, gpt-4o-mini, 74 queries); see[UIJudgeBench](https://github.com/gojiplus/uijudge-bench)for the full-scale external benchmark**Deterministic Accessibility**- Vendored axe-core WCAG 2.1 A/AA checks, no API key or LLM variance\n\n*Making UI testing as simple as asking \"Does this look right?\"*", "url": "https://wpnews.pro/news/show-hn-layoutlens-ai-powered-visual-ui-testing", "canonical_source": "https://github.com/gojiplus/layoutlens", "published_at": "2026-08-23 02:33:56+00:00", "updated_at": "2026-08-23 02:44:33.045836+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-tools", "ai-products"], "entities": ["LayoutLens", "axe-core", "Playwright", "gpt-4o-mini", "LiteLLM", "Ollama", "vLLM", "Deque Labs"], "alternates": {"html": "https://wpnews.pro/news/show-hn-layoutlens-ai-powered-visual-ui-testing", "markdown": "https://wpnews.pro/news/show-hn-layoutlens-ai-powered-visual-ui-testing.md", "text": "https://wpnews.pro/news/show-hn-layoutlens-ai-powered-visual-ui-testing.txt", "jsonld": "https://wpnews.pro/news/show-hn-layoutlens-ai-powered-visual-ui-testing.jsonld"}}