{"slug": "llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards", "title": "LLM cross-checking catches specification bugs missed by UVM scoreboards", "summary": "Researchers developed Verify the Verifier, an LLM-based checker that compares traces from passing UVM regressions against specifications to catch bugs conventional scoreboards miss. Across 564 bugs inserted into four OpenTitan IPs, 68 were flagged as candidate scoreboard escapes and the framework confirmed 41. The work targets specification violations that pass normal regression testing, though false-positive rates, human review cost and generalisation beyond inserted bugs remain unstated.", "body_md": "Researchers present Verify the Verifier, an LLM-based checker that compares traces from passing UVM regressions with specifications. Of 564 bugs inserted into four OpenTitan IPs, 68 were candidate scoreboard escapes, and the framework confirmed 41.\n\nPassing tests can hide specification violations; cross-checking traces against specifications could catch costly hardware bugs earlier.\n\nWhat happened: an LLM anomaly cross-checker examined passing regression traces against specifications and confirmed 41 of 68 candidate escapes across four OpenTitan IPs. Why it matters: it targets bugs invisible to conventional scoreboards. Main uncertainty: false-positive rate, human review cost, and generalisation beyond inserted bugs and these IPs are not stated.\n\nSource: [arXiv](https://arxiv.org/abs/2611.26588v1)\n\n*AI disclosure: this article was produced with AI assistance and reviewed for accuracy.*", "url": "https://wpnews.pro/news/llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards", "canonical_source": "https://dev.to/paragonaimag/llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards-20el", "published_at": "2026-11-21 07:02:38+00:00", "updated_at": "2026-11-21 07:11:56.704635+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-research", "machine-learning"], "entities": ["Verify the Verifier", "OpenTitan", "UVM", "arXiv"], "also_reported_by": [], "takeaway": "An LLM cross-checker comparing passing UVM regression traces against specifications confirmed 41 of 68 candidate scoreboard escapes across four OpenTitan IPs, catching specification bugs that conventional scoreboards miss.", "recommendation": {"advice": "Cross-check traces from passing regressions against specifications with an LLM checker to surface scoreboard escapes, then budget human review for the candidate bugs it flags.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards", "markdown": "https://wpnews.pro/news/llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards.md", "text": "https://wpnews.pro/news/llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards.txt", "jsonld": "https://wpnews.pro/news/llm-cross-checking-catches-specification-bugs-missed-by-uvm-scoreboards.jsonld"}}