{"slug": "why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness", "title": "Why Chip Design AI Benchmarks Must Go Beyond Functional Correctness", "summary": "ChipAgents, working with NVIDIA, released Build-A-Chip-Bench, an end-to-end benchmark that scores AI agents on carrying a chip design from workload specification through a verified GDS layout across designs of 100K-5M standard cells. In results reported by ChipAgents, Claude Fable 5.1 posted the highest overall score at 16.6%, with designs required to pass hidden verification IP and a routed-GDS gate with no hold-time or design-rule violations before being scored on power, performance, and area. One sample task requires an INT4 inference accelerator for Qwen3-0.6B built with Verilator, OpenSTA, OpenROAD, and the Nangate45 PDK that meets a 400 MHz timing target, fits within 6 mm², and delivers at least 15 decode tokens per second at under 10 W with memory bandwidth pinned at 12.8 GB/s.", "body_md": "Demand for custom silicon is rising as physical AI, data centers, automotive systems, and edge devices require architectures optimized for specific workloads, memory bandwidth, latency, and energy efficiency. Delivering these designs requires teams to coordinate heterogeneous compute blocks, complex interconnects, and multiple domains while closing increasingly demanding power, performance, and area (PPA) targets.\n\nAI offers a way to expand engineering capacity and accelerate chip design cycles. Design and verification engineers already use AI to generate test cases, assist debugging, and implement portions of RTL. The next question is whether these capabilities can extend across an entire design flow, where decisions made during RTL implementation must hold up through verification, synthesis, and physical design.\n\nTo help answer that question, ChipAgents developed Build-A-Chip-Bench with NVIDIA to evaluate AI agents across the end-to-end digital IC design flow, from specification through physical implementation and verified Graphic Design System (GDS). The benchmark evaluates AI agents on realistic, demanding tasks that connect individual design steps to a measurable engineering outcome.\n\n## **How Build-A-Chip-Bench differs from existing benchmarks** \n\nBenchmarks such as CVDP and ChipBench have helped advance AI for chip design by providing common tasks for evaluating and comparing systems. As frontier models improve, however, we need more demanding evaluations to distinguish progress on individual tasks from the ability to execute a complete design flow.\n\n[View All](https://www.eetimes.com/category/sponsored-content/)\n\nBuild-A-Chip Bench raises the bar in two ways: task difficulty and end-to-end scope. It challenges agents to carry a design from workload specification through final GDS, connecting stages that narrower benchmarks evaluate in isolation. Success requires more than generating correct RTL. An agent must preserve functional correctness, satisfy PPA constraints, and resolve problems that emerge during synthesis and physical implementation.\n\n## **Testing a representative design flow** \n\nThe tasks exercise a representative specification-to-GDS flow and the dependencies across architecture, RTL, verification, and physical implementation at moderately large scale (designs of 100K-5M standard cells). In industry, these activities typically involve teams of architects, RTL designers, design verification (DV) engineers, physical design (PD) engineers, EDA engineers, and more. The benchmark requires an agent to manage the technical dependencies across these disciplines.\n\nEach task supplies a workload specification, a fixed interface, and an open-source EDA environment. The agent is responsible for architecture, microarchitecture, RTL, verification, and physical implementation. Reference designs establish achievable constraints. Hidden verification IP evaluates functional behavior against held-out tests, and a separate grader evaluates the final submission.\n\nOne example task requires an INT4 inference accelerator for Qwen3-0.6B using Verilator, OpenSTA, OpenROAD, and Nangate45 PDK. The implementation must meet a 400 MHz timing target, fit within 6 mm², and deliver at least 15 decode tokens per second at less than 10 W. Memory bandwidth is pinned at 12.8 GB/s. The agent selects the datapath architecture, buffering, and scheduling. Beyond RTL, it also writes testbenches, creates EDA flow scripts, and does manual floorplanning and placement as needed.\n\n## **Evaluating correctness and PPA** \n\nAll models were evaluated using fixed open-weight harnesses. Designs must pass functional verification against the hidden VIP and a physical gate requiring a routed GDS layout with no hold-time or design-rule violations. Designs passing both gates are scored on PPA relative to task constraints; failures receive zero. The overall score averages all tasks.\n\nClaude Fable 5.1 achieved the highest overall score at 16.6%, with functional and GDS pass rates of 65% and 75%, respectively. GPT-6 Astra led both pass rates at 85%, but its lower PPA score yielded an 8.7% overall score. These preliminary results do not establish production signoff or silicon validation. Learn more about the findings from our research. [URL]\n\n## **Where end-to-end design agents still struggle** \n\nThe evaluation reveals useful engineering judgment alongside inconsistent completion. In one run, an agent applied a post-route engineering change order (ECO), resizing one instance to eliminate a remaining slew violation, then passed final-netlist equivalence checks. Attempts to improve PPA, however, can introduce functional regressions and consume the remaining time budget in debugging.\n\nAgents can also delay architectural corrections while adjusting physical-design parameters. In one run, an agent documented a storage change that would reduce SRAM macros from 51 to 27, deferred it, and spent roughly six hours trying routing configurations before implementing it. Effective closure requires tracing physical failures back to the design decisions that produced them.\n\nTool runtime further limits iteration. A single place-and-route run can exceed four hours, requiring agents to prioritize changes and reserve sufficient time for final verification and implementation.\n\nThe score plateaus in these sample runs reflect the same failure modes: functional regressions during optimization, delayed architectural corrections, and slow tool loops that stall further improvement (Figure 3).\n\nFor engineering teams, evaluation should therefore measure whether agents preserve verified behavior during optimization, identify architectural causes of physical failures, and allocate runtime effectively. Work performed at any stage must remain valid in subsequent stages.\n\nAlthough the benchmark spans the design flow, it captures only part of the complexity of a production silicon program. Under the tested toolchain and resource budgets, frontier agents demonstrated the ability to execute the specification-to-GDS flow, but still struggled to preserve correctness and converge on strong PPA. The demonstrated capabilities point to substantial potential for AI-assisted silicon design, with engineers guiding design decisions and validating the resulting implementations.\n\n## **Call for partnerships** \n\nWe are working with industry partners to launch a live leaderboard with broader model coverage and an expanded set of tasks. We welcome collaboration with semiconductor teams, model developers, and researchers advancing AI-assisted silicon design.\n\nTo explore a partnership on building Build-A-Chip-Bench together, or discuss how we can help evaluate your models and agents on Build-A-Chip-Bench, contact research@chipagents.ai.", "url": "https://wpnews.pro/news/why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness", "canonical_source": "https://www.eetimes.com/why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness/", "published_at": "2026-10-21 13:00:00+00:00", "updated_at": "2026-10-21 13:41:18.341318+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-chips", "artificial-intelligence"], "entities": ["ChipAgents", "NVIDIA", "Build-A-Chip-Bench", "Claude Fable 5.1", "Qwen3-0.6B", "OpenROAD", "Verilator", "OpenSTA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness", "markdown": "https://wpnews.pro/news/why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness.md", "text": "https://wpnews.pro/news/why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness.txt", "jsonld": "https://wpnews.pro/news/why-chip-design-ai-benchmarks-must-go-beyond-functional-correctness.jsonld"}}