PINNACLE Measures the Work, Not the Model

Signal65's new PINNACLE benchmark, developed with Kamiwaza, evaluates AI systems on whole enterprise task completion rather than model scores or token throughput, finding that 43 of 44 configurations declined in performance when tested against messy data, with a median 28-point drop in whole-workflow completion. The benchmark, which measures cost per correct task and uses binary whole-job completion, showed only two configurations—Claude Opus 5 at 98.3% and GPT-5.6 Sol at 95.8%—completed at least 95% of jobs on messy data.

TL;DR — Key Takeaways - Signal65’s PINNACLE benchmark evaluates AI systems based on whether they complete entire enterprise tasks correctly, rather than relying primarily on model scores or token throughput. - Data quality had a major impact on performance, with 43 of 44 configurations declining when tested against duplicated, contradictory and partially migrated data. - PINNACLE measures cost per correct task, revealing that repeated context processing and failed attempts can make agentic AI workloads significantly more expensive than model price sheets suggest. Signal65’s PINNACLE benchmark moves the argument from abstract intelligence and token speed to a harder enterprise measure: whether the entire job came out right, how much capacity the platform sustained and what each correct result cost. Enterprise AI procurement has been built around a category error. Buyers compare model scores and token rates, then ask those numbers to stand in for a business outcome. The problem is not that conventional benchmarks are fraudulent. It is that they stop before the enterprise work begins. Once an agent has to navigate company files, reconcile inconsistent records, call tools and produce a deliverable that someone can accept, the unit of performance changes. The relevant question is no longer whether a model answered a test question. It is whether the system finished the job correctly, at the required volume and at a cost the business can defend. That is the unit Signal65 is testing with PINNACLE, a new benchmark suite developed in partnership with Kamiwaza. PINNACLE measures models, accelerators and complete solutions from the same scored runs. It treats correct work—not an answer, a token or a GPU—as the outcome that matters. What PINNACLE Actually Measures PINNACLE agents perform multi-step enterprise jobs against generated filesystems, databases and Python runtimes. The deeper workflows can require up to 80 or more rounds of tool use. Each environment and its answer key are generated fresh, with the answer key held outside the agent sandbox. The final deliverables are graded in code. No model judges another model’s response, and no human rater decides whether an answer feels correct. The strictness matters. PINNACLE reports deliverable-level accuracy, but whole-job completion is binary. A workflow that creates six of seven required deliverables correctly has still failed to deliver a usable package. That rule will produce harsher numbers than benchmarks that award credit for every successful intermediate step. It is also closer to the way production work is accepted or rejected. The launch data covers 44 configurations drawn from 30 base models and 12 model makers, including six hosted APIs. Open models were tested across NVIDIA H200, NVIDIA B300 and AMD Instinct MI300X systems using two serving engines. Reasoning and non-reasoning versions of the same weights appear as separate configurations. Scores are indexed to a fixed Gemma-4-31B-it reference configuration that equals 1,000. That makes the roster easier to compare over time, but the completion percentages published alongside the scores are the more intuitive reading. A score ratio is not an accuracy ratio, and PINNACLE does not claim that it is. The 28-Point Messy-Data Penalty The first finding is the one every enterprise AI buyer should sit with. PINNACLE runs each job twice: once against organized, governed data and once against data that includes duplication, partial migrations and contradictions. Forty-three of the 44 launch configurations lost ground in the second condition. The median configuration surrendered about 28 percentage points of whole-workflow completion, and the widest decline reached 64 points. Nine configurations completed at least 95% of the governed-data jobs. Only two cleared that threshold when the data was presented as found: Claude Opus 5 at 98.3% and GPT-5.6 Sol at 95.8% in Signal65’s testing. The obvious conclusion is that data quality matters. The more useful conclusion is that data sensitivity is itself a model property. A buyer can compare the same model on both sides of the line. A large drop with a high clean-data score suggests a scope that may become viable after data preparation. A low score in both conditions points to a capability problem that cleanup alone will not solve. That is information no single-condition leaderboard can provide. It also explains why a model that appears nearly perfect in a controlled evaluation may struggle when attached to the operating history of a real company. Nobody Buys Tokens PINNACLE’s second major contribution is cost per correct task. Hosted models are charged their full API bill from the runs that produced the score, and the tokens consumed by failed attempts are allocated across the tasks that finished. Open models are priced from the node hour, with utilization exposed as a variable rather than assumed away. The resulting economics look different from the model price sheet. In Signal65’s testing, input represented 65% to 91% of the hosted-model invoice because an agent repeatedly reads a growing context as it works. Pricing agentic work from output tokens alone understated the cost of a correct task by three to 11 times. A correct task using GPT-5.6 Sol cost $1.21 in the measured runs, not the dime that simple sticker-price math implies. The best midsize open models on an eight-GPU B300 node delivered a correct task for roughly eight to 15 cents and remained cheaper than the frontier APIs down to approximately one-quarter node utilization under Signal65’s $63-per-hour leased-node assumption. That does not settle the build-versus-buy question. Staff, platform engineering, security, power, monitoring and integration still belong in enterprise TCO. It does turn the argument into a measurable crossover point rather than a debate about whose price sheet looks friendlier. The Model Results Are Uncomfortable Claude Opus 5 led the overall PINNACLE launch table with a score of 4,558, followed by GPT-5.6 Sol and Claude Sonnet 5. The best open-weight configuration, GLM-5.2 with reasoning enabled, ranked fourth at 3,255. DeepSeek-V4-Flash, Kimi-K3 and Qwen3.8-27B followed among the leading open configurations. That produced a result guaranteed to draw attention: six of the launch roster’s top 10 configurations were Chinese open-weight models, while the other four were American hosted models. The best American open-weight entry sat at the 1,000-point baseline, which translated to 54.3% job completion versus 91.1% for GLM-5.2. Signal65 appropriately bounds that finding to this roster, this date and this category of enterprise agentic work. More American open models, including NVIDIA’s Nemotron family, are queued for testing. The fabrication results challenge another assumption. When PINNACLE asked questions whose answers were absent from the generated documents, Claude Opus 5 and GPT-5.6 Sol invented answers 7.6% and 10.3% of the time, respectively. Several leading open configurations landed between 0.4% and 1.9%. That does not mean open models hallucinate less in every context. It means the best open configurations in this suite were more willing to refuse when the documents did not contain the answer, enough for GLM-5.2 to lead PINNACLE’s Customer Operations persona. Then there is Llama. Under PINNACLE’s strict whole-job rule, the two tested Meta configurations completed no entire job correctly. They did produce correct intermediate deliverables, at 10.3% and 7.4%, so the harness clearly executed them. The zero is not a claim that Llama cannot do useful work. It is a claim that these configurations did not finish this work under these rules. For enterprises that built their first open-model programs around Llama, that is a result worth investigating rather than dismissing. The Silicon Result Is Capacity, Not Intelligence PINNACLE separates model quality from platform capacity. The same model produced the same quality score across the NVIDIA and AMD platforms Signal65 tested, within run-to-run noise. The box changed how much work could be sustained, not how smart the model became. On that capacity measure, an eight-GPU NVIDIA B300 node supported 3.15 times as many concurrent agents as an H200 node across seven tested models, with gains ranging from 1.86 times to six times. On the largest model in the comparison, sustained concurrency rose from 16 agents to 96. A B300 node sustained 320 concurrent agents on Qwen3.5-35B-A3B while maintaining the benchmark’s service gates. An AMD Instinct MI300X node sustained 128 concurrent agents on a 9B model and was able to run a 397B model because the system provided 1.5TB of high-bandwidth memory. AMD’s MI355X results are not part of the launch set and will follow after the system is characterized. There is no NVIDIA-versus-AMD capacity chart at launch. Signal65 is publishing within-vendor generation comparisons until each vendor has supplied the configuration it would recommend for the workload. That is the right restraint. A cross-vendor number that mixes silicon differences with the benchmark operator’s tuning choices would be more dramatic and less useful. A Benchmark Should Carry Its Boundary PINNACLE is ambitious, but it does not eliminate the need for skepticism. The environments are synthetic rather than copies of real enterprise data. That provides a known answer key and makes contamination much harder, but it cannot reproduce every organizational dependency or human judgment. The launch suite contains 280 agentic samples per configuration, with environment-score resolution stated at roughly five points. Small differences between neighboring models should not be read as definitive separation. The benchmark harness is not open for independent reproduction today. Signal65 publishes its methodology, governance framework, platform configurations and full concurrency ladders, including the first failing rung, but deterministic scoring is not the same as anyone being able to rerun the entire suite. That remains a legitimate question for a benchmark that wants to become an industry instrument. The vendor relationship deserves equal clarity. NVIDIA and AMD supplied hardware, engineering time and feedback. Signal65 says publication is unconditional, vendors receive a factual-accuracy review window without approval or veto rights, and every platform result carries its configuration and service gates. Those are meaningful safeguards. They do not make the commercial relationships disappear, which is why the governance and disclosures belong beside the numbers. PINNACLE also makes no claim to measure general intelligence, predict every on-the-job outcome or cover entire occupations. That may sound like boilerplate. It is actually the most important rule for reading the table. A benchmark becomes misleading when its boundary gets separated from its ranking. The Hard ROI Era Needs a Different Scoreboard PINNACLE will attract attention because of who won, who lost and where Chinese open-weight models landed. Its more durable contribution is the decision to connect correctness, capacity and cost in the same instrument. The enterprise does not buy tokens for their own sake. It does not buy benchmark points or theoretical context windows. It buys resolved incidents, reconciled records, completed analyses, answered customer questions and other finished work. Every failed attempt consumes budget. Every fabricated answer creates review or risk. Every model that collapses on messy data transfers work back to people and systems downstream. That is why the next generation of AI evaluation cannot end with, ‘Which model is smartest?’ The useful questions are narrower and harder: Which model completes this job? What happens when it meets our data? How many copies of the workflow can the platform sustain? Where is the build-versus-API crossover? And what does one correct result cost after the failures are counted? Conventional leaderboards remain useful within their boundaries. The enterprise mistake is treating them as evidence of production readiness. PINNACLE is an attempt to build a scoreboard for the work enterprises actually need to finish. Explore PINNACLE The live leaderboard, methodology and governance materials are available at pinnacle.signal65.com https://pinnacle.signal65.com . Disclosure: Techstrong.ai and Signal65 are both part of Futurum Group. Signal65 developed PINNACLE in partnership with Kamiwaza. NVIDIA and AMD provided hardware, engineering input and support under Signal65’s published governance rules; vendors have no approval or veto rights over published results.