# Nvidia critiques skill evaluation methods in new ACES paper > Source: > Published: 2026-08-25 16:38:45+00:00 Via nvidia.com # Nvidia critiques skill evaluation methods in new ACES paper The chipmaker's new framework measures what AI agents actually do at runtime, not just what their code looks like on paper Nvidia just told the AI industry that its standard way of grading agent skills is basically checking homework without reading the answers. The company’s new research paper, published on August 20, introduces a framework called Agentic Continuous Evaluation of Skills, or ACES, that replaces static code checks with live head-to-head trials to measure whether a skill actually makes an AI agent better at its job. The core finding is damning for existing methods. The correlation between traditional scan-only metrics and LLM-judge scores came in at a Spearman rho of just 0.14. For context, a perfect correlation would be 1.0. A coin flip would be closer to zero. The current industry standard for evaluating agent skills lands uncomfortably near the coin flip end of that spectrum. ## How ACES actually works The framework’s approach is deceptively simple in concept. Run an AI agent on a task with a given skill enabled, then run the same agent on the same task without that skill. Compare the results. The difference is what Nvidia calls “Skill Lift,” a quantitative measure of whether a skill actually moves the needle. In practice, the execution is considerably more involved. ACES conducts paired live-agent trials that keep support skills and decoy skills consistent between the two runs, isolating the variable being tested. It uses something called the Agent Trajectory Interchange Format (ATIF) to normalize performance trajectories across different agent architectures, making apples-to-apples comparisons possible even when the underlying agents differ. Each evaluation generates scores across six metrics: security, skill execution, skill efficiency, accuracy, goal accuracy, and behavior check. Traditional evaluation methods, which lean on structural checks, script linting, and LLM-as-Judge rubrics, tend to catch only a subset of these. The runtime-sensitive ones, like whether an agent actually executes a skill correctly under real conditions, slip through the cracks entirely. Nvidia tested ACES across 947 paired cases spanning 58 of 64 production skills. The mean composite Skill Lift landed at 0.2134, with 72.8% of cases showing a positive lift. The biggest performance gains showed up in skill execution, behavior check, and skill efficiency, precisely the metrics that static analysis tends to miss. Outcome-only lift, measuring just accuracy and goal accuracy, averaged a lower 0.1799. The gap between that number and the composite score suggests that process-oriented metrics capture meaningful value that pure outcome measurement overlooks. ## Why static evaluation falls short The AI agent ecosystem has borrowed heavily from traditional software development when it comes to quality assurance. Repository review processes typically prioritize static validation: does the code lint cleanly, does the structure follow the template, does an LLM judge think the skill description sounds reasonable. These checks have their place, but they fundamentally cannot answer the question that matters most. Does the skill make the agent better at doing things? Nvidia’s data makes this gap concrete. That Spearman rho of 0.14 means organizations relying solely on static scans and LLM-judge rubrics are operating with evaluation tools that have almost no predictive power over actual runtime performance. Skills that pass every static check can still fail to improve agent behavior. Skills that look rough around the edges in a code review might deliver meaningful performance gains in practice. ## The enterprise AI implications Nvidia released its SkillEvaluator as an open-source implementation of the ACES framework, specifically its Tier 3 live evaluation component. The tool integrates directly into Nvidia’s Verified Agent Skills pipeline, which serves as a quality gate for skills entering the company’s enterprise AI ecosystem. Reusable skills in the agent context are dynamic packages of instructions and references that agents can access on demand, essentially plug-and-play capabilities that extend what an agent can do without requiring a full retraining cycle. A companion technical blog and SkillEvaluator documentation were released alongside the paper, signaling that Nvidia views this not as a research curiosity but as production infrastructure. **Disclosure:** This article was edited by Editorial Team. For more information on how we create and review content, see our [Editorial Policy](https://cryptobriefing.com/editorial-policy/).