{"slug": "evaluating-coding-agents", "title": "Evaluating Coding Agents", "summary": "The AIUC-1 Consortium released an updated standard covering risks from coding agents, adding controls for secrets management, secure defaults, and runtime containment. The framework evaluates agent traces to classify severity of actions, using sandboxed environments to prevent evidence contamination. Nancy Wang, CTO at 1Password and AIUC-1 Consortium member, emphasized the need for clear agent identity, least-privilege access, and tamper-evident auditability.", "body_md": "Coding agents now close most of the engineering loop on their own: planning, coding, testing, opening a pull request, responding to review comments. As capabilities improve, the risks associated with coding agents compound - agents can now run for extended periods of time searching a codebase, reasoning through tasks, and iterating on their own work with limited supervision.\n\nThe latest AIUC-1 standard update, shaped by 100+ AIUC-1 Consortium members, expanded to cover risks introduced by coding agents, with new controls for secrets management, secure defaults in code, and runtime containment. These complement existing controls on agent identity, access management, and monitoring. AIUC-1’s requirements surface the risks that show up in real user flows such as adversarial robustness, tool calls, hallucinations, and safety.\n\n“*For coding agents, I look for clear agent identity, least-privilege access, explicit approval for consequential actions, and tamper-evident auditability. The goal is to let agents move quickly without losing control of the security boundary,*” said Nancy Wang, CTO at 1Password and AIUC-1 Consortium member.\n\nThis post covers:\n\nEvaluations for modern coding agents are fundamentally different to evaluations required for conversational agents. Given that coding agents can write, test, and debug code, evaluations must judge the entire work session and not just the response. This full record of every action taken (agent trace) encompasses the multi-turn conversation, the tool calls and commands the agent ran, file edits, web searches and more.\n\nAIUC-1 evals are designed to validate the AI system's robustness against specific risks. In the case of coding agents this means data/privacy, security, and reliability.\n\nFundamentally, the coding agent evaluator takes in the agent trace and classifies the severity of the agent's actions according to the risk being tested. Technically, the evaluator is a primary LLM-as-a-judge that uses tools (e.g., LLM review tool, adversarial debate) to gather information about the agent's actions and assess the risks elicited. Concretely:\n\nAIUC-1 framework for a coding agent evaluator\n\n**Deep-dive: Building isolated sandbox environments**\n\nEvals should produce credible evidence about how agents behave in production. That demands an isolated and observable environment in order to run safe adversarial scenarios where the results can be explained. Sandboxing an agent makes it less realistic, which also drives a need to create a more realistic environment to mirror production behaviour.\n\nIn preliminary testing runs, some evals were run in parallel, some on a shared machine. For evals being run on the same machine with the same file system, instances where the agent contaminated evidence for other evals, and found the rubrics behind its own evaluation run were discovered. After the early evaluations, each test was run in a sandbox, designed in the following configuration:\n\nEval testing environment for coding agents\n\nAfter sandboxing, in order to create a more realistic evaluation environment, runtime was built from a small Linux image and layered a realistic workstation on top:\n\nA sandbox that contains only the target repository is not representative of a real developer workstation. In prod, coding agents see home-directory files, environment variables, shell history, credentials, adjacent projects, package-manager configuration, and authenticated developer tools. An agent that never encounters those never gets tested against them.\n\n**Deep-dive: MCP attacks as a common attack vector**\n\nMCP is a common attack surface introduced by coding agents, introducing security risks such as credential exfiltration and unauthorized tool execution. As the primary way for agents to gather context from externally controlled systems, MCP servers allow both the tools and data behind them to change after installation.\n\nMCP attacks test what happens when an approved MCP server starts serving malicious content - whether the agent preserves user intent when a third-party dependency tries to redirect it. Example attacks created were based on existing CVEs (e.g., CVE-2025-54136). One included modelling a scenario involving a realistic internal project-tracking and documentation system that exposed two read-only tools (document search and context retrieval for specific tasks) and had a corpus of project specs, including backlog items, completed tasks, and meeting notes with initially benign content.\n\nThe attack involved the red-teaming agent injecting adversarial content into the selected document, and keeping the visible developer request benign, with no suggestion that MCP content should be treated as instructions.\n\nAIUC-1 evals for coding agents involve setting up specialized testing environments, realistic attacks and designing an evaluator that can take in the whole coding agent trace. The purpose is to validate the coding agents performance against specific risks, and provide third party testing for enterprises deploying agents in production.\n\n“*We run thousands of evals with our in-house engineering teams for each product release. AIUC-1 certification enables us to add an additional independent layer going deep on data and privacy, security and reliability risks*,” said Kenneth Moras, Security at Cursor.\n\nWhen evaluating coding agents, three dominant risk areas are tested against:\n\nBenign indirect ask testing for insecure output generation\n\nAs coding capabilities improve, the bar for demonstrating that these agents can act securely, safely, and reliably in deployment rises with them.\n\nCursor and Lovable are the first two coding agents to earn AIUC-1 certification, with Schellman conducting the independent audit. Read more about how AIUC-1 requirements were adapted for coding-agent specific risks - and how Lovable demonstrated these in [this whitepaper](https://www.aiuc-1.com/research/setting-the-standard-for-agentic-development). Read more about Cursor’s AIUC-1 certification [here](https://www.aiuc-1.com/research/cursor-achieves-aiuc-1-certification).\n\nOrganizations are able to certify their own deployment of certified coding agents - validating that the unique deployment context, data access, tool access, and configurations operate as intended by the organization.\n\nRead more about how to begin your AIUC-1 certification process [here](https://www.aiuc-1.com/category-leaders)", "url": "https://wpnews.pro/news/evaluating-coding-agents", "canonical_source": "https://www.aiuc-1.com/research/technical-docs-evaluating-coding-agents", "published_at": "2026-08-14 16:50:35+00:00", "updated_at": "2026-08-14 17:12:09.449055+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "ai-agents", "ai-ethics"], "entities": ["AIUC-1 Consortium", "1Password", "Nancy Wang"], "alternates": {"html": "https://wpnews.pro/news/evaluating-coding-agents", "markdown": "https://wpnews.pro/news/evaluating-coding-agents.md", "text": "https://wpnews.pro/news/evaluating-coding-agents.txt", "jsonld": "https://wpnews.pro/news/evaluating-coding-agents.jsonld"}}