Beyond overload: Yubico’s measured approach to AI adoption

Yubico, a security key maker, reported that approximately 46% of AI-flagged security findings were downgraded during human triage, most commonly from High to Medium severity, based on agentic assessments of 29 open source repositories using Anthropic's Claude Code Security and OpenAI's Codex Security. The company adopted a measured approach to AI, using an approximately 1000-line prompt with multi-stage orchestration to improve accuracy and control, and highlighted that models frequently overstated severity.

Earlier this year, I shared my perspective on “ AI Overload https://www.yubico.com/blog/ai-overload-a-cisos-perspective/ ,” voicing the apprehension felt by many CISOs regarding the flood of non-useful content, increased social engineering, and the massive spike in the threat landscape without a commensurate amount of immediate value. The tension between supporting business demand for AI and protecting the business remains a unique challenge for risk management organizations. We cannot afford to be spectators; we must navigate the hype with clarity and rigor. With this in mind, I am sharing some of Yubico’s journey navigating the AI landscape with the hope that there is something to be passed along to others in the industry who may be dealing with similar challenges. The company’s AI strategy is being driven by the twin goals of accelerating innovation and improving efficiency. AI adoption offers the promise of cost savings through direct labor substitution e.g., replacing support agents with chatbots , process acceleration, and cost avoidance e.g., catching software bugs prior to release . The adoption of AI across Yubico includes a gradient of use cases across the company, but the more obvious use case is in the secure development lifecycle for both development and analysis. Our efforts to integrate AI were accelerated in response to the announcement of Anthropic’s Project Glasswing https://www.anthropic.com/glasswing and OpenAI’s Aardvark https://openai.com/index/introducing-aardvark/ programs to minimize unwanted surprises. In this post, I won’t touch on vibe coding or democratized software development, as I think it deserves its own focus. Inside Yubico’s approach to AI adoption We prioritized the review of our open source software due to its accessibility and to minimize the impact of potential mistakes as we gained comfort with the process and tooling. The assessment included holistic code analysis of 29 repositories using Anthropic’s Claude Code Security and OpenAI’s Codex Security. What we learned about prompting is that models benefit from explicit direction and narrow goals. We started experimentation with orchestrated multi-stage prompts to have better: Accuracy : Fewer mistakes because each step is narrower. Control : One can inspect or validate intermediate outputs. Complex reasoning : Useful for analysis, planning, coding, legal review, data extraction, etc. Consistency : Each stage can have its own rules or format. We settled on an approximately 1000 line prompt that includes consuming a threat model, enumeration of weakness patterns, context about the software’s privilege level, and specific direction on the type of threats we wanted it to evaluate for. We then used a main agent responsible for initial finding review with subagents focused on individualized assessment tasks to mitigate against shallow findings and task drift. Anthropic’s blog https://claude.com/blog/using-llms-to-secure-source-code , which was posted after our initial evaluation, aligns closely with our experience and recommendations on how to incorporate AI into the security assessment process. Agentic assessments were conducted by two security engineers, both operating independently on each repository which resulted in at least two independent evaluations for them. Observations and findings across the two runs were consolidated and AI assisted triage. The LLM was asked to present the findings, reference relevant source code, and reason about the observations and impact. These were interactive sessions with the security engineers who attempted to use code flow, vulnerability exposure, and attack chains during triage. On numerous occasions, we hit guardrails that were intended to prevent adversarial testing but they significantly impeded the triage process. Human triage found that the models frequently overstated severity. Approximately 46% of the cases were downgraded, most commonly from High to Medium. After triage, the findings were categorized by weakness type to support broader trend analysis. The assessment ultimately produced 448 verified findings and only a handful of false positives. All findings relate to security hygiene and defense-in-depth improvements, and they were distributed relatively evenly across the programming languages assessed. The models were less effective at identifying complex or higher-impact vulnerabilities. As a control test, we evaluated them against a codebase containing a known flaw in its WebAuthn implementation, later published as YSA-2026-02. Even after we refined the prompts and adjusted the supplied context, the models did not identify the issue consistently. They also frequently refused requests involving the analysis of multi-step attack chains or the validation of how several lower-severity weaknesses might be combined into a higher-impact exploit. At the time, these assessments used general-purpose models operating under their default cybersecurity safeguards. The assessment took four weeks and incurred approximately $3,000 in direct model-usage costs. Given the breadth and depth of the review, we considered both the product-security team’s investment and the direct model costs reasonable. Since completing the initial assessment, Yubico has enrolled in OpenAI’s Trusted Access for Cyber program and incorporated Codex Security and Claude Code into its secure development lifecycle. These changes have reduced inappropriate refusals during authorized defensive validation and accelerated the triage process. The introduction of AI into our assessment and validation process has allowed the security team to perform more comprehensive assessments that we expect will lead to an increase in code quality and improvements in defensive programming. Presently, we lack the confidence required to reduce our manual effort or to reduce our investments in other infrastructure. However, it does allow us to push skills and automation closer to the developer, whether classically trained or otherwise. We continue to evaluate our approach as we expect the tools and models will continue to rapidly improve.