A Framework-Agnostic Testing Methodology for AI Agents (61 sources, 58 test blocks, OWASP Agentic Top 10)

A developer has open-sourced a framework-agnostic testing methodology for AI agents, including a 61-source benchmark map and 58 universal test blocks across seven tiers, with full coverage of the OWASP Agentic Top 10. The methodology includes evaluation techniques such as LLM-as-Judge, pass@k, and automated red-teaming, and aligns with regulatory frameworks like NIST AI RMF and the EU AI Act. A real-world case study using PheronAgent demonstrates its application.

How do you actually test an AI agent? Not "does it respond," but: does it route to the right tool, chain calls correctly, recover from failure, resist prompt injection, and stay within cost/latency budget? I spent weeks working through this on a running agent, and open-sourced the entire methodology — framework-agnostic , so it applies regardless of your language, runtime, or toolset. • 61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena, AgentDojo, LongMemEval and more, categorized by what they actually measure • 58 universal test blocks across 7 tiers L1–L4, Error Recovery, Multi-Turn, Security . Each block = a tool-agnostic capability definition + a concrete reference implementation • Full OWASP Top 10 for Agentic Applications 2026 ASI01–ASI10 mapped to 6 universal security test blocks • Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k, trajectory vs end-state, observability OpenTelemetry GenAI , automated red-teaming garak, PyRIT, DeepTeam • Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001 Take Part II, replace the reference-implementation fields with your own agent's tool names and expected outputs. The universal capability definitions need no changes. Blank templates are included. PheronAgent a macOS agent with 50+ native/MCP tools is included as a real reference case study — but the methodology is the product, not the agent. No marketing narrative: STORY.md documents the real bugs, real test runs, and real corrections that shaped each version. Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome.