As AI agents become more capable and their use expands across cybersecurity and other domains, the need for effective evaluation grows. An agent investigating a codebase makes a sequence of decisions: where to look, which tools to use, how to interpret the evidence, and when to submit a finding. Evaluating that work requires clear task definitions, consistent execution conditions, and a way to examine both the process and the outcome.
We’re introducing Agentic FAITH, an extension of Cisco Foundation AI’s FAITH evaluation framework. It brings tasks that require multiple steps and tool use into the same framework teams already use for question-answering benchmarks. Researchers and engineers can now use one workflow to run cybersecurity evaluations, grade outcomes, and analyze results across both kinds of tasks.
One workflow for cybersecurity evaluations #
FAITH already brings cybersecurity benchmarks for threat intelligence, vulnerability classification, and security knowledge into a common evaluation workflow. Agentic support extends that workflow to tasks such as exploring a repository with terminal tools, giving teams one framework for running and analyzing both traditional and agentic evaluations.
The workflow stays familiar: query runs the benchmark and collects records, eval grades the outcomes, and summarize reports the results. The run-all command combines these stages. Teams using FAITH can run and analyze their new agentic benchmarks through these same commands.
For agentic tasks, FAITH manages the interaction between the model and its tools. The model chooses an action, receives the tool’s output, and uses that result to decide what to do next. A run continues until the agent submits an answer or reaches a stopping condition defined by the benchmark. Figure 2. For terminal tools, FAITH dispatches the call to the configured environment and returns the result to the model. It records these interactions for later inspection.
See it in action with VulnLoc #
The first integrated agentic benchmark is VulnLoc, which evaluates how well agents locate vulnerable files in a repository and avoid false alarms on patched code.
Each task gives the agent a repository snapshot and a description of a vulnerability class. The agent explores the code through terminal tools, searches for relevant logic, and follows what it finds. It then submits the files it suspects contain the vulnerability or explicitly reports that it found no matching vulnerability.
VulnLoc can run locally with Docker for trusted inputs or on Kubernetes through Agent Sandbox. The execution environment enforces read-only access to the repository.
FAITH integrates both phases of the benchmark:
- Phase A: find the affected files. The agent inspects the repository before a security fix. File-set F1 measures how well its submitted files match the benchmark’s ground truth, accounting for both missed files and incorrect selections.
- Phase B: recognize the fix. The agent inspects the corresponding patched repository and reports that the specified vulnerability is absent. The primary metric is true-negative rate.
Together, these phases evaluate the ability to find vulnerabilities and avoid false alarms on patched code. Completion status is recorded so a correct report of no matching vulnerability can be distinguished from a missing submission or an execution failure.
Figure 3. File paths and submissions are illustrative.
Build evaluations around your tasks #
VulnLoc is the first integration in an extensible benchmark architecture. Benchmark authors can define how tasks are loaded, which tools an agent can use, where those tools execute, and how submissions are extracted and scored. They can reuse existing components or implement and register new ones in FAITH.
Benchmarks also configure prompts, turn limits, tool-call budgets, and timeouts. Making these conditions explicit helps teams understand the setup behind a result and evaluate models under a consistent set of constraints.
Deployment configuration can override the benchmark’s default environment, letting teams adapt the execution setup while keeping the task definition intact.
Understand each run and keep your progress #
FAITH saves the conversation history, tool calls and responses, submitted answers, execution errors, and the reason each task ended. If an agent misses a vulnerable file, researchers can examine whether it inspected that code, what information it received, and whether it exhausted its execution budget. These records help teams investigate failures and identify changes to explore in subsequent evaluations.
Long evaluations also benefit from completed-task checkpointing. FAITH saves each completed task before starting the next one. When that storage remains available, an interrupted evaluation can reuse eligible completed results. Work resumes at task boundaries, reducing the need to repeat completed tasks after an interruption.
Get started with Agentic FAITH #
The new FAITH technical documentation brings together installation instructions, an agentic quickstart, model configuration, deployment guidance, and instructions for creating benchmarks.
Start with a small VulnLoc evaluation. Choose a model and endpoint that support structured tool calls through the OpenAI-compatible Chat Completions interface, configure the Docker environment, and run a task. Agentic FAITH supports hosted endpoints and appropriately configured models served through a compatible API. Inspect the saved execution record alongside the score before expanding the evaluation.
Get started with the Agentic FAITH quickstart, explore the code on GitHub, and contribute the benchmarks you want to evaluate next. We’re building a common framework for the growing range of ways AI systems are used in cybersecurity, and we welcome benchmarks that help the community measure those capabilities.