{"slug": "designing-ai-evals-clarity-now-and-visualization-next", "title": "Designing AI Evals: Clarity Now and Visualization Next", "summary": "A developer from Google Cloud's DevRel team demonstrates how to design objective evaluations for AI agent skills using open-source frameworks like Inspect AI and Harbor. The investigation uses Gemini models as solvers and a previous-generation model as a grader to assess skill performance, with plans to extend evaluations using visualizations in Google Sheets and Data Studio. The series aims to help developers decide whether a skill is worth adopting by providing reproducible benchmark scripts and a decoupled grader architecture.", "body_md": "Let's say you're testing out new AI tools. Perhaps you implement and run analytics for an Ad Agency and hope to automate deploying your standard event schema, or are a podcast producer automating generating social copy from your newest ep. While modern, newly trained LLMs can likely one-shot a lot of these tasks – this specificity might necessitate wasting tokens and time repeatedly prompting them with the same resources, descriptions and scripts. With this in mind, you investigate tooling whether that be an MCP server, an agent skill or an [agent plugin](https://agent-plugins.org/specification).\n\nThe problem is, how do you know if a skill (whether developed by you or open sourced by someone else) is worth your time or, perhaps more importantly, your tokens and quota? How can one go about designing these (more) objective evaluations of AI tools and, from there, collecting and analyzing relevant metrics? That is exactly the context for using open source eval frameworks, like [Inspect AI](https://inspect.aisi.org.uk/) and [Harbor](https://github.com/harbor-framework/harbor) to [evaluate agent skills using open source frameworks](https://codelabs.developers.google.com/codelabs/evaluate-agent-skills-using-open-source-frameworks). But how do you [\"extend the evaluation\"](https://codelabs.developers.google.com/codelabs/evaluate-agent-skills-using-open-source-frameworks#4), use visualizations to spot trends and collaboratively explore alternative paths forward using Google Sheets and Data Studio?\n\nThese questions and more are exactly what I’m hoping to demonstrate for you in this series!\n\nWhile you are more than welcome to simply read about how and why I conducted my own investigation, you can alternatively follow my lead and [run the benchmark scripts](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#reproduction-guide).\n\nSo, if you’re hoping to play along, take a moment to [complete the aforementioned codelab](https://codelabs.developers.google.com/codelabs/evaluate-agent-skills-using-open-source-frameworks) and come back when you’re done. Don’t worry, we’ll still be here when you get back!\n\nNote: This blog series contains AI-generated diagrams\n\nalongside actual screenshots and hand-drawn edits of\n\nboth. AI also assisted in minor copy editing.\n\nIn the codelab, we learned how to run evals with Gemini CLI, Inspect and Inspect SWE in an isolated Docker Sandbox to understand how well each skill aids the agent in answering the same question.\n\nFor those playing along at home, please additionally install `inspect view`\n\nnow before you need it to run the evals below; be prepared to have to wait for a couple minutes for evals to finish depending on your machine and quota usage.\n\nRegarding our new [source files](https://github.com/GoogleCloudPlatform/devrel-demos/tree/main/agents/inspect-agent-skills-eval#downloading-the-demo-giget) while they are heavily commented and hopefully written in a self describing way, I’ll explain further subsequently.\n\nFor the purpose of my investigation demo, I used three different models: `google/gemini-3.5-flash-lite`\n\nand `google/gemini-3.6-flash`\n\nas \"solvers\" (the models under evaluation), and `google/gemini-3.1-flash-lite`\n\nas a \"grader\" (the model rating the runs). The three of them (and models more broadly) differ in many ways but more specifically on [problem solving ability, speed and cost](https://ai.google.dev/gemini-api/docs/models).\n\nWe intentionally offloaded grading for this demo to a previous-generation model because by changing rubric criteria to strict binary decisions and applying a reduction programmatically, it delivers sufficiently robust evaluations without burning through solver quota. For a *production* evaluation system, consider investigating using newer and more capable models as graders because they will likely have narrower confidence intervals.\n\nFor a full technical breakdown of this decoupled grader architecture and to find out how you could substitute in your own choice of models, see the README section on [Decoupled Grader & Multidimensional Rubrics](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#3-decoupled-grader--multidimensional-rubrics).\n\nReproducibility & Local Setup Note: If you are following along locally, clone the domain skill definitions into`google-skills/`\n\nbefore running benchmark sweeps:\n\n```\ngit clone https://github.com/google/skills.git google-skills\n```\n\nAll evaluations in this series were benchmarked on\n\nPython 3.13using`inspect-ai`\n\n(`v0.3.247`\n\n),`inspect-swe`\n\n(`v0.2.66`\n\n),`inspect-viz`\n\n(`v0.4.1`\n\n), and`pandas`\n\n(`v3.0.3`\n\n). If upstream PyPI releases introduce breaking changes, check the README's[Environment & Dependencies specification]for exact version pins and instructions on how to reproduce the configuration.\n\nFor this investigation I used a [new eval script](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/skills-eval.py); while the specifics of the script are important for anyone who hopes to make their own evals or run the code as we go, if you're primarily interested in analysis and visualization, feel free to skip to the next section.\n\nThe original script ran a small batch of tests on a local machine. While this version can be and was run on a local machine, it prioritizes three architectural dimensions needed for running automated tests as part of a larger scale development flow:\n\nFor a complete technical breakdown of these architectural pillars—including external configuration schemas (`questions.json`\n\n, `thrifty_system_prompt.txt`\n\n), solver rate-limiting defenses (`version=\"0.51.0\"`\n\npinning), grader quota decoupling, and score curving math formulas see the README's [Evaluation Pipeline Architecture & Technical Reference](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#1-external-configurations).\n\nThese commands create a matrix of evals subject to model x skill condition x sample x epoch. For a quick visual (that I’ll repeatedly reference throughout this), take a look below.\n\n**Caption**: Visualization of the eval matrix that will be scored and compared. In Inspect AI, rubric elements, represented here as Facts, are graded Correct (C) or Incorrect (I).\n\nWith all that said, it’s finally time for the rubber to meet the road. I ran our evals with the following command\n\n```\ninspect eval skills-eval.py \\\n  --model google/gemini-3.5-flash-lite,google/gemini-3.6-flash \\\n  --time-limit 300 \\\n  --epochs 2 \\\n  --max-tasks 4 \\\n  -T web_access=false\n```\n\nThis command runs a 4-way parallel sweep across models and skill conditions while enforcing a 300-second task timeout and disabling web search to minimize token spend. See the README for a complete [CLI Parameter Reference](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#cli-parameter-reference) and details on [External Configuration Template Mechanics](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#1-external-configurations).\n\nIf you intend to and haven't done so already, **run the command line**. If you're interested in how to read over and interact with the Terminal User Interface, please refer to the [previous codelab](https://codelabs.developers.google.com/codelabs/evaluate-agent-skills-using-open-source-frameworks).\n\nWatch the running command line long enough to determine it hasn’t crashed or hit an obvious error and then maybe take a moment to make some food, grab some water or take a walk around the block. Enjoy the small things in life, okay? Worst case, you get a 20 minute walk out of it, not so bad eh!?\n\nWith that said and done, reading the terminal output is only the most basic way of understanding the evals. Now that you have the raw eval logs, what do you need to do to begin to use these analytically and, more importantly, persuasively?\n\nTo start off, I simply ran\n\n```\ninspect view\n```\n\nand clicked the link to open the browser to the GUI.\n\nFor step-by-step diagnostic trace analysis in the browser GUI, see the README section on [Local Diagnostic Trace Analysis (inspect view)](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#local-diagnostic-trace-analysis-inspect-view).\n\nMine came out like this (after a little bit of column drag-and-drop):\n\nWhile we covered the rudiments of looking over the inspect view GUI in [Evaluate agent skills using open source frameworks](https://codelabs.developers.google.com/codelabs/evaluate-agent-skills-using-open-source-frameworks), there is significantly more that can be done with it. Speaking of which, the **TASK ARGS** column (not pictured) serves as a quick reference for what arguments were passed to a task.\n\nWhile LLM’s are inherently stochastic (and given floating point hardware, nondeterministic), when running evals we are trying to use sampling to characterize the average observable metrics (e.g. correctness, latency and token usage) which our particular configuration model x skill scores on relevant questions. As such, the eval tasks were set up to characterize how a change to an independent variable (model or skill) affect dependent variables (the aforementioned metrics).\n\nGrouping evaluation runs by model in `inspect view`\n\nallows for a direct \"eyeball\" inspection of how skill inclusion alters accuracy across baseline controls—**given a specific model, what is the impact of adding a skill?**\n\nI have added some annotations to make metric change from baseline to skilled a little more apparent:\n\nThe above shows that\n\nAnother simple measurement that can be done is to look at the `TOKENS`\n\ncolumn where gemini-api actually led to a decrease in usage relative to its baseline in `3.6-flash`\n\n.\n\nWhile the general relationship between tokens and score is difficult to parse visually, one notable result: for 3 out of 4 task x model configurations, gemini-3.6-flash used more tokens than the comparable 3.5-flash-lite run.\n\nNow to analyze the inverse; in more concrete terms: **given a particular skill, what is the effect of changing the underlying model?**\n\nWhat was the effect of the change from `3.5-flash-lite`\n\nto `3.6-flash`\n\n?\n\nFor 3 out of 4 of these, the changes between the two models amounted to an up to 45% increase in accuracy. For whatever reason, however, changing the `gcloud`\n\nskill from using one model to another led to a slight decrease.\n\nWhile these are both valuable findings, they warrant further questions:\n\nFor the first of those, if you want to make more representative samples, know that you should conduct further research using more questions, samples and epochs and compare those metrics with these. For the sake of this blog series though, I’ll leave that for you to do.\n\nFor the second though, we can and will endeavor to do so. That said, it might help in designing follow up analysis to think a little bit about common patterns encountered during evals. This is especially true for those running evals themselves (possibly on different skills or rubrics); you are likely to encounter very different metrics than mine, and thus I’ll lay out some common patterns and follow up actions to investigate them further.\n\nWhen analyzing evaluation runs, comparing skilled execution against baseline controls typically maps to five distinct diagnostic outcomes—ranging from **High-Efficiency Capability Lift** (best) to **Context Overload & Skill Regression** (worst).\n\nFor a complete breakdown of this diagnostic taxonomy and actionable audit steps for each outcome, see the README breakdown of [Diagnostic Mental Models for Eval Comparison](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#diagnostic-mental-models-for-eval-comparison-skill-vs-baseline).\n\nWhile high-level metric summaries alert you to outcomes like cost bloat or skill regression, opening an individual sample surfaces the **Sample Details** panel for granular trace analysis:\n\nThis tab shows the exact multi-turn conversation between the solver agent, the sandbox shell, and external tools. Use it to diagnose model reasoning versus environment noise:\n\n`-T web_access=false`\n\n) and automated time limits (`300 seconds`\n\n) were correctly injected into the container environment.Here is an example of one case where the skill was activated.\n\n**This means:** The agent actually ingested `gemini-api`\n\ninstead of relying on baseline pre-training memory.\n\nOf note though, some of the tasks where skills were made available did NOT activate the skill. If you’re a skill author, you may want to rewrite the “activation criteria” (aka what situation calls for using the skill) such that it's more applicable to the specifics of relevant tasks.\n\nAuditing transcript details allows you to distinguish model reasoning loops (e.g., repeated redundant tool calls) from sandbox environment noise (e.g., container timeouts or missing binary dependencies). For step-by-step diagnostic trace auditing procedures, see the README reference on [Sandbox Noise vs. Model Reasoning](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#sandbox-noise-vs-model-reasoning).\n\nClicking the **Scoring Tab** displays the empirical breakdown of our custom `multi_scorer`\n\non a sample:\n\nThe `multi_scorer`\n\naggregates the results of checking the sample’s answer against individual binary yes/no Facts using `model_graded_qa`\n\n. The generated list of scores in the range [0.0, 1.0] is supplied to our `custom_reducer`\n\nwhich calculates their arithmetic mean. From there, it applies quadratic score curving (\nmean2\n) to ensure that the further from correct the mean of the supplied answers are, the lower the score is pulled. This allows the most correct answers to stand out immediately. In the **Scoring Tab**, this downward curve maps a raw 5/6 fact score (\n≈0.8333\n) down to a normalized 0.65 sample score. For complete formulas and math derivations, see the README section on [Atomic Fact Verification & Quadratic Curving](https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/README.md#3-decoupled-grader--multidimensional-rubrics).\n\nWhile `inspect view`\n\nprovides outstanding deep-dive diagnostics for individual sample traces, evaluating dozens of models across multiple skill domains gets confusing fast (as you may have seen above). Weighing the correlations between skill inclusion vs exclusion and model changes against each other can be confounding and more art than science if we don’t find better ways to reason over them; it requires a structured matrix overview.\n\nRemember that mess of sectors of a 3d space I showed you earlier representing all the configs? Well, unless we find numeric ways to collapse that or quantify deeper comparison, you won’t get the granular information you need to choose between two similarly capable alternatives. Worse yet, you won’t be able to communicate this to the people with their hands at the purse strings (unless of course, that's you too).\n\nTo make this easier, next time we’ll start down the road of understanding and displaying these things visually: As visualized in the cohort matrix linked above, understanding agent capability factor-by-factor requires cohort slicing across multi-dimensional metrics.\n\nIn **Part 2**, we scale our analysis from single log UI inspection to aggregate scoreboards with `inspect viz`\n\n!", "url": "https://wpnews.pro/news/designing-ai-evals-clarity-now-and-visualization-next", "canonical_source": "https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii", "published_at": "2026-08-18 07:00:00+00:00", "updated_at": "2026-08-18 07:12:18.140904+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-agents", "large-language-models", "developer-tools"], "entities": ["Google Cloud", "Inspect AI", "Harbor", "Gemini", "Google Sheets", "Data Studio", "Docker", "Gemini CLI"], "alternates": {"html": "https://wpnews.pro/news/designing-ai-evals-clarity-now-and-visualization-next", "markdown": "https://wpnews.pro/news/designing-ai-evals-clarity-now-and-visualization-next.md", "text": "https://wpnews.pro/news/designing-ai-evals-clarity-now-and-visualization-next.txt", "jsonld": "https://wpnews.pro/news/designing-ai-evals-clarity-now-and-visualization-next.jsonld"}}