{"slug": "how-to-design-ai-evaluations-you-can-actually-trust", "title": "How to Design AI Evaluations You Can Actually Trust", "summary": "Google's Developer Relations team published a suite of Agent Skills for Google products on GitHub and outlined five rules for designing trustworthy AI evaluations. The rules emphasize understanding the evaluation framework, using isolated credentials, ensuring prompts and graders are complementary, testing diverse use cases, and avoiding overfitting. The team stresses that reliable evaluations are critical for improving AI tools and avoiding wasted token budgets.", "body_md": "As part of my work at Google, we are publishing a suite of [Agent Skills for Google products and technologies on GitHub](https://github.com/google/skills). These [agent skills](https://agentskills.io/home) are designed to help AI agents interact with our technologies. But how do you test that these skills are useful and work as expected? My team in Developer Relations has been focused on this question, because having reliable signals on their performance is critical to help us improve them over time.\n\nJust as you wouldn't deploy a production API without writing unit tests, you should apply the same standard to your AI agents. As Joe Spiro showed in the [ Designing AI Evals](https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii) post series, scaling AI tools means moving beyond \"vibe testing\" in a terminal. Instead, you should set up a structured, automated evaluation pipeline to benchmark your integration. The evaluations (evals) are the\n\nHowever, AI evaluations cost real tokens. You need to make sure that you use these tokens as efficiently as possible. They need to provide real value that helps you build better tools. Writing good evaluations is critical. Poor evaluations provide false signals, waste your token budget, and create noise in your metrics.\n\nHere are five rules we learned to design better evaluations you can trust. Follow them to ensure that every token you spend produces a useful metric.\n\nBefore writing evaluations, you need to understand the setup and limitations of your chosen framework. This includes systems like Harbor, Inspect AI, or integrations in development tools like in the [Agent Development Kit](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation). Does it use an ephemeral sandbox? What tools are available? How is the output captured\n\n`gcloud`\n\nsession with access to a Google Cloud project), create ephemeral resources or credentials that isolate and limit access so they don't impact other evaluations. Alternatively, you could provide mock tools instead of real test credentials.\nIf your evaluations show a high baseline accuracy (i.e., without your agent tool), it might not prove its value, or the evaluation prompts are too easy.\n\nYou cannot grade an agent on something you did not explicitly ask it to do. Your evaluation prompts and graders should be complementary. This means that they should only test for things included in the prompt.\n\nAgents possess inherent model knowledge and might skip your custom tools entirely to arrive at the correct answer. (That on its own is some useful feedback!)\n\n`help`\n\ncommand or followed a rigid sequence of steps.\nA strong evaluation suite tests real, diverse use cases. But testing the same capability repeatedly causes overfitting and creates noisy metrics.\n\nYou cannot improve AI tools if you can't measure them accurately. If you treat AI evaluations with the same focus as traditional unit tests, you improve the quality of your metrics and get more robust signals.\n\nBy applying these five rules, you eliminate false signals that waste your token budget. Instead of generating noise, your test suite gives you actionable feedback you can use to guide your engineering decisions and improve your tools.\n\nFiguring out what to test is only the first step. A well-designed evaluation is only useful if the scorer grading answers is reliable and returns meaningful results. In my next post, we will look at how to test. You will learn how to write lean, atomic rubrics that minimize ambiguity for an LLM grader and make every token count.\n\n*Photo by William Warby on Unsplash*", "url": "https://wpnews.pro/news/how-to-design-ai-evaluations-you-can-actually-trust", "canonical_source": "https://dev.to/googleai/how-to-design-ai-evaluations-you-can-actually-trust-41c3", "published_at": "2026-09-01 16:35:00+00:00", "updated_at": "2026-09-01 16:54:45.629985+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-research", "developer-tools"], "entities": ["Google", "Agent Skills", "GitHub", "Joe Spiro", "Harbor", "Inspect AI", "Agent Development Kit"], "alternates": {"html": "https://wpnews.pro/news/how-to-design-ai-evaluations-you-can-actually-trust", "markdown": "https://wpnews.pro/news/how-to-design-ai-evaluations-you-can-actually-trust.md", "text": "https://wpnews.pro/news/how-to-design-ai-evaluations-you-can-actually-trust.txt", "jsonld": "https://wpnews.pro/news/how-to-design-ai-evaluations-you-can-actually-trust.jsonld"}}