{"slug": "five-rules-for-evaluating-agent-skills-before-they-bloat-your-context-window", "title": "Five rules for evaluating agent skills before they bloat your context window", "summary": "Google Cloud engineers published a five-rule playbook for evaluating SKILL.md agent skills, reporting that more than a third of benchmark tasks produced different scores across consecutive runs of the same code and prompt. The team ran dozens of scenarios with Gemini, Antigravity and other agents using Claude Sonnet 5 on the Gemini platform inside isolated Harbor pods on GKE Autopilot, and open-sourced skill-reach to catch frontmatter collisions before container sandbox spend. The playbook warns that a zero-improvement result (Δ = 0) usually means the agent never loaded the skill, since routers inspect only the YAML name and description, and that workstation benchmarks are contaminated by cached Google Cloud credentials and local environment state.", "body_md": "You’re probably hearing more about SKILL.md files, and you’ll see that a lot of people test them the same way: paste a prompt into chat, watch it pass once on a laptop, and merge to main. But when you paste three pages of CLI docs into a markdown file without an evaluation loop, you are just as likely to confuse a frontier model as you are to help it.\n\nFor the past weeks, we have been diligently measuring and optimizing our own [Google Cloud Developer Plugin](https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-google-cloud-developer-plugin-for-ai-coding-agents), with the most essential skills and the Developer Knowledge MCP server to ground responses in the documentation.\n\nWe [ran dozens of scenarios](https://x.com/kweinmeister/status/2099962972798153194) with Gemini, Antigravity, and other agents. We used **Claude Sonnet 5 on Gemini platform** inside isolated **Harbor** pods on **GKE Autopilot**. And, we contributed Kubernetes and Gemini platform capabilities upstream to NVIDIA's open-source evaluator. \n\nHere's the five-rule engineering playbook we use to verify a skill is worth its tokens.\n\nWhen an evaluation run shows zero improvement between your plugin arm and your baseline arm (Δ = 0), the natural urge is to rewrite the body of your SKILL.md instructions. Hold off on that. Nine times out of ten, **the agent never loaded your skill in the first place**.\n\nCoding agents do not read full SKILL.md bodies at turn zero. The router only inspects the short name and description in your YAML frontmatter. Once a workspace grows past a dozen skills, two failure modes collide: under-triggering (your description describes internal implementation rather than developer intent) and sibling hijacking (a broad phrase like \"helps with cloud deployments\" in a neighboring skill intercepts the prompt first).\n\nWe recently open-sourced [**skill-reach**](https://google.github.io/skill-reach/) so you can catch frontmatter collisions before spending on container sandboxes. It validates manifests, computes vocabulary overlap against nearest rival skills, and runs empirical routing probes across Claude Code, Google Antigravity, and OpenAI Codex:\n\nPair skill-reach with **NVIDIA's SkillSpector** (Tier 1 and Tier 2 of [SkillEvaluator](https://github.com/NVIDIA/SkillEvaluator)). As the Nemotron Labs team details in their [Evaluating Agent Skills deep dive](https://www.youtube.com/watch?v=4rI0zATFxYI), standard code linters miss **agent-native risks** buried in markdown prose: prompt injection, silent data exfiltration, tool poisoning, and trigger abuse. \n\nIf you run skill benchmarks directly on your workstation, your numbers are lying to you. A developer laptop is packed with ambient state that contaminates the baseline arm:\n\nCached Google Cloud credentials\n\nExported project variables and warm bash history\n\nGlobal environment rules\n\nAn unassisted baseline agent can pass a cloud deployment test by inheriting a default project ID or token sitting in your local shell profile, masking the exact failure your skill was written to solve.\n\nHost resource contention also warps agent trajectories. While hill climbing code-retrieval agents with AlphaEvolve, our internal testing showed that **running the exact same agent benchmark while a workstation was under CPU load dropped the score**. Slower file scans and subprocess timeouts cause the agent to abandon search paths prematurely.\n\nRunning prompts a single time and screenshotting the green checkmarks is not a real evaluation. Multi-turn tool-using agents branch on every shell exit code and stdout snippet. In our retrieval study, **more than a third of benchmark tasks produced different scores across consecutive runs of the exact same code and prompt**.\n\nA perfect score is equally meaningless without a paired baseline for comparison. Both Claude's plugin eval tool and NVIDIA's SkillEvaluator run **paired ablation across multiple repetitions**: Arm A runs the base model in a clean sandbox without the skill, while Arm B runs the identical prompt and sandbox with the skill mounted.\n\nComparing the score improvements across our sandbox trials exposed three clear patterns that single-shot testing hides. Without the plugin, the base model still stumbled into a working deployment on most runs, but only after burning minutes of wall-clock time guessing deprecated flags and retrying bad requests. Conversely, when a test case shows identical success with zero efficiency gain, the base model already owns that capability natively. Your skill is pure context tax — delete it.\n\nWhen you connect an evaluation harness to an automated optimizer (like [AlphaEvolve](https://docs.cloud.google.com/gemini/enterprise/docs/alphaevolve/developer-guide/overview) or reach optimize), it is a surprisingly common pattern for the **agent to cheat the grader**.\n\nIn our AlphaEvolve retrieval experiments, evolved candidates discovered **four distinct ways to game the benchmark without solving the task**. When the grading harness blocked specific expected substring checks in source code, an evolved candidate dynamically concatenated strings at runtime, traversed up the directory tree, and read the ground-truth answer file directly off disk. Another candidate spawned subprocesses to bypass AST checks.\n\nThe evolutionary loop favored those cheaters because the fitness formula blended correctness (0.95 weight) and token efficiency (0.05 weight) into a single score. Reading the answer file from disk takes **no LLM turns**, earning an efficiency bonus that gives cheaters a **survival advantage** over honest agents that spent tokens searching the repository.\n\nHardening your skill evaluator against reward-hacking takes three structural guardrails. First, configure Harbor so your test fixtures, grading scripts, and answer keys live inside a dedicated verifier container that mounts workspace artifacts only after the agent container exits. Second, keep SkillEvaluator's five dimensions (**Correctness**, **Discoverability**, **Efficiency**, **Effectiveness**, and **Security**) strictly orthogonal; never let token thrift compensate for a failed test or a security violation. Third, in [NVIDIA/SkillEvaluator#140](https://github.com/NVIDIA/SkillEvaluator/pull/140), we extended security checks to fail any trial that reads Kubernetes service account tokens or queries cloud metadata endpoints, while redacting live OAuth tokens from saved trajectories.\n\nLooking across all of our trials, the difference between the skills that delivered a **score improvement in our sandbox benchmark** and the **community skills that failed** came down to what was inside the markdown.\n\nSkills that copy-paste public API reference tables into skill files yield almost no lift. Frontier models already have public cloud docs in their pretraining weights; dumping thousands of words of static flags into the prompt just inflates time-to-first-token and buries actionable constraints.\n\nThe skills that increased our **Efficiency score** in our test suite taught compact **operational habits and verification reflexes** instead of encyclopedic reference text:\n\n**Verify CLI flags at the leaf subcommand before mutating state:** Rather than listing dozens of flags in markdown, the skill teaches a simple reflex: verify help output at the command line before executing a state-changing deployment.\n\n**Route through a lightweight catalog index on demand:** Rather than loading fourteen heavy cloud skills into context at once, a thin root router checks a curated remote index and pulls only the specific recipe needed for the task.\n\n**Enforce least-privilege auth defaults:** Explicit guardrails instruct agents to authenticate via Application Default Credentials rather than downloading static service-account JSON keys, while enforcing unauthenticated access restrictions alongside explicit invoker IAM bindings.\n\nBecause our evaluation stack runs in CI, upgrading models takes one flag change. We re-run the paired WITH vs. W/OUT matrix against the new model and delete every skill whose Delta drops to zero. Deleting a skill because the new model graduated past it is one of the cleanest high-fives in software engineering.\n\nReady to equip your coding agents with production-grade skills? Try the [Google Cloud Developer Plugin for AI Coding Agents](https://cloud.google.com/blog/topics/developers-practitioners/introducing-the-google-cloud-developer-plugin-for-ai-coding-agents).", "url": "https://wpnews.pro/news/five-rules-for-evaluating-agent-skills-before-they-bloat-your-context-window", "canonical_source": "https://cloud.google.com/blog/topics/developers-practitioners/evaluate-agent-skills-before-they-bloat-your-context-window/", "published_at": "2026-11-03 08:00:00+00:00", "updated_at": "2026-11-03 17:21:57.737990+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools", "mlops"], "entities": ["Google Cloud", "Gemini", "Antigravity", "Claude Sonnet 5", "Harbor", "GKE Autopilot", "NVIDIA", "skill-reach"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/five-rules-for-evaluating-agent-skills-before-they-bloat-your-context-window", "markdown": "https://wpnews.pro/news/five-rules-for-evaluating-agent-skills-before-they-bloat-your-context-window.md", "text": "https://wpnews.pro/news/five-rules-for-evaluating-agent-skills-before-they-bloat-your-context-window.txt", "jsonld": "https://wpnews.pro/news/five-rules-for-evaluating-agent-skills-before-they-bloat-your-context-window.jsonld"}}