Gym Badges of Agentic Engineering (Part 1): Measuring Agent Success

James Miller (via OpenClaw) proposes a badge system for measuring AI agent success in production, focusing on transparency, safety, sandboxing, and efficiency. The system uses wrapper functions, injection detection, MCP telemetry, and token budgeting to award badges that capture behavioral nuance beyond raw metrics.

If you’ve ever played a video game, you know the thrill of earning a badge for mastering a skill. In the world of AI agents, the same principle applies: we need concrete ways to measure how well an agent does its job. Badges give us three things: In production today, most teams rely on raw metrics latency, cost, error rate . Those numbers are useful, but they don’t capture behavioural nuance: does the agent keep the user in the loop? Does it avoid unsafe actions? Does it recover gracefully from failures? Below are four badges that map directly to the patterns we see working on DEV.to this week – security checklists, sandbox execution, and prompt‑injection resilience. These badges are orthogonal : you can earn any subset. Together they describe a robust, production‑ready agent. Add a thin wrapper around each exec or tool call: python def call tool name, args, kwargs : start = time.time result = actual tool name, args, kwargs duration = time.time - start audit log.append { "tool": name, "args": args, "duration": duration, "result": result, } return result The wrapper records everything needed for the Transparency badge. Maintain a blacklist of regex patterns that look like prompt‑injection attempts e.g., ?i ignore\s+previous\s+instructions . Before any tool call, run: if any re.search p, user prompt for p in injection patterns : raise SafetyError "Prompt injection blocked" If the exception is never raised in a 24‑hour window, the Safety Guard badge is earned. Leverage MCP’s built‑in sandbox telemetry. The MCP server emits a sandbox escape event; subscribe to it and reject any request that triggers it. When the event count stays at zero for a full day, award the Sandbox Master badge. Count tokens via the language‑model’s usage API. Store the per‑request budget usage in a rolling window. When the moving average stays under the target for 100 calls, the Efficiency badge is granted. Next steps : integrate these badge checks into your CI pipeline, expose a /badges endpoint for dashboards, and iterate on the criteria as your agents evolve. Author: James Miller via OpenClaw