{"slug": "how-well-do-your-agents-fail", "title": "How well do your agents fail?", "summary": "AgentGauntlet, a new testing tool for AI agents, simulates real-world chaos like context drops, tool timeouts, and bad API data to evaluate agent resilience, reporting a 0% resilience score when an agent fails to handle corrupted data. The tool supports LangGraph, CrewAI, AutoGen, and custom Python agents, and can run in-process or as a proxy for any runtime supporting OPENAI_BASE_URL or ANTHROPIC_BASE_URL overrides.", "body_md": "Most agent frameworks pass a test once and stop checking. Real agents run in production. Context drops. Tools time out. APIs return bad data. This happens every day, not once. AgentGauntlet treats this as the normal case, not a rare failure. AgentGauntlet gives you two ways to test your own agent against this kind of chaos.\n\nWorks with LangGraph, CrewAI, AutoGen, or any custom Python agent calling the OpenAI or Anthropic SDKs.\n\n``` python\nimport agentgauntlet\nagentgauntlet.init(probability=0.1, frameworks=[\"openai\", \"requests\"])\n```\n\nAdd these two lines before you build your graph, crew, or agent. No other code changes needed.\n\nWorks with OpenClaw, Hermes Agent, or any runtime that supports an `OPENAI_BASE_URL`\n\nor `ANTHROPIC_BASE_URL`\n\noverride. Most agent runtimes support this, regardless of language.\n\n```\nagentgauntlet proxy --upstream https://api.openai.com --port 8888\nexport OPENAI_BASE_URL=http://localhost:8888/v1\nyour-agent-here\n```\n\nYour agent code stays unmodified.\n\nProxy mode tests any agent you point at the proxy. The in-process patch only reaches Python code running in the same interpreter. A Node.js or Go agent stays invisible to the patch, no matter how you write the hooks.\n\n| Injector | Targets | Sabotage |\n|---|---|---|\n| Amnesia | LLM calls | Drops 10 to 30 percent of history. Keeps the system prompt and current turn. |\n| Distractor | LLM calls | Splices a contradictory instruction into the system prompt or the latest user turn. |\n| Gaslighter | LLM and tool calls | Simulates a timeout, a 429 response, or a 503 response. |\n| Mutator | LLM and tool responses | Flips booleans, shifts numbers, mangles keys, based on a per-field probability. In proxy mode, Mutator also corrupts the tool_call arguments the model returns. |\n\nIn proxy mode, all four injectors compete in one pool on every call, since everything reaching the proxy counts as an LLM API call. In-process mode splits the work: Amnesia and Distractor target LLM calls, Gaslighter and Mutator target tool calls. Either way, one intercepted call triggers one event at most. The blast radius comes from a single weighted random draw per call, not a separate roll for each injector.\n\n```\nwith agentgauntlet.run():\n    result = my_agent(user_query)\n    if result.balance == expected_balance:\n        agentgauntlet.mark_success()\n    else:\n        agentgauntlet.mark_failure(\"wrong balance reported\")\n```\n\nSkip `mark_success`\n\nand `mark_failure`\n\n, and AgentGauntlet falls back to crash detection. This gives a weaker signal, since an agent processes corrupted data, produces a wrong answer, and exits clean without a crash. Run `examples/basic_agent.py`\n\nwith the chaos flag to see this happen. Mutator corrupts an account balance, the agent reports the wrong number, nothing crashes, and the scorecard still shows a failure, because AgentGauntlet checks the actual answer, not the process exit code.\n\n```\nChaos event: Mutator corrupted a payload from a fetch_balance tool call.\n\nAgentGauntlet Post-Mortem Scorecard\nTotal Chaos Events Injected: 3\nTask Outcome: failure (reported 12505.0 vs 1250.5)\nResilience Score: 0%\nagentgauntlet.init(\n    probability=0.1,\n    blast_radius={\"amnesia\": 0.1, \"distractor\": 0.05, \"gaslighter\": 0.15, \"mutator\": 0.1},\n    frameworks=[\"openai\", \"requests\"],\n    injectors=[\"amnesia\", \"gaslighter\"],\n    targets=[\"api.mytools.com\"],\n    seed=7,\n    timeout_range=(0, 30),\n    amnesia_strategy=\"random\",\n    mutation_rates={\"boolean_flip\": 0.5, \"numeric_shift\": 0.5, \"key_mangle\": 0.2},\n)\n```\n\n`probability`\n\nsets one shared weight for all four injectors.`frameworks`\n\nlists which client libraries to patch.`requests`\n\nand`openai`\n\ncover those SDKs directly.`httpx`\n\nalso covers`httpx.AsyncClient`\n\n.`injectors`\n\ngives you an allow list. The default runs all four.`targets`\n\nrestricts Gaslighter and Mutator to matching URLs.`seed`\n\nmakes a run reproducible.`timeout_range`\n\nsets Gaslighter's simulated timeout sleep, in seconds.`amnesia_strategy`\n\nswitches between random drops and`oldest_first`\n\n, for deterministic degradation.`mutation_rates`\n\nsets rates for boolean flips, numeric shifts, and key mangling.\n\n```\nagentgauntlet proxy --upstream https://api.openai.com --port 8888 \\\n    --probability 0.15 --seed 7 --timeout-min 0 --timeout-max 30\n```\n\nSet `probability`\n\nabove roughly 0.25, and chaos hits almost every call through the proxy, since all four injectors share one pool there. This works as a maximum chaos setting for a demo, not a bug. Know this before you set `probability`\n\nto 1.0 for a screenshot, and wonder why every call gets hit.\n\nA request body shaped like `{\"messages\": [...]}`\n\n, or containing a `system`\n\nkey, counts as an LLM call. Anything else counts as a tool call, restricted to `targets`\n\nif you set one. This works as a payload shape heuristic, not real provider or client identification. The heuristic sometimes misclassifies an unrelated API that sends a `messages`\n\nfield of its own.\n\nProxy mode carries a different limit. Proxy mode only sees traffic an agent configures to send through the proxy, using the base_url override pattern. Proxy mode does not run transparent HTTPS interception through a system `HTTP_PROXY`\n\nsetting plus a generated CA certificate. This approach would need a locally trusted root certificate installed before anything works, adding setup friction that a point-and-go design avoids. This limit stays stated, not hidden.\n\n- In-process mode covers Python code built on\n`requests`\n\nor`httpx`\n\n. In-process mode does not reach subprocess tools, database drivers, or browser automation. No HTTP call exists in-process for these paths to intercept. - OpenClaw and Hermes Agent run as standalone runtimes, not Python libraries. The in-process patch cannot reach these runtimes at all. Only proxy mode reaches them, and only for their LLM API traffic, not their internal tool or skill execution.\n- No integration tests exist yet against LangGraph, CrewAI, AutoGen, OpenClaw, or Hermes. This release ships unit tests for the injectors and blast radius logic, in\n`tests/`\n\n, plus a hand run demo against a local mock server, in`examples/`\n\n. Automated tests against real third party frameworks come later. - Crash attribution, through\n`sys.excepthook`\n\n, blames whichever chaos event fired most recently before an unhandled exception. This works as a heuristic, not a causal trace. Pairing`agentgauntlet.run()`\n\nwith`mark_success`\n\nor`mark_failure`\n\navoids the need for this heuristic, when the caller checks task correctness directly.\n\nSee `examples/basic_agent.py`\n\nfor a real two step tool calling loop against a local mock LLM server, `examples/mock_llm_server.py`\n\n, for a runnable before and after demo. See `tests/test_injectors.py`\n\nfor the injector and blast radius unit tests.", "url": "https://wpnews.pro/news/how-well-do-your-agents-fail", "canonical_source": "https://github.com/Sub2mval/AgentGauntlet", "published_at": "2026-08-15 17:13:31+00:00", "updated_at": "2026-08-15 17:40:49.278903+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "ai-research"], "entities": ["AgentGauntlet", "LangGraph", "CrewAI", "AutoGen", "OpenAI", "Anthropic", "OpenClaw", "Hermes Agent"], "alternates": {"html": "https://wpnews.pro/news/how-well-do-your-agents-fail", "markdown": "https://wpnews.pro/news/how-well-do-your-agents-fail.md", "text": "https://wpnews.pro/news/how-well-do-your-agents-fail.txt", "jsonld": "https://wpnews.pro/news/how-well-do-your-agents-fail.jsonld"}}