How well do your agents fail?

AgentGauntlet, a new testing tool for AI agents, simulates real-world chaos like context drops, tool timeouts, and bad API data to evaluate agent resilience, reporting a 0% resilience score when an agent fails to handle corrupted data. The tool supports LangGraph, CrewAI, AutoGen, and custom Python agents, and can run in-process or as a proxy for any runtime supporting OPENAI_BASE_URL or ANTHROPIC_BASE_URL overrides.

Most agent frameworks pass a test once and stop checking. Real agents run in production. Context drops. Tools time out. APIs return bad data. This happens every day, not once. AgentGauntlet treats this as the normal case, not a rare failure. AgentGauntlet gives you two ways to test your own agent against this kind of chaos. Works with LangGraph, CrewAI, AutoGen, or any custom Python agent calling the OpenAI or Anthropic SDKs. python import agentgauntlet agentgauntlet.init probability=0.1, frameworks= "openai", "requests" Add these two lines before you build your graph, crew, or agent. No other code changes needed. Works with OpenClaw, Hermes Agent, or any runtime that supports an OPENAI BASE URL or ANTHROPIC BASE URL override. Most agent runtimes support this, regardless of language. agentgauntlet proxy --upstream https://api.openai.com --port 8888 export OPENAI BASE URL=http://localhost:8888/v1 your-agent-here Your agent code stays unmodified. Proxy mode tests any agent you point at the proxy. The in-process patch only reaches Python code running in the same interpreter. A Node.js or Go agent stays invisible to the patch, no matter how you write the hooks. | Injector | Targets | Sabotage | |---|---|---| | Amnesia | LLM calls | Drops 10 to 30 percent of history. Keeps the system prompt and current turn. | | Distractor | LLM calls | Splices a contradictory instruction into the system prompt or the latest user turn. | | Gaslighter | LLM and tool calls | Simulates a timeout, a 429 response, or a 503 response. | | Mutator | LLM and tool responses | Flips booleans, shifts numbers, mangles keys, based on a per-field probability. In proxy mode, Mutator also corrupts the tool call arguments the model returns. | In proxy mode, all four injectors compete in one pool on every call, since everything reaching the proxy counts as an LLM API call. In-process mode splits the work: Amnesia and Distractor target LLM calls, Gaslighter and Mutator target tool calls. Either way, one intercepted call triggers one event at most. The blast radius comes from a single weighted random draw per call, not a separate roll for each injector. with agentgauntlet.run : result = my agent user query if result.balance == expected balance: agentgauntlet.mark success else: agentgauntlet.mark failure "wrong balance reported" Skip mark success and mark failure , and AgentGauntlet falls back to crash detection. This gives a weaker signal, since an agent processes corrupted data, produces a wrong answer, and exits clean without a crash. Run examples/basic agent.py with the chaos flag to see this happen. Mutator corrupts an account balance, the agent reports the wrong number, nothing crashes, and the scorecard still shows a failure, because AgentGauntlet checks the actual answer, not the process exit code. Chaos event: Mutator corrupted a payload from a fetch balance tool call. AgentGauntlet Post-Mortem Scorecard Total Chaos Events Injected: 3 Task Outcome: failure reported 12505.0 vs 1250.5 Resilience Score: 0% agentgauntlet.init probability=0.1, blast radius={"amnesia": 0.1, "distractor": 0.05, "gaslighter": 0.15, "mutator": 0.1}, frameworks= "openai", "requests" , injectors= "amnesia", "gaslighter" , targets= "api.mytools.com" , seed=7, timeout range= 0, 30 , amnesia strategy="random", mutation rates={"boolean flip": 0.5, "numeric shift": 0.5, "key mangle": 0.2}, probability sets one shared weight for all four injectors. frameworks lists which client libraries to patch. requests and openai cover those SDKs directly. httpx also covers httpx.AsyncClient . injectors gives you an allow list. The default runs all four. targets restricts Gaslighter and Mutator to matching URLs. seed makes a run reproducible. timeout range sets Gaslighter's simulated timeout sleep, in seconds. amnesia strategy switches between random drops and oldest first , for deterministic degradation. mutation rates sets rates for boolean flips, numeric shifts, and key mangling. agentgauntlet proxy --upstream https://api.openai.com --port 8888 \ --probability 0.15 --seed 7 --timeout-min 0 --timeout-max 30 Set probability above roughly 0.25, and chaos hits almost every call through the proxy, since all four injectors share one pool there. This works as a maximum chaos setting for a demo, not a bug. Know this before you set probability to 1.0 for a screenshot, and wonder why every call gets hit. A request body shaped like {"messages": ... } , or containing a system key, counts as an LLM call. Anything else counts as a tool call, restricted to targets if you set one. This works as a payload shape heuristic, not real provider or client identification. The heuristic sometimes misclassifies an unrelated API that sends a messages field of its own. Proxy mode carries a different limit. Proxy mode only sees traffic an agent configures to send through the proxy, using the base url override pattern. Proxy mode does not run transparent HTTPS interception through a system HTTP PROXY setting plus a generated CA certificate. This approach would need a locally trusted root certificate installed before anything works, adding setup friction that a point-and-go design avoids. This limit stays stated, not hidden. - In-process mode covers Python code built on requests or httpx . In-process mode does not reach subprocess tools, database drivers, or browser automation. No HTTP call exists in-process for these paths to intercept. - OpenClaw and Hermes Agent run as standalone runtimes, not Python libraries. The in-process patch cannot reach these runtimes at all. Only proxy mode reaches them, and only for their LLM API traffic, not their internal tool or skill execution. - No integration tests exist yet against LangGraph, CrewAI, AutoGen, OpenClaw, or Hermes. This release ships unit tests for the injectors and blast radius logic, in tests/ , plus a hand run demo against a local mock server, in examples/ . Automated tests against real third party frameworks come later. - Crash attribution, through sys.excepthook , blames whichever chaos event fired most recently before an unhandled exception. This works as a heuristic, not a causal trace. Pairing agentgauntlet.run with mark success or mark failure avoids the need for this heuristic, when the caller checks task correctness directly. See examples/basic agent.py for a real two step tool calling loop against a local mock LLM server, examples/mock llm server.py , for a runnable before and after demo. See tests/test injectors.py for the injector and blast radius unit tests.