Most agent frameworks pass a test once and stop checking. Real agents run in production. Context drops. Tools time out. APIs return bad data. This happens every day, not once. AgentGauntlet treats this as the normal case, not a rare failure. AgentGauntlet gives you two ways to test your own agent against this kind of chaos.
Works with LangGraph, CrewAI, AutoGen, or any custom Python agent calling the OpenAI or Anthropic SDKs.
import agentgauntlet
agentgauntlet.init(probability=0.1, frameworks=["openai", "requests"])
Add these two lines before you build your graph, crew, or agent. No other code changes needed.
Works with OpenClaw, Hermes Agent, or any runtime that supports an OPENAI_BASE_URL
or ANTHROPIC_BASE_URL
override. Most agent runtimes support this, regardless of language.
agentgauntlet proxy --upstream https://api.openai.com --port 8888
export OPENAI_BASE_URL=http://localhost:8888/v1
your-agent-here
Your agent code stays unmodified.
Proxy mode tests any agent you point at the proxy. The in-process patch only reaches Python code running in the same interpreter. A Node.js or Go agent stays invisible to the patch, no matter how you write the hooks.
| Injector | Targets | Sabotage |
|---|---|---|
| Amnesia | LLM calls | Drops 10 to 30 percent of history. Keeps the system prompt and current turn. |
| Distractor | LLM calls | Splices a contradictory instruction into the system prompt or the latest user turn. |
| Gaslighter | LLM and tool calls | Simulates a timeout, a 429 response, or a 503 response. |
| Mutator | LLM and tool responses | Flips booleans, shifts numbers, mangles keys, based on a per-field probability. In proxy mode, Mutator also corrupts the tool_call arguments the model returns. |
In proxy mode, all four injectors compete in one pool on every call, since everything reaching the proxy counts as an LLM API call. In-process mode splits the work: Amnesia and Distractor target LLM calls, Gaslighter and Mutator target tool calls. Either way, one intercepted call triggers one event at most. The blast radius comes from a single weighted random draw per call, not a separate roll for each injector.
with agentgauntlet.run():
result = my_agent(user_query)
if result.balance == expected_balance:
agentgauntlet.mark_success()
else:
agentgauntlet.mark_failure("wrong balance reported")
Skip mark_success
and mark_failure
, and AgentGauntlet falls back to crash detection. This gives a weaker signal, since an agent processes corrupted data, produces a wrong answer, and exits clean without a crash. Run examples/basic_agent.py
with the chaos flag to see this happen. Mutator corrupts an account balance, the agent reports the wrong number, nothing crashes, and the scorecard still shows a failure, because AgentGauntlet checks the actual answer, not the process exit code.
Chaos event: Mutator corrupted a payload from a fetch_balance tool call.
AgentGauntlet Post-Mortem Scorecard
Total Chaos Events Injected: 3
Task Outcome: failure (reported 12505.0 vs 1250.5)
Resilience Score: 0%
agentgauntlet.init(
probability=0.1,
blast_radius={"amnesia": 0.1, "distractor": 0.05, "gaslighter": 0.15, "mutator": 0.1},
frameworks=["openai", "requests"],
injectors=["amnesia", "gaslighter"],
targets=["api.mytools.com"],
seed=7,
timeout_range=(0, 30),
amnesia_strategy="random",
mutation_rates={"boolean_flip": 0.5, "numeric_shift": 0.5, "key_mangle": 0.2},
)
probability
sets one shared weight for all four injectors.frameworks
lists which client libraries to patch.requests
andopenai
cover those SDKs directly.httpx
also covershttpx.AsyncClient
.injectors
gives you an allow list. The default runs all four.targets
restricts Gaslighter and Mutator to matching URLs.seed
makes a run reproducible.timeout_range
sets Gaslighter's simulated timeout sleep, in seconds.amnesia_strategy
switches between random drops andoldest_first
, for deterministic degradation.mutation_rates
sets rates for boolean flips, numeric shifts, and key mangling.
agentgauntlet proxy --upstream https://api.openai.com --port 8888 \
--probability 0.15 --seed 7 --timeout-min 0 --timeout-max 30
Set probability
above roughly 0.25, and chaos hits almost every call through the proxy, since all four injectors share one pool there. This works as a maximum chaos setting for a demo, not a bug. Know this before you set probability
to 1.0 for a screenshot, and wonder why every call gets hit.
A request body shaped like {"messages": [...]}
, or containing a system
key, counts as an LLM call. Anything else counts as a tool call, restricted to targets
if you set one. This works as a payload shape heuristic, not real provider or client identification. The heuristic sometimes misclassifies an unrelated API that sends a messages
field of its own.
Proxy mode carries a different limit. Proxy mode only sees traffic an agent configures to send through the proxy, using the base_url override pattern. Proxy mode does not run transparent HTTPS interception through a system HTTP_PROXY
setting plus a generated CA certificate. This approach would need a locally trusted root certificate installed before anything works, adding setup friction that a point-and-go design avoids. This limit stays stated, not hidden.
- In-process mode covers Python code built on
requests
orhttpx
. In-process mode does not reach subprocess tools, database drivers, or browser automation. No HTTP call exists in-process for these paths to intercept. - OpenClaw and Hermes Agent run as standalone runtimes, not Python libraries. The in-process patch cannot reach these runtimes at all. Only proxy mode reaches them, and only for their LLM API traffic, not their internal tool or skill execution.
- No integration tests exist yet against LangGraph, CrewAI, AutoGen, OpenClaw, or Hermes. This release ships unit tests for the injectors and blast radius logic, in
tests/
, plus a hand run demo against a local mock server, inexamples/
. Automated tests against real third party frameworks come later. - Crash attribution, through
sys.excepthook
, blames whichever chaos event fired most recently before an unhandled exception. This works as a heuristic, not a causal trace. Pairingagentgauntlet.run()
withmark_success
ormark_failure
avoids the need for this heuristic, when the caller checks task correctness directly.
See examples/basic_agent.py
for a real two step tool calling loop against a local mock LLM server, examples/mock_llm_server.py
, for a runnable before and after demo. See tests/test_injectors.py
for the injector and blast radius unit tests.