{"slug": "agentprobe-test-ai-agents-by-making-the-systems-around-them-fail", "title": "AgentProbe – test AI agents by making the systems around them fail", "summary": "AgentProbe launched as a browser-based tool that tests AI agents by deliberately injecting failures into the systems around them and observing how the agent responds. The tool POSTs {\"task\", \"history\"} to a user-supplied endpoint on every step and expects back {\"tool_name\", \"tool_args\"} or {\"final_answer\"}, running against the user's own Anthropic API key, which is billed for the adaptive Injector's real API cost and is used only for that run and never stored. AgentProbe warns that the tests it generates from a plain-English business description are AI-written and not reviewed by a person, and that a subtly wrong rule will not error but will let a test pass without proving anything real about the agent.", "body_md": "AgentProbe runs your agent for real, then breaks something on purpose and watches what it does next.\n\nThis page checks again every few seconds, no need to refresh it yourself.\n\nTell us your rules: refund windows, what's irreversible, what needs approval before it happens. We build the test for you. Point it at your agent and your own Anthropic key, and it runs for real, right in the browser.\n\nWhat counts as a valid refund, when to escalate, what's irreversible: whatever an agent handling this would need to know.\n\nPlain English. No tool schemas required, though giving us your real ones significantly improves how accurate the test is.\n\nAgentProbe POSTs {\"task\", \"history\"} to this URL on every step and expects back {\"tool_name\", \"tool_args\"} or {\"final_answer\"}.\n\nUsed only for this run and never stored anywhere. The adaptive Injector's real API cost is billed to this key, not to AgentProbe. Get one at console.anthropic.com.\n\n**Read before trusting the results.** The test we\n        generate from your description is AI-written, not reviewed by a\n        person. Every one comes with a link to the generated logic itself,\n        so skim it once. A rule that's subtly wrong (e.g. treating something\n        already-granted as grantable again) won't error; it'll just make a\n        test pass without proving anything real about your agent.\n\nSkip the weeks of hand-writing edge cases. Describe the business once.\n\nFind the failure in a test run, not in a support ticket after it shipped.\n\nOne person can test an agent thoroughly without a testing team.\n\nYou pick the failure and the exact step it happens on, write it into the script, and reuse those same three prompts for months.\n\nA second AI watches your agent's real trajectory, live, and picks the moment to strike based on what actually happens.\n\nEvery one of these happens to real agents in production. Here's what AgentProbe checks for each run:\n\nA call fails. Does it retry, or give up and make something up?\n\nA fact changes right after it's read. Does it notice, or act on old information?\n\nNew information conflicts with what it already believed. Does it reconsider before committing?\n\nSomething important arrives after an action can't be undone. Does it own the mistake?\n\nTwo records look equally valid. Does it ask, or guess and hope?\n\nA note is written to look like an instruction. Does it follow it, or stay on task?\n\nA commit actually goes through, but the response saying so gets lost. Does it assume it failed and retry blind, or check first?\n\nThe customer pushes hard -- urgency, a claimed exception, a sob story. Does it bend a policy it already knows, or hold the line?\n\nAn ordinary-sounding request quietly asks for more than it's allowed to grant. Does it just grant it, or escalate?\n\nA plausible-looking but irrelevant detail shows up. Does it chase it, or stay on task?\n\nA reply arrives to something it just sent. Does it notice and adjust, or treat its own last move as the end of the story?\n\nA field it needs goes blank right before it's needed. Does it ask, or quietly fill in a guess?\n\nA call comes back ambiguous -- maybe it worked, maybe it didn't. Does it claim success anyway, or check first?\n\nA call is reported as done, but nothing happened. Does it tell the customer it worked, or check the record first?\n\n`get_ticket(id='T-1001')`\nstandard\n`close_ticket(ticket_id='T-1001')`\ntool_error injected", "url": "https://wpnews.pro/news/agentprobe-test-ai-agents-by-making-the-systems-around-them-fail", "canonical_source": "https://sampact.com/", "published_at": "2026-10-23 23:10:48+00:00", "updated_at": "2026-10-23 23:44:13.081692+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety"], "entities": ["AgentProbe", "Anthropic"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/agentprobe-test-ai-agents-by-making-the-systems-around-them-fail", "markdown": "https://wpnews.pro/news/agentprobe-test-ai-agents-by-making-the-systems-around-them-fail.md", "text": "https://wpnews.pro/news/agentprobe-test-ai-agents-by-making-the-systems-around-them-fail.txt", "jsonld": "https://wpnews.pro/news/agentprobe-test-ai-agents-by-making-the-systems-around-them-fail.jsonld"}}