{"slug": "our-ai-agent-told-a-customer-it-processed-a-refund-it-didn-t", "title": "Our AI Agent Told a Customer It Processed a Refund. It Didn't.", "summary": "A developer building a demo support bot discovered that its AI agent falsely claimed to have processed a refund without actually calling the refund tool. This led to the creation of AgentNemesis, a project that uses OpenTelemetry traces to detect agent failures such as loops, unverified claims, broken promises, and broken handoffs. The demo agents run on Gemini 2.5 Flash.", "body_md": "I want to start with the moment that actually made this project feel real, not theoretical. We had a demo support bot running, testing random refund flows, and at some point it responded with something like: *\"I've gone ahead and processed your refund of $34.50.\"* Clean sentence. Confident. Nothing in the terminal complained. No stack trace, no red text, nothing.\n\nExcept it never called the refund tool. It just said it did.\n\nThat's the moment this whole project stopped being an idea and started feeling like an actual problem worth solving.\n\nNormal software either works or it visibly breaks. A server times out, a request 500s, a queue backs up, and something somewhere logs an error. You get a signal. AI agents don't give you that signal in the same way. An agent can respond instantly, with zero errors, and still be lying to the person it's talking to. It can also get stuck doing the same thing over and over, technically \"working,\" while quietly burning API calls and money the whole time. None of that shows up as a failure in the way engineers are trained to look for failures. It just looks like a normal, healthy response.\n\nSo the question we kept circling back to was pretty simple: how do you catch an agent lying, if lying doesn't throw an error?\n\nThe answer we landed on: stop trusting what the agent says, and start checking it against what it actually did.\n\nThe project is called **AgentNemesis**. The core idea is that every time our demo agent does anything (tool calls, decisions, its final response) we record it as a trace using OpenTelemetry, and ship that trace to SigNoz. Then a separate piece of the system reads that trace back and checks it for four specific kinds of failure:\n\n**Loops.** If the agent calls the same tool with the same input three times in a row without anything changing, that's flagged. It's usually a sign the agent is stuck, and every one of those repeated calls is still costing money even though nothing's actually happening.\n\n**Claims the agent never verified.** If the final response tells the user something as a fact (*\"your order was delivered,\"* say) but there's no earlier tool call that actually looked that up, we flag it. It means the agent just said something, without checking.\n\n**Promises it didn't keep.** This one's the refund example. The agent says it did something, and we go back through the trace looking for the tool call that would prove it. If that call isn't there, or the numbers don't match what it said, that's a broken promise.\n\n**Broken handoffs.** This one only applies to our multi-agent setup (we have a small planner, researcher, writer chain). Basically, did the researcher's actual findings make it into what the writer used, or did something get lost or ignored along the way.\n\nEvery conversation gets scored based on how many of these show up, and the dashboard shows you not just \"this one's bad\" but exactly which tool call was missing, or which line in the response wasn't backed by anything real.\n\nThe demo agents themselves run on Gemini 2.5 Flash.\n\nReading OpenTelemetry's docs is one thing. Getting all of this to actually work, with two people and limited time, taught me a few things I genuinely didn't expect going in.\n\n**First:** signing up for SigNoz Cloud wasn't as simple as I assumed. Their signup currently leans toward work email addresses, and I hit that wall partway through, at a point where I really didn't have spare hours to burn on an account creation problem. If I'm being honest with myself, this is entirely on me for not checking it earlier. Any time a project depends on a specific external tool, checking whether you can actually sign up for it should happen on day one, not whenever you first try to use it for real.\n\n**Second:**: I assumed getting a SigNoz Cloud account would be a five-minute step. It def wasn't. Their signup currently leans toward work email addresses, and I don't have one, so I had to actually go find a way around that instead of just clicking through like I expected. Lesson noted: check the signup requirements for any tool your project depends on right at the start, not the moment you actually need the account.\n\n**Third,** and this one actually hurt a little: we had a button on the dashboard that would trigger our demo scripts by spawning a Python process locally. Worked fine on my machine. The second we deployed the app to Vercel, that whole flow quietly stopped working, because Vercel doesn't give you a persistent Python runtime or support long-running background processes the way a normal server does. It's the kind of thing that works right up until the moment someone other than you tries to click the button, which is a genuinely bad time to find out. What we ended up doing was running all our demo scenarios ahead of time, so the deployed dashboard only ever needs to display results that already exist, instead of trying to generate anything live. If I could go back, I'd test the actual deployed version of anything that \"does\" something, way earlier than I did.\n\n**Fourth:** at one point we considered just having another LLM read the agent's response and judge whether it seemed accurate. It felt like the obvious shortcut. But something about using one AI's judgment to catch another AI lying felt like it defeated the point. What we do instead, matching tool calls against exact claims in the text, is a lot less clever-looking. But every single flag it raises comes with real, checkable proof behind it. You can point at the exact missing tool call. That mattered more to us than being fancy about it.\n\nNone of this works without traces being structured, queryable things instead of just walls of text in a log file. That's the part OpenTelemetry gives you, and SigNoz gave us a place to send it, pull it back out by conversation, and query it however we needed. The part I liked most, honestly, is that we don't just read from SigNoz, we write back into it too. Our trust scores go back in as custom metrics, so SigNoz itself can alert on them the same way it'd alert on latency or error rate. It's not a one-way relationship. The observability tool ends up watching the thing we built to watch the agent.\n\nThe thing that stuck with me most from this whole project is how normal a failing AI agent looks from the outside. No errors, no crashes, no kinda red numbers anywhere. its literally just a sentence that happens to not be true. Catching that means going back to something pretty old-fashioned: actually comparing what happened against what was said. Turns out the same tracing tools built for debugging a slow database query work here too. You just have to know to ask them a different question.", "url": "https://wpnews.pro/news/our-ai-agent-told-a-customer-it-processed-a-refund-it-didn-t", "canonical_source": "https://dev.to/rockyrockers/our-ai-agent-told-a-customer-it-processed-a-refund-it-didnt-25a3", "published_at": "2026-07-25 20:45:18+00:00", "updated_at": "2026-07-25 21:01:24.200528+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-products", "developer-tools"], "entities": ["AgentNemesis", "OpenTelemetry", "SigNoz", "Gemini 2.5 Flash"], "alternates": {"html": "https://wpnews.pro/news/our-ai-agent-told-a-customer-it-processed-a-refund-it-didn-t", "markdown": "https://wpnews.pro/news/our-ai-agent-told-a-customer-it-processed-a-refund-it-didn-t.md", "text": "https://wpnews.pro/news/our-ai-agent-told-a-customer-it-processed-a-refund-it-didn-t.txt", "jsonld": "https://wpnews.pro/news/our-ai-agent-told-a-customer-it-processed-a-refund-it-didn-t.jsonld"}}