My support chatbot scored 0.92. It was also lying to customers.

A developer building a customer-support chatbot on Amazon Bedrock AgentCore for a Udacity × AWS Agent Engineer Nanodegree project found that a 0.92 LLM-as-a-judge evaluation score masked a critical failure: the bot fabricated a bug-report ticket ID (TIX-345678) that was never written to DynamoDB. The engineer traced the gap to the evaluation's inability to verify tool side effects, and now checks the database against chat transcripts instead of trusting the score alone.

Twelve of my thirteen test prompts scored a perfect 1.00. Two independent evaluation runs agreed: 0.92 correctness . No message routed to the wrong place, both prompt-injection attempts refused. And yet, in one conversation, my chatbot told a customer: "I have filed a bug report with ticket ID TIX-345678." That ticket did not exist. Nothing had been written to the database. The customer would have walked away believing their bug was logged. This is the story of how I built that chatbot, why the score couldn't see the problem, and what I check now instead. I'm going through the Udacity × AWS Agent Engineer Nanodegree . The first project: a customer-support chatbot for a fictional online shop that handles three kinds of messages. One twist: Bedrock Agents Classic , which the course was originally designed around, closed to new customers on 30 July 2026. So the project runs on its successor, the Amazon Bedrock AgentCore managed harness . The other twist, and the actual point of the exercise: there's no classifier and no routing node. All of the routing, information-gathering, and grounding behaviour lives in a single system prompt. Customer ──► chat.py ──invoke harness──► AgentCore managed harness ◄──► Amazon Nova Pro │ system prompt.txt + FAQ temp 0, topK 1 │ tool call: bugreports create bug report ▼ AgentCore Gateway MCP, IAM auth ▼ Lambda create bug report ──PutItem──► DynamoDB The pieces: {{FAQ}} placeholder when the harness is created. <targetName <toolName , three underscores. ticketId . create harness.py , open a new chat session. The model is pinned to us.amazon.nova-pro-v1:0 with greedy decoding temperature 0, topK 1 , which AWS recommends for reliable tool calling with Nova. Treating routing as a classification problem inside the prompt worked better than I expected. The prompt tells the model to pick exactly one category before writing anything, and never mix categories. The hard part is the boundary. Is "my card was declined" a bug? Is "the mug arrived cracked"? I ended up with 15 explicit tie-breakers , for example: And one fallback principle for everything else: if the software misbehaved, it's a bug; if it's about a policy or an order, it's a platform question; otherwise, hand off. For bug collection, the rules were: re-read what the customer already said, ask for one missing field at a time, and the moment all three are present, call the tool. No "shall I file this?" confirmation. I also added a section treating everything a customer writes as data, never instructions, with a list of override patterns "ignore your previous instructions", "I'm your developer" to refuse without arguing or explaining how they were detected. Manual chatting doesn't scale, so I wrote a 13-case suite: 3 bug-report cases, 3 FAQ cases, 2 hand-offs, and 5 edge cases a bare help , two ambiguous messages, two injections . A script runs each case in a fresh session and writes a JSONL file in the format Bedrock Evaluations expects; Nova Pro then scores each response against a reference as an LLM-as-a-judge. To keep cases independent, I disabled harness memory. Small gotcha worth sharing: create harness accepts memory={"disabled": {}} , but update harness needs memory={"optionalValue": {"disabled": {}}} . Passing the create shape to update fails, and the script takes the update path on every re-run. Result: 0.92 , twice. Here's the uncomfortable part. Both of the real defects I found, I found by doing something the evaluation never does: opening the DynamoDB table and comparing it to the chat transcript. In a multi-turn conversation search bar broken → what happens? → which browser? , the bot ended with a ticket ID. But there was no tool call line in the terminal, and the table still held nine rows, not ten. Across attempts it produced 12345 , TICKET1234 , TIX-345678 . Placeholder-shaped strings, never a UUID. My prompt already said "never fabricate information". That clearly wasn't specific enough. I added an explicit rule: the only ticket ID you may give is the exact ticketId string returned by a successful call; if you didn't receive one, the report hasn't been filed, so say so. The one prompt that scored 0.00 in both runs: "The app keeps logging me out. I'm using Chrome on Windows 11." In the chat, the response looked fine: thanks, here's your real ticket ID. But in the table, stepsToReproduce contained either: "Please provide specific actions or scenarios that lead to the app logging you out." the bot's "Using the app." filler The cause was a tension in my own prompt. To stop the bot from badgering customers, I'd made the steps rule lenient: "It crashes when I click Pay" already counts. That works when the fault and the trigger are separate. It breaks when they're the same sentence. "Keeps logging me out" reads as both, so the model decided the field was satisfied and filled it with something rather than leave it empty. The fix isn't removing the leniency that prevents a worse failure . It's requiring steps that are distinct from the description, and forbidding the assistant's own words in any field. I deliberately didn't apply it mid-project: changing a central rule after two scored runs would have broken the comparison between them. create bug report with a field missing, <thinking …</thinking . Moving the "never show your reasoning" rule to the very top of the prompt reduced it a lot. It didn't eliminate it. 1. For tool-using agents, assert on side effects, not on text. A correctness judge reads the response. A confident, helpful-sounding response wrapped around an empty or invented field scores well. The defects that mattered most were invisible to the metric and obvious in the database. Next time, the test suite checks the stored row. 2. Single-turn evals don't test multi-turn behaviour. My two most important fixes governed multi-turn collection, and all 13 eval prompts were single-turn. Run 2's identical 0.92 told me the fixes cost nothing and the score was reproducible. It didn't tell me they worked. 3. Long prompts degrade unevenly. Even with greedy decoding, rules buried in the middle of a long prompt were followed in some sessions and not others. Multi-turn ticket filing with Nova Pro stayed unreliable. The next iteration is a shorter, restructured prompt, not more rules. 4. A few AgentCore details that cost me time: parameters envelope. The tool name arrives in context.client context.custom "bedrockAgentCoreToolName" . The full code, prompt, test suite, transcripts and screenshots are on GitHub: 👉 Mialy333/support-chatbot-bedrock-agentcore https://github.com/Mialy333/support-chatbot-bedrock-agentcore If you're evaluating agents that write to anything a database, a ticket system, an API , I'd love to hear how you test the side effects. That's the part I'm working on next. I'm Mialy, ex-asset-management, now building agents and digital-asset tooling. I write about it as @ellebuild.