{"slug": "jul-13-watching-ai-be-confidently-wrong", "title": "\"[Jul 13] Watching AI Be Confidently Wrong\"", "summary": "An engineer developing an algorithmic trading system found that an AI reviewing financial reports confidently flagged genuine earnings as 'impossible' and 'contaminated data,' based solely on outdated training data. The engineer also discovered a bug where the AI attached the wrong company name to correct financial data, and noted that a smarter model could be more dangerous when input is flawed. The engineer fixed the naming bug and added safeguards, concluding that confident AI judgments of 'that doesn't make sense' deserve extra scrutiny.", "body_md": "*An AI reviewing a report flagged real earnings as an \"impossible number\" and called it data contamination — it turned out to be genuine*\n\n*This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).*\n\nToday happened to fall on a loose homeschooling day, so I had more time to work than a typical weekday. Maybe that's why so much came out of it.\n\nAn AI helping review a report saw one company's financial figures come in unusually high on a certain metric and declared, very confidently, that \"a number like this has never appeared at any point in this industry or company size\" — flagging it as contaminated data.\n\nChecking it myself, that wasn't the case. The earnings really were that good — a recent boom in the relevant industry meant a figure that would have been \"impossible\" by old standards was now real.\n\nAsking the AI why it had judged this way, it turned out its whole basis for \"this has never happened\" was just world knowledge frozen at its training cutoff. It had no idea a real change had happened since then.\n\nThinking back, something similar happened recently too — another AI tasked with checking whether a report's content was real or fabricated wrongly judged an actual, recent event as \"never happened.\" Today's incident was the same trap.\n\nA judgment like \"this number is too big to make sense\" sounds logically solid on the surface, but it only holds within the range of past data that AI happens to know. The claims worth trusting are the ones that hold regardless of world knowledge, like \"these two numbers inside the document contradict each other.\" A claim of the form \"this doesn't exist in the world\" — however confidently an AI states it — I've decided not to take at face value without checking it myself.\n\nLooking closely at one report, I found that the section handling financial analysis was pulling the correct ticker but attaching a completely different company's name to it. The numbers themselves belonged to the right company; only the label on top was wrong.\n\nThe cause was simple — the part that looks up financial data by ticker and the part that attaches a human-readable company name were running independently, and the naming logic silently failed to recognize the domestic ticker format. Because the numbers were correct, it looked fine at a glance, but the report was actually going out with the right data under the wrong label — which made it tricky to catch. I fixed the name injection and, as a safeguard, added a rule that defaults to neutral whenever a company name can't be resolved but the report still reaches an extreme verdict.\n\nI swapped the local model used for writing reports for a bigger, smarter one. On clean input, it was clearly better — the older model tended to retreat into \"hold\" whenever things were ambiguous, while the new one reaches firm conclusions backed by stated reasoning.\n\nBut that created a dilemma. When fed input that contained the mislabeled-name bug above, the old model's vague \"hold\" verdict caused relatively little harm, while **the new model confidently used the wrong information as grounds for an extreme verdict, including a strong sell.**\n\nA smarter model can be more dangerous — as judgment improves, so does its ability to package a mistake in more convincing form when the input is wrong. So I only rolled the new model out live after fixing the naming bug first, and for now I've also turned on a monitor that automatically pulls any unusually extreme verdict out of the report until a human reviews it.\n\nThe biggest lesson from today: the more confident an AI's \"that doesn't make sense\" answer sounds when verifying something, the more it deserves a second look. That confidence isn't necessarily from actually checking — it can just be logic that only holds together inside the AI's own outdated view of the world.", "url": "https://wpnews.pro/news/jul-13-watching-ai-be-confidently-wrong", "canonical_source": "https://dev.to/finaltype/jul-13-watching-ai-be-confidently-wrong-6lb", "published_at": "2026-08-22 11:04:20+00:00", "updated_at": "2026-08-22 11:43:37.088633+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-products", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/jul-13-watching-ai-be-confidently-wrong", "markdown": "https://wpnews.pro/news/jul-13-watching-ai-be-confidently-wrong.md", "text": "https://wpnews.pro/news/jul-13-watching-ai-be-confidently-wrong.txt", "jsonld": "https://wpnews.pro/news/jul-13-watching-ai-be-confidently-wrong.jsonld"}}