{"slug": "why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it", "title": "Why My LLM Guardrail Flagged the Right Answers (And Why I Refused to Fix It)", "summary": "An engineer building an open-source go-to-market strategy simulator found that a guardrail designed to catch LLM hallucinations flagged every number from a frontier API model as a false positive, while correctly catching 7.2% of fabrications from a local 8B model. The guardrail's strict matching logic deliberately avoids understanding arithmetic, and the engineer refused to 'fix' it because eliminating false positives would make the system less trustworthy.", "body_md": "Ask a Large Language Model (LLM) to write an executive strategy document based on your machine learning pipeline’s outputs, and sooner or later, it will invent a number. It will confidently claim a “47% lift” when your optimizer actually said 12%. In a boardroom setting, this single hallucinated metric is enough to burn the credibility of the entire dashboard. Due to this, most data teams either force a human into the loop for every generated figure, or they simply refuse to ship the LLM layer at all.\n\nI decided to take the third route while building [GTM Wargame](https://github.com/abhinandan-084/GTM-Wargame) (an open source go-to-market strategy simulator). I wanted to treat “**don’t hallucinate**” as a systemic architectural property, rather than relying on prompt engineering tricks. To do this, every number the agents were allowed to use - SHAP attributions, optimizer outputs, market context was stored in a strict Ground Truth Pool. Before any text reaches the UI, a **ConsistencyChecker **cross-references every number the agents write against that pool.\n\nBuilding this guardrail was the easy part. The harder question was: **How often does it actually fire in a real world scenario, and when it does, is it right?**\n\nTo find out, I ran 60 paired simulations pitting a frontier API model against a local 8B model, and I hand audited every flagged number against a deterministically rebuilt ground truth. The first finding was expected: **the local model fabricated numbers at a materially higher rate (7.2%)**. The second finding is what this article is really about: every single flag the guardrail raised against the frontier model was a false positive. The model wasn’t hallucinating. It was doing correct arithmetic that my checker simply had no way to understand.\n\nMy first instinct as an engineer was to “fix” the checker. By the end of the audit, I realized those false positives are the guardrail working as designed, and expanding its matching logic to eliminate them would make the whole system less trustworthy.\n\nThe guardrail’s logic is deliberately boring, which is exactly why it works:\n\nBefore testing it on real models, I evaluated the checker itself as a classifier using 75 labeled cases built from authentic pipeline artifacts. The cases were split into **clean cases, injected cases (seeded with unmatchable numbers), and trap cases (filled with naive-checker bait formatting like $1,234, negative signs, and k-notation of real pool values)**. The checker scored a perfect 1.000 precision and 1.000 recall, with zero false positives across the 258 numbers it ran on.\n\n```\nresult = ConsistencyChecker.validate_response(   text=manager_summary,   shap_info=shap_values,   market_context=market_context,   opt_results=optimizer_output,)# Returns: {\"is_valid\": False, \"hallucinated_values\": [47.0], \"error_msg\": \"...\"}\n```\n\nHowever, a perfect score deserves skepticism. The labels for this evaluation were defined by the checker’s own matching rules, so by construction, this eval couldn’t disagree with itself. It validated everything layered around the rule - number extraction, currency and comma cleaning, sign handling, pool flattening. This however says nothing about whether the matching rule itself was the right architectural contract. That question needs real models.\n\nI ran the full pipeline (XGBoost forecasting, SHAP attributions, SciPy budget optimization, followed by the agent boardroom) across 30 identical seeds. The pipeline runs on a synthetic sales dataset generated by the repo itself, so no proprietary data is involved and every result can be rebuilt from a seed. Both models faced the exact same scenarios: Gemini (gemini-3-flash-preview, the repo’s pinned default) over the API, and Llama 3.1 8B Instruct (Q4_K_M) running fully locally through llama.cpp. These runs were recorded in July 2026 − preview models change over time, so the exact version and date matter for anyone comparing against these numbers. The checker scored the analyst and strategist nodes of every run.\n\nBefore looking at the outputs, I established a strict methodological rule: **A flag is not a fabrication. A flag means a number failed pool matching. That is an upper bound on hallucination, not a measurement of it. Reporting raw flag rates as “fabrication rates” is exactly the kind of unverified claim the guardrail exists to prevent.**\n\nHere are the raw flag counts:\n\nTwo things immediately stood out. First, the frontier model engaged with the quantitative data far more densely, writing nearly four times as many numbers per run yet triggered fewer absolute flags. Second, 13 total flags across all runs is a small enough population to audit exhaustively by hand.\n\nFor every flagged run, I used an offline audit script with zero LLMs in the loop to deterministically rebuild that run’s exact Ground Truth Pool from its seed. It revalidates the stored agent transcripts, and asserts the flags reproduced exactly, ensuring I was examining exactly what the checker saw at runtime.\n\nThen, I ran two mechanical passes per flag:\n\nThe derivation pass comes with a built-in trap. With roughly 50 values in a pool, the number of pairwise combinations runs into the thousands. By pure chance, some derivation will land near almost any small number. Therefore, a mechanical hit isn’t enough. The final classification required semantic support from the text: the model had to explicitly name the operands it was combining. Any ambiguous flags defaulted to **FABRICATED**.\n\nHere are the final verdicts:\n\nEvery single one of the local model’s 10 flags survived the audit as a genuine hallucination. Conversely, not a single one of the frontier model’s flags was an actual fabrication.\n\nAll three of Gemini’s flags followed the exact same pattern. Let’s look at the flag from Seed 1015. The ground truth pool contained an optimized price of 535.68 and a market leader price of 715.18. The LLM strategist wrote: “*…our reliance on a 25 percent price gap relative to the leader (715.18) makes us highly vulnerable to any aggressive downward moves from the competition.*”\n\nCheck the arithmetic: (715.18 − 535.68) / 715.18 = 25.1%. Both operands are real pool values. Both are named in the surrounding text. The model computed a mathematically correct, highly relevant business metric. And the checker flagged it, simply because the number “25” wasn’t explicitly stored in the original pool. Seeds 1019 and 1023 produced the same pattern with different numbers, each verifiably correct, each flagged.\n\nI call this category **correct-but-derived**. It’s a failure mode that standard guardrail evaluations miss entirely because the eval’s labels are defined by the checker’s own matching rules. It took real model behavior and a manual audit to expose the flaw.\n\nThe obvious engineering patch writes itself: teach the checker arithmetic. Extend matching logic to cover pairwise derivations over ground truth pool values and all three of Gemini’s false positives would vanish.\n\nI actually built that logic (it’s the derivation pass used in the audit script), but running it offline convinced me to never put it into a live guardrail. Here is why:\n\nThe entire value of a guardrail is that a green light signifies a strict, undeniable truth: this number exists in the raw data. Every time the guardrail matching contract is expanded, the value of that green light is silently diluted.\n\nSo, the contract stays narrow. The checker verifies existence, not derivability. Deriving new true values is legitimate LLM behavior that falls outside the contract. It gets flagged, and the human user resolves it.\n\nNow, let’s address the local model, because this is where the guardrail proved it was heavily load-bearing. After the audit, the true fabrication rates separated cleanly:\n\nThe character of the local model’s hallucinations is highly instructive. It didn’t just get numbers slightly wrong; it invented entire quantitative structures that the pipeline never produced. It hallucinated an entire ROI table − $100k spend, $500k revenue” − in a pipeline that computes neither revenue nor ROI. My personal favorite was from Seed 1017: “*Reduce budget by 20% to $X.*” The model fabricated a metric and left the template variable sitting right next to it in the exact same sentence unfilled.\n\nIn my last article, I wrote about the “[ Abstraction Tax](https://medium.com/towards-artificial-intelligence/optimizing-local-llm-inference-on-constrained-hardware-783a14af365d)”, the throughput you sacrifice when using convenient local inference wrappers on constrained hardware. This is the other side of the coin:\n\nThis presents an operational paradox. The deployments that force engineers onto local models − regulated data, air-gapped systems, strict client confidentiality − are the exact environments where output verification is the hardest to outsource, and where a hallucinated number does the most damage. If your privacy requirements push you toward local inference, a grounding check isn’t just a nice-to-have feature − it is the only thing that makes the deployment defensible.\n\nTwo caveats. First, this guardrail benchmarks numerical grounding against a known pool, not semantic correctness. Second, K=30 runs on a single local model quantization is enough to establish confidence intervals, but it is not a definitive industry leaderboard. The takeaway isn’t “Gemini beats Llama”; the takeaway is that fabrication rates vary wildly across deployment architectures, and you need to measure yours.\n\nI set out to measure how often LLMs invent numbers when summarizing ML outputs, and I walked away with a system design principle I didn’t expect: **the audit matters more than the guardrail**.\n\nRaw metrics told a highly misleading story (0.6% vs 7.2% flag rates). Only a manual classification audit revealed the truth: a 0% vs 7.2% actual fabrication rate, where every single flag against the frontier model was a mathematically correct derivation that the checker was right to be suspicious of, but wrong to condemn.\n\nIf you are building numerical guardrails for LLMs, here are the rules I’m carrying forward:\n\nAll code, raw data, and empirical setups used to generate these profiles are fully reproducible. The checker evaluation can be rebuilt offline without API keys, and every flagged transcript is available in the [GTM Wargame repository](https://github.com/abhinandan-084/GTM-Wargame).\n\n[Why My LLM Guardrail Flagged the Right Answers (And Why I Refused to Fix It)](https://pub.towardsai.net/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it-0db77efb0644) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it", "canonical_source": "https://pub.towardsai.net/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it-0db77efb0644?source=rss----98111c9905da---4", "published_at": "2026-07-17 17:01:03+00:00", "updated_at": "2026-07-17 17:26:18.547285+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-tools", "ai-agents"], "entities": ["Gemini", "Llama 3.1 8B Instruct", "GTM Wargame", "ConsistencyChecker", "XGBoost", "SciPy", "llama.cpp"], "alternates": {"html": "https://wpnews.pro/news/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it", "markdown": "https://wpnews.pro/news/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it.md", "text": "https://wpnews.pro/news/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it.txt", "jsonld": "https://wpnews.pro/news/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it.jsonld"}}