{"slug": "five-bugs-in-my-llm-app-that-never-threw-an-error", "title": "Five bugs in my LLM app that never threw an error", "summary": "A developer spent two and a half weeks building an AI agent that reads code review history and distills conventions, but encountered five silent bugs that never threw errors. The bugs included Python's truthiness of non-empty strings, Pydantic rejecting null for string fields, and mismatched wildcard patterns between SQL LIKE and fnmatch, causing the agent to silently ignore most rules. The developer fixed these issues by adding Pydantic validators and converting patterns, ensuring the agent behaves as intended.", "body_md": "I spent two and a half weeks building an agent that reads a project's code review history and remembers what it decided. 86,321 comments from pandas, distilled into 298 conventions, served out of CockroachDB behind a Lambda.\n\nEvery serious bug I hit had the same shape. Nothing raised. Tests passed. The demo worked. And somewhere in the middle, a component was doing absolutely nothing.\n\nHere are five, with the code.\n\n`bool(\"false\")`\n\nis `True`\n\nThe whole point of the system is that it refuses when it does not know. The answering prompt returns JSON with an `answered`\n\nfield.\n\n```\nresult = await chat.complete_json(messages)\nif result.get(\"answered\"):\n    return result[\"answer\"]\n```\n\nA model that writes `\"answered\": \"false\"`\n\ninstead of `\"answered\": false`\n\nturns a refusal into an answer, because a non-empty string is truthy. The one behaviour the project exists to guarantee, defeated by a quotation mark.\n\nNow every model response goes through a Pydantic model that fails towards doing nothing:\n\n```\nclass AnswerOutput(ModelOutput):\n    answered: bool = False   # coerces \"false\" -> False\n    answer: str = \"\"\n```\n\nAn answer becomes a refusal. A drafted rule becomes unusable. A contradiction verdict becomes \"compatible\", which retires nothing. A garbled response is not evidence for writing to memory.\n\n`null`\n\ndiscarded whole responses, six times out of six\nThe fix for bug 1 caused bug 2, which is funnier than it was at the time.\n\nThe prompt that turns a maintainer's comment into a rule says:\n\n\"rationale\": \"why, one sentence, only if they gave a reason\"\n\nA maintainer who states a convention flatly gets `null`\n\nback, which is correct JSON for \"no reason given\". And:\n\n```\nclass DraftedRule(BaseModel):\n    statement: str = \"\"\n    rationale: str = \"\"   # null is not a str\n```\n\nPydantic rejects `null`\n\nfor a plain `str`\n\n. Validation is all or nothing. So the entire response was thrown away and replaced with the inert default, which downstream reads as the model having declined.\n\nA real maintainer correction became `{\"status\":\"ignored\",\"reason\":\"no convention stated\"}`\n\n. Six reproductions, six failures, on a payload that was otherwise exactly what I asked for.\n\n```\nclass ModelOutput(BaseModel):\n    @field_validator(\"*\", mode=\"before\")\n    @classmethod\n    def _null_text_is_empty(cls, value, info):\n        if value is not None:\n            return value\n        field = cls.model_fields.get(info.field_name)\n        return \"\" if field is not None and field.annotation is str else value\n```\n\nA `null`\n\nwhere a string was expected means empty, not malformed. Note the `field.annotation is str`\n\ncheck: `scope_pattern: str | None`\n\ngenuinely means None, and coercing that one would turn \"applies everywhere\" into a pattern matching nothing.\n\nEach rule stores a pattern for which files it applies to. I matched with `fnmatch`\n\n.\n\n```\nfnmatch(\"pandas/core/frame.py\", \"pandas/core/%\")   # False\n```\n\nThe patterns were **SQL LIKE**, not globs. `pandas/tests/%`\n\n, `pandas/core/%`\n\n, `pandas/tests/%/conftest.py`\n\n. Nothing asked the extraction prompt for that and nothing documented it. The model just wrote it, presumably because the rules were destined for a database.\n\n`%`\n\nis not an fnmatch wildcard. So 68 of 79 anchored rules silently matched nothing, and the agent ran entirely on the two rules that happened to be written `pandas/**/*.py`\n\n, which cover 63% of the repository.\n\nIt looked like it was working the entire time. The fix is one line, and the bug was invisible for a week:\n\n```\ncleaned = pattern.strip().strip(\"`\").lstrip(\"./\").replace(\"%\", \"*\")\n```\n\nThe agent comments on pull requests, choosing which conventions apply by embedding a description of the change and taking the nearest rules.\n\nIt commented on 38 of 40. I assumed a bad threshold and started tuning. At 1.05 it spoke on 95% of pull requests; at 0.95, on 18%.\n\nThen I measured the distribution instead. For 25 real pull requests, distance to the nearest rule versus the tenth nearest:\n\n| query phrasing | median nearest | median spread, 1st to 10th |\n|---|---|---|\n| title only | 0.954 | 0.097 |\n| title plus directories | 0.947 | 0.100 |\n| paths and counts | 0.958 | 0.105 |\n| written as a question | 0.933 | 0.095 |\n\nThe nearest rule is not meaningfully nearer than the tenth, under any phrasing. Every rule is a general statement about the same codebase, so every rule is similar to any description of a change to it.\n\nThere was nothing to threshold. Retrieval now runs on path matching, which is a fact, and similarity only orders what the paths already admitted.\n\n`author_association`\n\ndescribes the present\nI filtered to maintainer comments using GitHub's `author_association`\n\nfield.\n\nIt reports whether someone has write access **now**, not when they wrote the comment. One reviewer wrote 74,077 comments on pandas between 2012 and 2025, more than twice anyone else. He has since left the org, so all 74,077 come back as `CONTRIBUTOR`\n\n.\n\nFiltering on that field discarded about a third of the corpus, including the most experienced reviewers the project ever had. No error, no warning, just a smaller number that looked plausible.\n\nNone of these raised. Four of the five I found by counting something rather than by reading code: how often does it speak, how many rules match anything, how far apart are the distances really.\n\nA library used wrong fails loudly. A model used wrong returns confident, well-formatted output that is quietly detached from what you meant.\n\nSo pick a number that should hold if the system works, and go and look at it. Not \"does it return results\" but \"how often does it speak, and is that the rate I intended\". Not \"did it cite something\" but \"do the citations resolve\".\n\nCode is at [github.com/pyarchana/precedent](https://github.com/pyarchana/precedent), MIT.", "url": "https://wpnews.pro/news/five-bugs-in-my-llm-app-that-never-threw-an-error", "canonical_source": "https://dev.to/chanadev/five-bugs-in-my-llm-app-that-never-threw-an-error-1ejk", "published_at": "2026-08-30 02:31:58+00:00", "updated_at": "2026-08-30 03:22:36.051173+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["CockroachDB", "Lambda", "Pydantic", "pandas"], "alternates": {"html": "https://wpnews.pro/news/five-bugs-in-my-llm-app-that-never-threw-an-error", "markdown": "https://wpnews.pro/news/five-bugs-in-my-llm-app-that-never-threw-an-error.md", "text": "https://wpnews.pro/news/five-bugs-in-my-llm-app-that-never-threw-an-error.txt", "jsonld": "https://wpnews.pro/news/five-bugs-in-my-llm-app-that-never-threw-an-error.jsonld"}}