{"slug": "my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken", "title": "My AI feature was failing 26% of the time. Nothing looked broken.", "summary": "A developer behind TaskFlow, a project management app, traced a 26% failure rate in its AI Quick Add feature to a 300-token max_tokens cap that truncated the output of a newly swapped-in reasoning model (openai/gpt-oss-120b), which the provider rejected as invalid JSON. After building a 71-case eval harness across four prompt suites and raising the limit to 900 tokens, quick-add went from 35/50 to 50/50 and the full suite from 78.9% to 98.6%, though the latest nightly sits at 69/71 with one endpoint still capped at 400 tokens and uncovered by evals.", "body_md": "I built TaskFlow, a project management app with an AI feature called Quick Add. You type *\"assign the API docs to Priya by Friday\"* and it creates a task with a title, an assignee, and a due date.\n\nFor twelve days in August, it quietly stopped doing that for about a quarter of requests. Users got a task titled *\"assign the API docs to Priya by Friday\"* — no assignee, no date. No error. No crash. Just a worse product.\n\nMy test suite caught it on day one. I didn't look until day twelve. This post is about both of those things.\n\nA normal unit test checks that a function returned. It can't tell you whether the model got the date right. LLM output looks fluent even when it's wrong, so \"it didn't crash\" means very little.\n\nSo I built an eval harness: **71 test cases across 4 prompt suites** — quick-add (50), extract-tasks (9), decompose (6), and today (6). Each case is an input plus the fields the model should return.\n\nA few decisions that turned out to matter:\n\nGroq retired the model I was using, `llama-3.3-70b-versatile`, on August 16. I switched to `openai/gpt-oss-120b` on August 24.\n\nThe nightly suite went red that same night. It stayed red for twelve consecutive nights before I looked properly.\n\nTwo things kept it hidden:\n\n`400 Failed to validate JSON`. That reads like an infrastructure issue, so my first instinct was to blame the provider.\nThe error message was a dead end. The distribution wasn't.\n\n| Suite | Result | `max_tokens` | \n|---|---|---|\n| quick-add | 35/50 | **300** | \n| extract-tasks | 9/9 | 1500 | \n| decompose | 6/6 | 1800 | \n| today | 6/6 | 900 | \n\nEvery failure was in the one suite with the smallest token limit.\n\nThe token data confirmed it. Of the quick-add cases that returned, the five largest completions were **283, 284, 288, 298 and 298** tokens — jammed right up against the 300 limit. The old model's largest was 64.\n\nThe new model is a **reasoning model**. It spends tokens thinking before it writes any JSON. With a 300-token cap, it was running out mid-object and returning truncated JSON, which the provider then rejected.\n\nThe first fix was one number: `max_tokens` from 300 to 900, in two places that must always match — the production controller and the eval harness.\n\nThat exposed three more bugs the 400s had been hiding. Once the model could finish its answers, three date cases returned `due: null`:\n\nThe prompt gave the model a 10-day calendar and told it never to compute weekdays itself. So anything outside that window came back empty. The proof: \"in 3 days\" and \"by July 9\" already passed, because both fall inside it.\n\nI split the rule. Weekday phrases still resolve from the calendar; absolute dates and offsets are computed forward from today. The prompt now says the calendar is *\"a 10-day window, not a limit on what you may answer.\"*\n\nPriority had a similar gap — it only recognised words like \"asap\" or \"critical\", so \"this is blocking the release\" came back as unstated. It now judges described impact, not tone.\n\n**Quick-add went from 35/50 to 50/50. The full suite went from 78.9% to 98.6%.**\n\nAs of the latest nightly, it's **69/71 — 97.2%**. Two cases fail.\n\n**One is a genuine quality miss.** The \"today\" planner includes a task it should filter out. I haven't fixed it because it's a ranking judgment, not a rule I can state cleanly — and I'd rather keep one honest failure than overfit the prompt to a single test case.\n\n**The other is the same bug coming back.** Same truncated-JSON signature, now in the \"today\" suite at its 900-token cap. My fix was too narrow: I raised the limit where it was failing, instead of asking which other endpoints had limits sized for a non-reasoning model. One endpoint is still at 400 tokens with no eval coverage at all.\n\n*TaskFlow is live at [taskflow-dpsa.vercel.app](https://taskflow-dpsa.vercel.app/). Code: [github.com/angelina10504/taskflow](https://github.com/angelina10504/taskflow).*", "url": "https://wpnews.pro/news/my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken", "canonical_source": "https://dev.to/angelina_gupta/my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken-3c5f", "published_at": "2026-09-26 13:44:43+00:00", "updated_at": "2026-09-26 14:00:21.132034+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["TaskFlow", "Groq", "llama-3.3-70b-versatile", "openai/gpt-oss-120b", "angelina10504"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken", "markdown": "https://wpnews.pro/news/my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken.md", "text": "https://wpnews.pro/news/my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken.txt", "jsonld": "https://wpnews.pro/news/my-ai-feature-was-failing-26-of-the-time-nothing-looked-broken.jsonld"}}