# My AI feature was failing 26% of the time. Nothing looked broken. > Source: > Published: 2026-09-26 13:44:43+00:00 I built TaskFlow, a project management app with an AI feature called Quick Add. You type *"assign the API docs to Priya by Friday"* and it creates a task with a title, an assignee, and a due date. For twelve days in August, it quietly stopped doing that for about a quarter of requests. Users got a task titled *"assign the API docs to Priya by Friday"* — no assignee, no date. No error. No crash. Just a worse product. My test suite caught it on day one. I didn't look until day twelve. This post is about both of those things. A normal unit test checks that a function returned. It can't tell you whether the model got the date right. LLM output looks fluent even when it's wrong, so "it didn't crash" means very little. So I built an eval harness: **71 test cases across 4 prompt suites** — quick-add (50), extract-tasks (9), decompose (6), and today (6). Each case is an input plus the fields the model should return. A few decisions that turned out to matter: Groq retired the model I was using, `llama-3.3-70b-versatile`, on August 16. I switched to `openai/gpt-oss-120b` on August 24. The nightly suite went red that same night. It stayed red for twelve consecutive nights before I looked properly. Two things kept it hidden: `400 Failed to validate JSON`. That reads like an infrastructure issue, so my first instinct was to blame the provider. The error message was a dead end. The distribution wasn't. | Suite | Result | `max_tokens` | |---|---|---| | quick-add | 35/50 | **300** | | extract-tasks | 9/9 | 1500 | | decompose | 6/6 | 1800 | | today | 6/6 | 900 | Every failure was in the one suite with the smallest token limit. The token data confirmed it. Of the quick-add cases that returned, the five largest completions were **283, 284, 288, 298 and 298** tokens — jammed right up against the 300 limit. The old model's largest was 64. The new model is a **reasoning model**. It spends tokens thinking before it writes any JSON. With a 300-token cap, it was running out mid-object and returning truncated JSON, which the provider then rejected. The first fix was one number: `max_tokens` from 300 to 900, in two places that must always match — the production controller and the eval harness. That exposed three more bugs the 400s had been hiding. Once the model could finish its answers, three date cases returned `due: null`: The prompt gave the model a 10-day calendar and told it never to compute weekdays itself. So anything outside that window came back empty. The proof: "in 3 days" and "by July 9" already passed, because both fall inside it. I split the rule. Weekday phrases still resolve from the calendar; absolute dates and offsets are computed forward from today. The prompt now says the calendar is *"a 10-day window, not a limit on what you may answer."* Priority had a similar gap — it only recognised words like "asap" or "critical", so "this is blocking the release" came back as unstated. It now judges described impact, not tone. **Quick-add went from 35/50 to 50/50. The full suite went from 78.9% to 98.6%.** As of the latest nightly, it's **69/71 — 97.2%**. Two cases fail. **One is a genuine quality miss.** The "today" planner includes a task it should filter out. I haven't fixed it because it's a ranking judgment, not a rule I can state cleanly — and I'd rather keep one honest failure than overfit the prompt to a single test case. **The other is the same bug coming back.** Same truncated-JSON signature, now in the "today" suite at its 900-token cap. My fix was too narrow: I raised the limit where it was failing, instead of asking which other endpoints had limits sized for a non-reasoning model. One endpoint is still at 400 tokens with no eval coverage at all. *TaskFlow is live at [taskflow-dpsa.vercel.app](https://taskflow-dpsa.vercel.app/). Code: [github.com/angelina10504/taskflow](https://github.com/angelina10504/taskflow).*