My AI feature was failing 26% of the time. Nothing looked broken.

A developer behind TaskFlow, a project management app, traced a 26% failure rate in its AI Quick Add feature to a 300-token max_tokens cap that truncated the output of a newly swapped-in reasoning model (openai/gpt-oss-120b), which the provider rejected as invalid JSON. After building a 71-case eval harness across four prompt suites and raising the limit to 900 tokens, quick-add went from 35/50 to 50/50 and the full suite from 78.9% to 98.6%, though the latest nightly sits at 69/71 with one endpoint still capped at 400 tokens and uncovered by evals.

I built TaskFlow, a project management app with an AI feature called Quick Add. You type "assign the API docs to Priya by Friday" and it creates a task with a title, an assignee, and a due date. For twelve days in August, it quietly stopped doing that for about a quarter of requests. Users got a task titled "assign the API docs to Priya by Friday" — no assignee, no date. No error. No crash. Just a worse product. My test suite caught it on day one. I didn't look until day twelve. This post is about both of those things. A normal unit test checks that a function returned. It can't tell you whether the model got the date right. LLM output looks fluent even when it's wrong, so "it didn't crash" means very little. So I built an eval harness: 71 test cases across 4 prompt suites — quick-add 50 , extract-tasks 9 , decompose 6 , and today 6 . Each case is an input plus the fields the model should return. A few decisions that turned out to matter: Groq retired the model I was using, llama-3.3-70b-versatile , on August 16. I switched to openai/gpt-oss-120b on August 24. The nightly suite went red that same night. It stayed red for twelve consecutive nights before I looked properly. Two things kept it hidden: 400 Failed to validate JSON . That reads like an infrastructure issue, so my first instinct was to blame the provider. The error message was a dead end. The distribution wasn't. | Suite | Result | max tokens | |---|---|---| | quick-add | 35/50 | 300 | | extract-tasks | 9/9 | 1500 | | decompose | 6/6 | 1800 | | today | 6/6 | 900 | Every failure was in the one suite with the smallest token limit. The token data confirmed it. Of the quick-add cases that returned, the five largest completions were 283, 284, 288, 298 and 298 tokens — jammed right up against the 300 limit. The old model's largest was 64. The new model is a reasoning model . It spends tokens thinking before it writes any JSON. With a 300-token cap, it was running out mid-object and returning truncated JSON, which the provider then rejected. The first fix was one number: max tokens from 300 to 900, in two places that must always match — the production controller and the eval harness. That exposed three more bugs the 400s had been hiding. Once the model could finish its answers, three date cases returned due: null : The prompt gave the model a 10-day calendar and told it never to compute weekdays itself. So anything outside that window came back empty. The proof: "in 3 days" and "by July 9" already passed, because both fall inside it. I split the rule. Weekday phrases still resolve from the calendar; absolute dates and offsets are computed forward from today. The prompt now says the calendar is "a 10-day window, not a limit on what you may answer." Priority had a similar gap — it only recognised words like "asap" or "critical", so "this is blocking the release" came back as unstated. It now judges described impact, not tone. Quick-add went from 35/50 to 50/50. The full suite went from 78.9% to 98.6%. As of the latest nightly, it's 69/71 — 97.2% . Two cases fail. One is a genuine quality miss. The "today" planner includes a task it should filter out. I haven't fixed it because it's a ranking judgment, not a rule I can state cleanly — and I'd rather keep one honest failure than overfit the prompt to a single test case. The other is the same bug coming back. Same truncated-JSON signature, now in the "today" suite at its 900-token cap. My fix was too narrow: I raised the limit where it was failing, instead of asking which other endpoints had limits sized for a non-reasoning model. One endpoint is still at 400 tokens with no eval coverage at all. TaskFlow is live at taskflow-dpsa.vercel.app https://taskflow-dpsa.vercel.app/ . Code: github.com/angelina10504/taskflow https://github.com/angelina10504/taskflow .