{"slug": "we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-report", "title": "We read every message our first six users sent. The worst bug was our own progress report.", "summary": "A developer building a cloud AI coding agent read every message sent by its first six users and found that the worst bug was in the agent's own progress reporting. The agent's fallback summary only counted file writes and commands, ignoring reads and searches, so a 117-second turn involving 8 file reads and 10 searches was reported as 'nothing happened,' causing one user to abandon the session. The developer also discovered that the evaluation harness was misreporting performance, leading to false conclusions about model quality.", "body_md": "We build a cloud AI coding agent. Over 21 days, six real people signed up and used it.\n\nSix. Not a launch, not traction, not a growth story — six people, small enough that we could sit down and read every message they ever sent it. So we did.\n\nHere is everything that broke, including the one that was our fault in a way we didn't expect.\n\nThat string appears three times across six users. One of them typed `Continue`\n\n, waited, and got it back. It was the last message of their session. They didn't return.\n\nOur first assumption was that the agent had stalled — a hung tool call, a dead model, something obvious. It hadn't.\n\nThe message comes from a fallback. When the model returns no prose, we describe what it did instead:\n\n``` js\nconst summary = fullResponse || this.describeTurnActions(turnActions);\n```\n\nAnd `describeTurnActions`\n\ncounted this:\n\n```\nif (type === \"write_file\" || type === \"edit_file\") {\n  files.push(action.path);\n} else if (type === \"run_command\") {\n  commands++;\n}\n```\n\nWrites and commands. That is the whole census.\n\nNow the turn that ended that user's session: **117 seconds. 254,000 tokens. 8 file reads. 10 searches.** The agent opened the codebase, searched it, read through it, and then stopped without editing anything.\n\nEvery one of those actions was invisible to the counter. So the product took two minutes of real work and reported it to a paying-attention human as *nothing happened*.\n\nThere were two separate bugs sitting inside one sentence:\n\nWe spent our first hour debugging (2) while believing it was (1).\n\nAnother user's first turn produced 48 files and 41 commands — a real application skeleton. Then they typed `continue.`\n\nand got **2 files and 73 commands**. Then again: **2 files, 59 commands**.\n\nDozens of shell commands producing almost nothing. From the outside that reads as thrash, and it may well be.\n\nBut look at what I just told you about that turn. File counts and command counts. It is the same instrument as #1, and I do not yet know whether those 73 commands were flailing or were one long legitimate diagnosis. We are not currently equipped to tell the difference. That is the actual finding.\n\nOne user wrote a 518-character casual prompt — the kind of thing you type without thinking hard about it. The agent worked, hit its token budget partway through, and wrote an honest handoff explaining where it had got to.\n\nThat is the correct behaviour, and I will take it over a confident lie every time. It is still a build that stopped short of a working app, and the user experienced it as the product giving up.\n\nSame user, very next message. Their words, not our paraphrase.\n\nThe cause: scaffolded Vite apps were going out without a `base`\n\nset, so behind our preview proxy every asset resolved to a path that did not exist. A blank white page, underneath a promise of a live preview.\n\nThat one is fixed and deployed. It had also been sitting fixed in a branch for a while before anyone noticed it was not in production, which is its own small lesson.\n\nOne user wrote a genuinely principal-architect-grade PRD, ran it, and got 36 files — prisma schema, auth, API routes, middleware. Then they closed the tab.\n\n**They came back 12.8 hours later and kept building.**\n\nNo onboarding sequence. No email. No nudge. They came back the next day because the thing they had started was worth continuing. At six users that is not a statistic, but it is the single most encouraging thing in the data.\n\nAnd in that same session the agent hit a Next.js 16 async-params breakage across three route handlers, diagnosed it, and fixed all three unprompted. Nobody asked it to.\n\nTwice in one week we looked at a bad number, concluded the model was underperforming, and were wrong — because the thing doing the counting was broken.\n\nThe first is the one above: reads and searches were not in the census, so work looked like idleness.\n\nThe second was our evaluation harness. Fixtures were scoring high on trajectory quality while failing nearly all of their acceptance checks. Two measurements of the same run, disagreeing wildly. We spent real money iterating on the agent before anyone asked the obvious question — and the acceptance check was the thing that was wrong. The agent had been fine.\n\nBoth times, the fault was in the ruler and we went looking in the thing being measured.\n\n**Check what your instrument counts before you conclude the model is bad at its job.** It costs about twenty minutes. We skipped it twice.\n\nThe empty-turn message now knows the difference between \"investigated and stopped\" and \"genuinely nothing\":\n\n```\nif (files.length === 0 && commands === 0) {\n  if (reads > 0 || searches > 0) {\n    return `**I looked, but I did not change anything this turn.** I ${looked} ` +\n      `and stopped without editing — usually that means I could not work out ` +\n      `the next step on my own.\\n\\n` +\n      `Tell me what to do next and I will act on it.`;\n  }\n  return `**That turn produced nothing — that is a fault on our side, not ` +\n    `something you did wrong.**\\n\\nSend your last message again.`;\n}\n```\n\nTwo things worth saying about that snippet.\n\nOne: it does not fix the agent stopping early. It fixes us lying about it. Those are different jobs, and conflating them is how you end up shipping a nicer error message and calling the bug closed.\n\nTwo: read the second branch again. When it really is our fault, the product says so, in those words. A user who is told \"no file changes were made\" concludes they prompted it wrong. That is the worst available outcome — they blame themselves, and they leave.\n\nTwo of those six prompts are 15,000-character specifications for named products. Those are somebody's business, written into a text box they reasonably assumed was private. They would have made a far more interesting post than this one.\n\nThere are no names, no product concepts and no prompt text anywhere above, and there will not be unless the person says yes.\n\nIf you would rather watch the agent working than read about it failing, there is an uncut run here: [https://youtu.be/2sR4H1q9QCQ](https://youtu.be/2sR4H1q9QCQ) — the interesting part is at 1:12, where its own build check fails and it goes and fixes the config itself.", "url": "https://wpnews.pro/news/we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-report", "canonical_source": "https://dev.to/trynocoder/we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-progress-report-20lh", "published_at": "2026-08-27 15:02:06+00:00", "updated_at": "2026-08-27 15:18:29.075206+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-products"], "entities": ["Next.js", "Vite"], "alternates": {"html": "https://wpnews.pro/news/we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-report", "markdown": "https://wpnews.pro/news/we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-report.md", "text": "https://wpnews.pro/news/we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-report.txt", "jsonld": "https://wpnews.pro/news/we-read-every-message-our-first-six-users-sent-the-worst-bug-was-our-own-report.jsonld"}}