We build a cloud AI coding agent. Over 21 days, six real people signed up and used it.
Six. Not a launch, not traction, not a growth story — six people, small enough that we could sit down and read every message they ever sent it. So we did.
Here is everything that broke, including the one that was our fault in a way we didn't expect.
That string appears three times across six users. One of them typed Continue
, waited, and got it back. It was the last message of their session. They didn't return.
Our first assumption was that the agent had stalled — a hung tool call, a dead model, something obvious. It hadn't.
The message comes from a fallback. When the model returns no prose, we describe what it did instead:
const summary = fullResponse || this.describeTurnActions(turnActions);
And describeTurnActions
counted this:
if (type === "write_file" || type === "edit_file") {
files.push(action.path);
} else if (type === "run_command") {
commands++;
}
Writes and commands. That is the whole census.
Now the turn that ended that user's session: 117 seconds. 254,000 tokens. 8 file reads. 10 searches. The agent opened the codebase, searched it, read through it, and then stopped without editing anything.
Every one of those actions was invisible to the counter. So the product took two minutes of real work and reported it to a paying-attention human as nothing happened.
There were two separate bugs sitting inside one sentence:
We spent our first hour debugging (2) while believing it was (1).
Another user's first turn produced 48 files and 41 commands — a real application skeleton. Then they typed continue.
and got 2 files and 73 commands. Then again: 2 files, 59 commands.
Dozens of shell commands producing almost nothing. From the outside that reads as thrash, and it may well be.
But look at what I just told you about that turn. File counts and command counts. It is the same instrument as #1, and I do not yet know whether those 73 commands were flailing or were one long legitimate diagnosis. We are not currently equipped to tell the difference. That is the actual finding.
One user wrote a 518-character casual prompt — the kind of thing you type without thinking hard about it. The agent worked, hit its token budget partway through, and wrote an honest handoff explaining where it had got to.
That is the correct behaviour, and I will take it over a confident lie every time. It is still a build that stopped short of a working app, and the user experienced it as the product giving up.
Same user, very next message. Their words, not our paraphrase.
The cause: scaffolded Vite apps were going out without a base
set, so behind our preview proxy every asset resolved to a path that did not exist. A blank white page, underneath a promise of a live preview.
That one is fixed and deployed. It had also been sitting fixed in a branch for a while before anyone noticed it was not in production, which is its own small lesson.
One user wrote a genuinely principal-architect-grade PRD, ran it, and got 36 files — prisma schema, auth, API routes, middleware. Then they closed the tab.
They came back 12.8 hours later and kept building.
No onboarding sequence. No email. No nudge. They came back the next day because the thing they had started was worth continuing. At six users that is not a statistic, but it is the single most encouraging thing in the data.
And in that same session the agent hit a Next.js 16 async-params breakage across three route handlers, diagnosed it, and fixed all three unprompted. Nobody asked it to.
Twice in one week we looked at a bad number, concluded the model was underperforming, and were wrong — because the thing doing the counting was broken.
The first is the one above: reads and searches were not in the census, so work looked like idleness.
The second was our evaluation harness. Fixtures were scoring high on trajectory quality while failing nearly all of their acceptance checks. Two measurements of the same run, disagreeing wildly. We spent real money iterating on the agent before anyone asked the obvious question — and the acceptance check was the thing that was wrong. The agent had been fine.
Both times, the fault was in the ruler and we went looking in the thing being measured.
Check what your instrument counts before you conclude the model is bad at its job. It costs about twenty minutes. We skipped it twice.
The empty-turn message now knows the difference between "investigated and stopped" and "genuinely nothing":
if (files.length === 0 && commands === 0) {
if (reads > 0 || searches > 0) {
return `**I looked, but I did not change anything this turn.** I ${looked} ` +
`and stopped without editing — usually that means I could not work out ` +
`the next step on my own.\n\n` +
`Tell me what to do next and I will act on it.`;
}
return `**That turn produced nothing — that is a fault on our side, not ` +
`something you did wrong.**\n\nSend your last message again.`;
}
Two things worth saying about that snippet.
One: it does not fix the agent stopping early. It fixes us lying about it. Those are different jobs, and conflating them is how you end up shipping a nicer error message and calling the bug closed.
Two: read the second branch again. When it really is our fault, the product says so, in those words. A user who is told "no file changes were made" concludes they prompted it wrong. That is the worst available outcome — they blame themselves, and they leave.
Two of those six prompts are 15,000-character specifications for named products. Those are somebody's business, written into a text box they reasonably assumed was private. They would have made a far more interesting post than this one.
There are no names, no product concepts and no prompt text anywhere above, and there will not be unless the person says yes.
If you would rather watch the agent working than read about it failing, there is an uncut run here: https://youtu.be/2sR4H1q9QCQ — the interesting part is at 1:12, where its own build check fails and it goes and fixes the config itself.