I spent months building an AI agent platform. I launched it on Product Hunt yesterday. Zero signups.
The comments were friendly. Three of the four asked for the same thing — not features, not integrations, not a lower price. They wanted to see what the agents did and what they cost. One put it better than my own landing page ever did: they liked that it wasn't "a black box."
So I went to make the cost dashboard better. Instead I found out my product had been lying to me for months, and the lies had a pattern.
Here's everything, with the code.
The Cost Analytics page reported $0.02 in total across ~100 executions. I'd assumed that meant the platform was cheap to run. It meant the data was being destroyed at write time.
cost_cents = int(
(llm_response.prompt_tokens * 0.5 / 1000)
+ (llm_response.completion_tokens * 1.5 / 1000)
)
A typical run on my platform is 56 prompt tokens and 45 completion tokens. That's 0.0843 cents. int()
makes it 0
.
Not some runs. Essentially every run — because almost every LLM call costs less than one cent. The production numbers: 189 agent runs, 2 with a non-zero cost. 99% of my cost data was zeroes, and the two survivors were just big enough to clear a whole cent.
The rates in that formula were correct. I checked them against the providers' pricing pages; the arithmetic is right. The bug is entirely int()
on a value that is almost never ≥ 1. A Decimal
would have been the textbook fix, but Decimal / float
raises TypeError
and ~80 call sites do arithmetic on this number, so I widened the column to a float and kept the unit (cents). It's a dashboard estimate, not money — Paddle handles money — so float rounding is irrelevant here.
Truncation at least loses precision. This one lost everything.
WorkflowExecution.total_cost_cents
and total_tokens_used
had no write site anywhere in the codebase. Not a broken write — no write. The columns had been NULL since the feature shipped. The workflow runner computed per-step token counts and dropped them on the floor.
I found this by grepping for assignments and getting no hits:
$ grep -rn "\.total_cost_cents = " src/
$ # (nothing)
The API served the field. The schema declared it. The dashboard summed it. It was never once set.
if llm_response.provider == "openai": # gpt-4o-mini billed at gpt-4o rates
Same provider, ~30x cheaper model, same price. Nobody noticed because of #1 — every number was already zero.
I fixed the data. Then I went to display it and found costCents
had been in the API response for months and was rendered nowhere in the frontend. Not on the run page, not anywhere:
$ grep -rn "costCents" src/ --include=*.tsx
$ # (nothing)
The TypeScript types never declared the field either — so the API was returning data that was invisible to the compiler. No error. Just absent.
And the workflow step list rendered this:
{step.tokensUsed && <span>{step.tokensUsed} tokens</span>}
WorkflowStepExecution
has no tokensUsed
column. The response schema has no such field. That line had never rendered once. It was in the UI, in code review, in the repo, doing nothing.
This is the one that stings.
The app has four analytics pages — Dashboard, Executions, Costs, System. They work. They have a nice sub-nav between them.
Nothing in the entire application linked to any of them. Not the main nav (Dashboard, Agents, Workflows, Executions, Marketplace, Teams, Schedules, Billing). Not the user menu. The only way in was typing the URL. And once you got there, there was no way back except the browser button.
The $0.02 that started this whole investigation was on a page no user could click to.
When I fixed it, I added the back-link to the shared AnalyticsLayout
component. Then a browser test told me the link wasn't there. AnalyticsLayout
is dead code — all four pages only import a sub-component from that file; the layout itself is never rendered. My fix for the unreachable feature went into unreachable code. I only caught it because I drove a real browser instead of trusting the diff.
While making Analytics reachable, I exposed a System Health tab to every logged-in user. Looking at a screenshot of my own fix, I stopped: should a normal user see system health?
It was worse than the question. /v1/health/detailed
had no auth dependency at all, and the reverse proxy sends /v1/*
straight to the backend. Anyone on the internet, no account, could poll:
uptime · Postgres latency · Redis latency · Celery worker count
WebSocket → connections: 1, users: 1 ← live user count
A real-time traffic gauge for my business, free to anyone who cared to watch. Public since the day it was written.
Why it survived every test: the test suite's conftest overrides get_current_user
. Any test using the shared client fixture authenticates as a fake user — so a missing auth dependency looks identical to a working one. The endpoint would have passed an auth test. The regression test I wrote deliberately bypasses that fixture:
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
resp = await ac.get("/v1/health/detailed") # no fixture, no override
assert resp.status_code in (401, 403)
Five separate bugs, one shape: built, tested, shipped, unreachable.
costCents
→ returned, never renderedcurrent_step_id
→ tracked in the DB, unused by the frontend (still true; it's next)Every one passed CI. 519 backend tests, 276 frontend tests, typecheck clean. Every one had a plausible-looking implementation. None of them worked end to end, and no test could have caught any of them, because each piece did exactly what its unit test said it should. The column stored an int. The endpoint returned JSON. The component rendered a field. The wiring between them was where the product lived, and nothing tested the wiring.
That's not bad luck. Five in one week is a process failure. Mine was that I verified changes and never verified features — I'd check that the code did what I wrote, not that a user could get to it and see a true number at the end.
The thing that actually found these was mundane: driving the real app, as a real user, and looking at the screen. Every single one was invisible from inside the codebase and obvious from the browser.
A test that mocks your auth cannot test your auth. The fixture that makes tests convenient is the fixture that hides the hole.
"The API returns it" is not a feature. Nobody can see your JSON.
Grep for the write, not the read. A field being read in 80 places says nothing about whether anything ever writes it.
If a number looks suspiciously good, it's probably a bug. $0.02 across 100 runs should have made me suspicious months earlier. I filed it under "efficient" instead of "broken."
Click the thing. Not the diff. Not the test. The thing.
Still zero signups. The observability is honest now, the costs are real, and the pages are reachable — and none of that changed the number, because a product being correct was never what was stopping people. That's a different post, and I haven't earned it yet.
But I'd rather have found all this from four polite comments than from the first customer who actually paid.
The product is AgentMesh — no-code AI agents for small businesses. Zero strangers have ever signed up, so please don't take this as a recommendation. Take it as a warning about your own repo.