{"slug": "my-agent-s-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product", "title": "My agent's p50 was 29s. Its p95 was 182s. That ratio decided the product.", "summary": "A developer's analysis of 67 timed turns against a live LLM agent revealed a median response time of 29.2 seconds but a p95 of 182 seconds, a ratio that shaped the product design for Porchlight, an AI-powered win-back tool. The agent, built on Minds by Animoca Brands, resolves member departure reasons with high recall and precision, outperforming a keyword baseline on 11 of 14 resolved cases. The developer decided against synchronous fan-out due to the latency distribution, opting for precomputed verdicts.", "body_md": "I have 67 timed turns against a live LLM agent, captured in a single batch run and written to disk:\n\n| seconds | |\n|---|---|\n| p50 | 29.2 |\n| p95 | 182.0 |\n| max | 210.3 |\n| n | 67 completed turns |\n\nThe median says *background job, that's fine*. The p95 says *you may never put a human in front of this*.\n\nThose are not two performance notes. They are a product spec — and I found that out the expensive way, by designing the product first and measuring second.\n\nA member cancels, and the *reason* vanishes — nobody writes it down. Months later the creator fixes the exact thing that drove people away, and the people who left for that reason are never told. The state of the art is a \"we miss you\" blast to everyone.\n\nPorchlight puts an agent — [Minds by Animoca Brands](https://hellominds.ai) — on the critical path in three places: a short warm **exit interview** that files a structured return-condition in the member's own words; **condition matching**, which asks whether *this* announcement genuinely resolves *that* person's reason for leaving; and a **win-back draft** that quotes the member back to themselves.\n\nThe middle one is the step that has to be an agent, and I wanted to prove that rather than assert it.\n\nEvery \"AI-powered\" claim should ship its control. Mine is twenty lines, it lives in the repo, and it runs on the same inputs on every demo run:\n\n``` js\n// src/keywordBaseline.ts — the \"dumb tool\" strawman\nconst STOP = new Set(['the','and','are','was','were','you','your','for','that','this','with',\n  'have','has','had','not','but','now','all','its',\"it's\",'been','back','big','news','just',\n  'about','from','they','them','our','out','get','got','weekly','more'])\n\nconst tokens = (s: string): string[] =>\n  (s.toLowerCase().match(/[a-z][a-z'-]{3,}/g) ?? []).filter((w) => !STOP.has(w))\n\n/** True iff the parting quote and the announcement share at least one salient keyword. */\nexport function keywordResolves(changeText: string, verbatimQuote: string): boolean {\n  const a = new Set(tokens(changeText))\n  return tokens(verbatimQuote).some((w) => a.has(w))\n}\n```\n\nHere is a real pair from the seed data. A member left saying:\n\n\"the long chatty sit-downs with guests were the whole reason i was here, now it is quick clips\"\n\nand the creator later announced:\n\n\"Big news — the deep-dive interviews are back, weekly.\"\n\nSame event. After stopwords, the announcement contributes `{deep-dive, interviews}`\n\nand the quote contributes `{long, chatty, sit-downs, guests, whole, reason, here, quick, clips}`\n\n. The intersection is empty, so `keywordResolves`\n\nreturns `false`\n\n— and no amount of stopword tuning will ever link \"clips\" to \"deep-dive\". The agent resolves it, and explains why.\n\nAcross 54 captured judgements the agent resolved 14 departures, **11 of which the keyword baseline scores 0.00 on** — while refusing 35 non-matching pairs at ≥0.90 confidence. The recall is the pitch; the precision is what makes it safe to actually send. If you are emailing real people who already left once, a false positive is worse than a miss.\n\nFine. The agent is load-bearing. Now the bill.\n\nSorted, those 67 samples look like this: fastest 10.3s, a long fat body between 15s and 50s, a handful in the 60–115s range, then five clustered at ~182s, then one at 210.3s — that last one being a 180s client timeout followed by a successful retry.\n\nThat is not a distribution you can hide behind a spinner.\n\nThe architecture I had sketched before measuring: visitor clicks *announce a change*, the server fans out across every open departure, results render. With 18 departures that is 18 turns. At p50 that's about nine minutes. At p95 it's closer to an hour. And even a *single* turn — the best case in the whole design — is a coin flip between ten seconds and three minutes.\n\nThree decisions, all downstream of that one ratio.\n\n**1. No synchronous fan-out, ever.** The public demo replays verdicts captured ahead of time by a separate `npm run precompute`\n\npass, which writes them to `src/liveCache.json`\n\nwith a `capturedAt`\n\nstamp on each one. Every verdict a visitor sees is real agent output; none of it is computed while they wait. The UI says when it was captured, because a replay that pretends to be live is a lie.\n\n**2. The one genuinely live path is bounded and rationed.**\n\n```\n/** Longest a visitor is asked to wait on a live turn before we give up on it. */\nconst WEB_DEADLINE_MS = 100_000\n\nfunction withDeadline<T>(work: Promise<T>, ms: number): Promise<T> {\n  return Promise.race([\n    work,\n    new Promise<never>((_, reject) =>\n      setTimeout(() => reject(new Error(`no reply within ${Math.round(ms / 1000)}s`)), ms).unref(),\n    ),\n  ])\n}\n```\n\n100 seconds is not a round number I liked; it is p50 with real headroom and deliberately *below* the 182s p95. It gives up on the slow tail on purpose rather than holding a browser open for three minutes. Some requests do fail, and the error message says exactly that — that this is a real call to a real agent and sometimes it is slow. Paired with 3 live calls per IP per 15 minutes, and scoped to one member the visitor picks rather than a fan-out.\n\n**3. No keyword fallback in the deployed app.** This is the decision I'd defend hardest. When the agent is slow or unreachable, the tempting move is to fall back to the cheap path — you always have one, because you built it as the control. But the cheap path is the exact mechanism the product exists to beat. Falling back to it means quietly shipping the strawman under the good name, and nobody would ever know. With no credentials the service returns `503`\n\nand says why.\n\n**A stale reply is a silent correctness bug.** Send-then-wait reads like it should just work:\n\n``` js\nconst before = await c.getLatestHistoryFingerprint(alias).catch(() => undefined)\nawait c.sendMessage({ alias, messageText: text })\nconst outcome = await c.waitForReply({\n  alias, timeoutMs: CONFIG.replyTimeoutMs,\n  afterFingerprint: before,      // captured BEFORE the send\n  sentMessageText: text,\n})\n```\n\nDrop `afterFingerprint`\n\nand you can be handed the *previous* turn's reply. It does not throw. It returns a completely plausible answer to a question you did not ask. In a system whose entire job is per-member judgement, that is a wrong email to a real person, and it is the worst class of bug to debug because nothing anywhere looks broken.\n\n**No JSON mode means scraping prose.** There is no schema/response-format option, so structured output means asking for JSON in the prompt, then going and finding it — `indexOf('{')`\n\n, `lastIndexOf('}')`\n\n, `JSON.parse`\n\nthe slice, hand it to Zod. It works, and it is brittle by construction: the reply can preface the JSON with commentary, fence it, or emit two objects. (Replies also arrive as HTML, which is its own small adventure in stripping tags *after* decoding entities rather than before.)\n\nSame agent, same prompt, same member quote, run weeks apart. A departure that said:\n\n\"you stopped doing the long-form lore videos i subscribed for\"\n\njudged against *\"the deep-dive interviews are back, weekly\"* resolved **true** in an early run and **false, confidence 0.60** in the full capture, with this rationale:\n\n\"The announcement restores long-form content but specifically as deep-dive interviews, not the lore videos the member subscribed for; the subject-matter mismatch means the member's core interest in lore is likely still unmet.\"\n\nThe second answer is better than the first. That is not the point. The point is that I had a recovered-revenue figure in my README that read like a constant, and it is not one — it is a snapshot of a single run. There is no temperature or seed exposed, so I cannot opt into determinism even where I'd want it.\n\nSo I rewrote the README to say the figures are per-run, and published the traces in both directions. If you derive a metric from a batch of LLM judgements, you have measured *that run*. Say so in the same sentence as the number, or someone will eventually try to reproduce it and conclude you made it up.\n\n`fetch failed`\n\nand succeeded on retry. With no error code or `retryable`\n\nflag, every failure has to be treated as retryable — which is wrong for 4xx-class problems.On rigor rather than as the story: 58 tests, 100% line/branch/function coverage, Playwright E2E, and a CI stage that fails the build if the deployed app comes up in mock mode.\n\nMeasure the tail before you draw the architecture. The p50 is a comfort; the p95 is the constraint. Mine bought a precompute cache, a 100-second deadline, a rate limit, and a refusal to ever fall back to the dumb path — and in retrospect those four decisions *are* most of the engineering.\n\nEverything the SDK taught me, latency data included, is in [FEEDBACK.md](https://github.com/edycutjong/porchlight/blob/main/FEEDBACK.md). To reproduce the numbers: `npm run precompute`\n\nwrites per-call timings straight into `src/liveCache.json`\n\n.\n\n**Code:** [github.com/edycutjong/porchlight](https://github.com/edycutjong/porchlight) · **Live:** [porchlight.edycu.dev](https://porchlight.edycu.dev) · **Try the real agent:** [try.porchlight.edycu.dev](https://try.porchlight.edycu.dev)\n\nThe most interesting thing you can do in the sandbox is try to fool it — describe a fix that *shouldn't* win someone back, and see whether it stays quiet. If it holds up, a star helps.", "url": "https://wpnews.pro/news/my-agent-s-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product", "canonical_source": "https://dev.to/edycutjong/my-agents-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product-c7n", "published_at": "2026-08-27 05:09:14+00:00", "updated_at": "2026-08-27 05:18:18.949786+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "developer-tools"], "entities": ["Porchlight", "Minds", "Animoca Brands"], "alternates": {"html": "https://wpnews.pro/news/my-agent-s-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product", "markdown": "https://wpnews.pro/news/my-agent-s-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product.md", "text": "https://wpnews.pro/news/my-agent-s-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product.txt", "jsonld": "https://wpnews.pro/news/my-agent-s-p50-was-29s-its-p95-was-182s-that-ratio-decided-the-product.jsonld"}}