{"slug": "i-split-the-wait-the-wire-ate-p99", "title": "I Split the Wait. The Wire Ate P99.", "summary": "A developer built a phase-split latency harness that separates a remote generate call into named clocks — connect, time-to-first-byte, body, patch apply, and pytest — to show that P99 stalls often come from the wire (TLS handshakes and cold sockets) rather than token decode. The stdlib-only Python tool logs one JSON line per run with phase stamps, status codes, byte counts, and test exit codes, and the author recommends pairing curl for handshake timing with Python for tail percentiles. The work was prepared as part of MonkeyCode's product outreach for its open-source coding assistant.", "body_md": "The generate call was not the slow part. The wait around it was.\n\nI stopped saying the model felt slow. That sentence is a shrug. It hides DNS, TLS, queueing, and decode together.\n\nYou ever stare at a spinner and blame tokens? I did, for too long. Then I split the wait into clocks I could name. P99 jumped out of the handshake. The body looked almost polite.\n\nThis is not a take on AI slogans. I do not need another slogan. I need a phase I can point at. If you cannot name the phase, you are guessing.\n\nI work in a tight loop. Prompt, remote generate, apply a patch, run one test. The happy path looks fine in demos. The ugly path lives in p99. That is the graph I kept.\n\nA mean latency is a lullaby. It soothes a standup. It also hides the stall that kills flow. I wanted a CDF, not a vibe. Fifty samples first. Then two hundred. I watched p50 and p99 diverge. That gap is the whole story.\n\nThink of a subway commute. Average wait looks civil on paper. One jammed train ruins the morning. Remote inference has that jammed train. It is often TLS or a cold socket. Sometimes it is a queue you never see.\n\nLocal metal will not show this tax. A laptop model skips the wire. That is why a remote path belongs in this lab. You need the network in the picture. Hide the wire and you will blame tokens forever.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option. A free remote path is enough to put the wire on the chart. Point this harness at that option, or at any URL you already trust.\n\nI am not selling a bake-off. I am selling a habit. Stamp the phases. Keep the CDF. Change one variable. Blame tokens last.\n\nI picked a small coding task on purpose. Explain a failing test. Ask for a patch. Apply it. Run pytest on one file. I did not chase quality scores. Quality is another lab. This lab is time. If the patch is nonsense, the test phase still counts. Failed tests are still clocks.\n\nEach run wrote one JSON line. Phase stamps. Status code. Byte count. Test exit. No model names in the log. The log is about clocks, not brands. I ran a dry-run first. Then a live endpoint. The mock taught me my client was sloppy. Connection reuse was off. That bug would have blamed the model. Sound familiar?\n\nI use two tools, not one dashboard. `curl` splits the handshake. Python keeps the CDF. urllib will not give you honest DNS on every box. I refuse to invent that column. `curl -w` still can. So I let curl name TLS. I let Python name the tail across many loops.\n\nSave the Python as `split_wait.py`. Stdlib only. No extra install.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Phase-split timer for a generate call plus a local test.\n\nDry-run prints a shape. Live mode needs --url.\nThis is a lab harness, not a vendor benchmark.\n\"\"\"\nfrom __future__ import annotations\n\nimport argparse\nimport json\nimport random\nimport statistics\nimport sys\nimport time\nimport urllib.error\nimport urllib.request\nfrom dataclasses import asdict, dataclass\n\n@dataclass\nclass Sample:\n    connect_ms: float\n    ttfb_ms: float\n    body_ms: float\n    apply_ms: float\n    pytest_ms: float\n    bytes_in: int\n    status: int\n    dry_run: bool\n\ndef pct(xs, q):\n    if not xs:\n        return 0.0\n    ys = sorted(xs)\n    i = min(len(ys) - 1, max(0, int(round((q / 100) * (len(ys) - 1)))))\n    return ys[i]\n\ndef dry_sample():\n    # Labeled synthetic shape for the article. Not a measurement.\n    def jitter(mid, spread):\n        return max(0.5, random.gauss(mid, spread))\n\n    ttfb = jitter(130, 35)\n    if random.random() < 0.05:\n        ttfb += 180  # fake tail, dry-run only\n    return Sample(\n        connect_ms=jitter(18, 8),\n        ttfb_ms=ttfb,\n        body_ms=jitter(95, 20),\n        apply_ms=jitter(7, 2),\n        pytest_ms=jitter(90, 15),\n        bytes_in=2048,\n        status=200,\n        dry_run=True,\n    )\n\ndef live_sample(url, payload, timeout):\n    t0 = time.perf_counter()\n    req = urllib.request.Request(url, data=payload, method=\"POST\")\n    req.add_header(\"Content-Type\", \"application/json\")\n    try:\n        with urllib.request.urlopen(req, timeout=timeout) as resp:\n            ttfb = time.perf_counter()\n            data = resp.read()\n            t1 = time.perf_counter()\n            status = getattr(resp, \"status\", 200)\n    except urllib.error.URLError as exc:\n        raise SystemExit(f\"request failed: {exc}\") from exc\n\n    t_apply0 = time.perf_counter()\n    time.sleep(0.005)  # stand-in for patch apply\n    apply_ms = (time.perf_counter() - t_apply0) * 1000\n\n    t_py0 = time.perf_counter()\n    time.sleep(0.02)  # replace with subprocess pytest\n    pytest_ms = (time.perf_counter() - t_py0) * 1000\n\n    return Sample(\n        connect_ms=(ttfb - t0) * 1000,\n        ttfb_ms=(ttfb - t0) * 1000,\n        body_ms=(t1 - ttfb) * 1000,\n        apply_ms=apply_ms,\n        pytest_ms=pytest_ms,\n        bytes_in=len(data),\n        status=status,\n        dry_run=False,\n    )\n\ndef summarize(path):\n    rows = []\n    with open(path, encoding=\"utf-8\") as f:\n        for line in f:\n            line = line.strip()\n            if line:\n                rows.append(json.loads(line))\n    if not rows:\n        print(\"no rows\")\n        return\n    phases = [\"connect_ms\", \"ttfb_ms\", \"body_ms\", \"apply_ms\", \"pytest_ms\"]\n    print(f\"n={len(rows)}  dry_run={rows[0].get('dry_run')}\")\n    print(f\"{'phase':<12} {'p50_ms':>8} {'p99_ms':>8}\")\n    for p in phases:\n        xs = [r[p] for r in rows]\n        print(f\"{p:<12} {pct(xs, 50):8.1f} {pct(xs, 99):8.1f}\")\n\ndef main():\n    ap = argparse.ArgumentParser(description=\"Split wait time. Blame tokens last.\")\n    ap.add_argument(\"--url\")\n    ap.add_argument(\"--n\", type=int, default=50)\n    ap.add_argument(\"--out\", default=\"wait.jsonl\")\n    ap.add_argument(\"--dry-run\", action=\"store_true\")\n    ap.add_argument(\"--summarize\")\n    ap.add_argument(\"--timeout\", type=float, default=30.0)\n    args = ap.parse_args()\n    if args.summarize:\n        summarize(args.summarize)\n        return\n    payload = json.dumps({\"prompt\": \"lab: do not send secrets\"}).encode()\n    with open(args.out, \"w\", encoding=\"utf-8\") as out:\n        for i in range(args.n):\n            if args.dry_run or not args.url:\n                sample = dry_sample()\n            else:\n                sample = live_sample(args.url, payload, args.timeout)\n            out.write(json.dumps(asdict(sample)) + \"\\n\")\n            print(f\"wrote {i + 1}/{args.n}\", file=sys.stderr)\n    summarize(args.out)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun the dry-run until the file format bores you. Then point it at a real URL.\n\n```\npython split_wait.py --dry-run --n 50 --out wait.jsonl\npython split_wait.py --summarize wait.jsonl\n```\n\nLive mode is one flag more. Keep secrets out of the payload.\n\n```\npython split_wait.py --url \"$ENDPOINT\" --n 50 --out wait.jsonl\n```\n\nThe summarize command prints a CDF slice. P50 and p99 side by side. I keep that pair. The mean can wait in the hallway.\n\nHandshake detail still belongs to curl. Python told me the tail existed. Curl told me where the handshake sat. I keep this next to the harness.\n\n```\ncurl -s -o /dev/null \\\n  -w \"dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\\n\" \\\n  \"$ENDPOINT\"\n```\n\nRun that twenty times on a cold client. Then run a reused session. Compare the tls field. You will feel the subway metaphor in your hands.\n\nHere is a dry-run shape from the harness. Treat it as a shape, not a benchmark. I generated it with `--dry-run`. It is not a product claim. It is not my laptop's truth either.\n\n```\nn=50  dry_run=True\nphase          p50_ms   p99_ms\nconnect_ms       17.4     41.2\nttfb_ms         128.0    312.7\nbody_ms          94.1    148.6\napply_ms          6.8     12.1\npytest_ms        89.4    129.0\n```\n\nLook at TTFB p99 in that shape. That is the jammed train. Tokens did not do that. Body did not do that. Pytest did not do that. I kept that table anyway, as a reminder of the form. Live numbers have to come from your endpoint. Mine will not travel.\n\nWhat did I change after I trusted the split? First I enabled keep-alive. One session. Many calls. Connect tails shrank. P50 barely moved. P99 fell hard. That is the lesson in one move. Averages hide reuse. Tails reveal it.\n\nThen I stopped opening a new client per prompt. Sounds obvious in a code review. Watch your agent loop anyway. Many wrappers build a fresh client every turn. They look clean. They tax the handshake. The spinner still smiles.\n\nI added a cheap probe too. A tiny POST with no real prompt. If the probe is already fat, skip the model blame. Fix the path. I also logged bytes. A fat prompt can look like a slow decode. It is often upload. Split TTFB from body or you will misread it.\n\nI do not keep a pretty dashboard for this. I keep a two-line CDF. P50 as a calm line. P99 as a rude one. When they kiss, the path is honest. When they split, I hunt the phase with the bigger gap. For a cold remote path, the gap often sits before the first byte. Decode can win on another day. Then the CDF would say so. I would believe the CDF. I would not believe the spinner.\n\nHere is the decision I actually use after the chart settles. If p99 lives in connect or TLS, reuse the client. If p99 lives in TTFB, suspect queueing or a cold remote. If p99 lives in body, then you may talk about tokens. If p99 lives in pytest, stop tuning prompts. Fix the test. That last one stings. I have tuned prompts while the suite did extra I/O. The chart called me out.\n\nFree remote inference is a spotlight, not a trophy. A free server puts the wire in your lap. You see handshake tax. You see cold starts. You see what a laptop model will never show. Spend the free samples on the tail. Do not spend them on another debate about whether AI coding counts as engineering. The clocks do not care.\n\nThis split does not grade models. It grades your wait. Do not use it to rank vendors. Do not use it as an SLA. Do not paste secrets into the payload. A free remote path is still a remote path. Redact prompts. Use a dummy repo. If your company forbids egress, stop. This is not for you.\n\nDo not treat dry-run numbers as evidence. They teach shape. Live numbers are yours. If you need guaranteed latency, this is the wrong lab. If you need eval scores, this is the wrong lab. If you just want to ship a patch tonight, skip the CDF. Write the patch.\n\nThe method also assumes you can name phases. If your SDK hides HTTP, you will blame the SDK. That is still useful. Wrap lower. If you cannot log bytes, you will confuse upload with decode. Log bytes.\n\nI will not quote a speedup factor. I already learned that lesson elsewhere. I will not name models I did not pin. I will not invent quotas. I will not pretend a free server is permanent. Availability changes. The split still works on the next URL.\n\nI will not tell you AI coding is engineering. I will not tell you it is not. I will tell you where my seconds went. You can pick the slogan later. Bring a CDF to that argument or stay home.\n\nThe spinner was a liar. The CDF was not. Split the wait. Keep p99. Change one variable. Blame tokens last.", "url": "https://wpnews.pro/news/i-split-the-wait-the-wire-ate-p99", "canonical_source": "https://dev.to/apppro_4800/i-split-the-wait-the-wire-ate-p99-5eo5", "published_at": "2026-09-19 09:32:00+00:00", "updated_at": "2026-09-19 09:54:21.456687+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["MonkeyCode", "Python", "curl", "pytest", "urllib"], "alternates": {"html": "https://wpnews.pro/news/i-split-the-wait-the-wire-ate-p99", "markdown": "https://wpnews.pro/news/i-split-the-wait-the-wire-ate-p99.md", "text": "https://wpnews.pro/news/i-split-the-wait-the-wire-ate-p99.txt", "jsonld": "https://wpnews.pro/news/i-split-the-wait-the-wire-ate-p99.jsonld"}}