I Split the Wait. The Wire Ate P99.

A developer built a phase-split latency harness that separates a remote generate call into named clocks — connect, time-to-first-byte, body, patch apply, and pytest — to show that P99 stalls often come from the wire (TLS handshakes and cold sockets) rather than token decode. The stdlib-only Python tool logs one JSON line per run with phase stamps, status codes, byte counts, and test exit codes, and the author recommends pairing curl for handshake timing with Python for tail percentiles. The work was prepared as part of MonkeyCode's product outreach for its open-source coding assistant.

The generate call was not the slow part. The wait around it was. I stopped saying the model felt slow. That sentence is a shrug. It hides DNS, TLS, queueing, and decode together. You ever stare at a spinner and blame tokens? I did, for too long. Then I split the wait into clocks I could name. P99 jumped out of the handshake. The body looked almost polite. This is not a take on AI slogans. I do not need another slogan. I need a phase I can point at. If you cannot name the phase, you are guessing. I work in a tight loop. Prompt, remote generate, apply a patch, run one test. The happy path looks fine in demos. The ugly path lives in p99. That is the graph I kept. A mean latency is a lullaby. It soothes a standup. It also hides the stall that kills flow. I wanted a CDF, not a vibe. Fifty samples first. Then two hundred. I watched p50 and p99 diverge. That gap is the whole story. Think of a subway commute. Average wait looks civil on paper. One jammed train ruins the morning. Remote inference has that jammed train. It is often TLS or a cold socket. Sometimes it is a queue you never see. Local metal will not show this tax. A laptop model skips the wire. That is why a remote path belongs in this lab. You need the network in the picture. Hide the wire and you will blame tokens forever. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option. A free remote path is enough to put the wire on the chart. Point this harness at that option, or at any URL you already trust. I am not selling a bake-off. I am selling a habit. Stamp the phases. Keep the CDF. Change one variable. Blame tokens last. I picked a small coding task on purpose. Explain a failing test. Ask for a patch. Apply it. Run pytest on one file. I did not chase quality scores. Quality is another lab. This lab is time. If the patch is nonsense, the test phase still counts. Failed tests are still clocks. Each run wrote one JSON line. Phase stamps. Status code. Byte count. Test exit. No model names in the log. The log is about clocks, not brands. I ran a dry-run first. Then a live endpoint. The mock taught me my client was sloppy. Connection reuse was off. That bug would have blamed the model. Sound familiar? I use two tools, not one dashboard. curl splits the handshake. Python keeps the CDF. urllib will not give you honest DNS on every box. I refuse to invent that column. curl -w still can. So I let curl name TLS. I let Python name the tail across many loops. Save the Python as split wait.py . Stdlib only. No extra install. bash /usr/bin/env python3 """Phase-split timer for a generate call plus a local test. Dry-run prints a shape. Live mode needs --url. This is a lab harness, not a vendor benchmark. """ from future import annotations import argparse import json import random import statistics import sys import time import urllib.error import urllib.request from dataclasses import asdict, dataclass @dataclass class Sample: connect ms: float ttfb ms: float body ms: float apply ms: float pytest ms: float bytes in: int status: int dry run: bool def pct xs, q : if not xs: return 0.0 ys = sorted xs i = min len ys - 1, max 0, int round q / 100 len ys - 1 return ys i def dry sample : Labeled synthetic shape for the article. Not a measurement. def jitter mid, spread : return max 0.5, random.gauss mid, spread ttfb = jitter 130, 35 if random.random < 0.05: ttfb += 180 fake tail, dry-run only return Sample connect ms=jitter 18, 8 , ttfb ms=ttfb, body ms=jitter 95, 20 , apply ms=jitter 7, 2 , pytest ms=jitter 90, 15 , bytes in=2048, status=200, dry run=True, def live sample url, payload, timeout : t0 = time.perf counter req = urllib.request.Request url, data=payload, method="POST" req.add header "Content-Type", "application/json" try: with urllib.request.urlopen req, timeout=timeout as resp: ttfb = time.perf counter data = resp.read t1 = time.perf counter status = getattr resp, "status", 200 except urllib.error.URLError as exc: raise SystemExit f"request failed: {exc}" from exc t apply0 = time.perf counter time.sleep 0.005 stand-in for patch apply apply ms = time.perf counter - t apply0 1000 t py0 = time.perf counter time.sleep 0.02 replace with subprocess pytest pytest ms = time.perf counter - t py0 1000 return Sample connect ms= ttfb - t0 1000, ttfb ms= ttfb - t0 1000, body ms= t1 - ttfb 1000, apply ms=apply ms, pytest ms=pytest ms, bytes in=len data , status=status, dry run=False, def summarize path : rows = with open path, encoding="utf-8" as f: for line in f: line = line.strip if line: rows.append json.loads line if not rows: print "no rows" return phases = "connect ms", "ttfb ms", "body ms", "apply ms", "pytest ms" print f"n={len rows } dry run={rows 0 .get 'dry run' }" print f"{'phase':<12} {'p50 ms': 8} {'p99 ms': 8}" for p in phases: xs = r p for r in rows print f"{p:<12} {pct xs, 50 :8.1f} {pct xs, 99 :8.1f}" def main : ap = argparse.ArgumentParser description="Split wait time. Blame tokens last." ap.add argument "--url" ap.add argument "--n", type=int, default=50 ap.add argument "--out", default="wait.jsonl" ap.add argument "--dry-run", action="store true" ap.add argument "--summarize" ap.add argument "--timeout", type=float, default=30.0 args = ap.parse args if args.summarize: summarize args.summarize return payload = json.dumps {"prompt": "lab: do not send secrets"} .encode with open args.out, "w", encoding="utf-8" as out: for i in range args.n : if args.dry run or not args.url: sample = dry sample else: sample = live sample args.url, payload, args.timeout out.write json.dumps asdict sample + "\n" print f"wrote {i + 1}/{args.n}", file=sys.stderr summarize args.out if name == " main ": main Run the dry-run until the file format bores you. Then point it at a real URL. python split wait.py --dry-run --n 50 --out wait.jsonl python split wait.py --summarize wait.jsonl Live mode is one flag more. Keep secrets out of the payload. python split wait.py --url "$ENDPOINT" --n 50 --out wait.jsonl The summarize command prints a CDF slice. P50 and p99 side by side. I keep that pair. The mean can wait in the hallway. Handshake detail still belongs to curl. Python told me the tail existed. Curl told me where the handshake sat. I keep this next to the harness. curl -s -o /dev/null \ -w "dns:%{time namelookup} connect:%{time connect} tls:%{time appconnect} ttfb:%{time starttransfer} total:%{time total}\n" \ "$ENDPOINT" Run that twenty times on a cold client. Then run a reused session. Compare the tls field. You will feel the subway metaphor in your hands. Here is a dry-run shape from the harness. Treat it as a shape, not a benchmark. I generated it with --dry-run . It is not a product claim. It is not my laptop's truth either. n=50 dry run=True phase p50 ms p99 ms connect ms 17.4 41.2 ttfb ms 128.0 312.7 body ms 94.1 148.6 apply ms 6.8 12.1 pytest ms 89.4 129.0 Look at TTFB p99 in that shape. That is the jammed train. Tokens did not do that. Body did not do that. Pytest did not do that. I kept that table anyway, as a reminder of the form. Live numbers have to come from your endpoint. Mine will not travel. What did I change after I trusted the split? First I enabled keep-alive. One session. Many calls. Connect tails shrank. P50 barely moved. P99 fell hard. That is the lesson in one move. Averages hide reuse. Tails reveal it. Then I stopped opening a new client per prompt. Sounds obvious in a code review. Watch your agent loop anyway. Many wrappers build a fresh client every turn. They look clean. They tax the handshake. The spinner still smiles. I added a cheap probe too. A tiny POST with no real prompt. If the probe is already fat, skip the model blame. Fix the path. I also logged bytes. A fat prompt can look like a slow decode. It is often upload. Split TTFB from body or you will misread it. I do not keep a pretty dashboard for this. I keep a two-line CDF. P50 as a calm line. P99 as a rude one. When they kiss, the path is honest. When they split, I hunt the phase with the bigger gap. For a cold remote path, the gap often sits before the first byte. Decode can win on another day. Then the CDF would say so. I would believe the CDF. I would not believe the spinner. Here is the decision I actually use after the chart settles. If p99 lives in connect or TLS, reuse the client. If p99 lives in TTFB, suspect queueing or a cold remote. If p99 lives in body, then you may talk about tokens. If p99 lives in pytest, stop tuning prompts. Fix the test. That last one stings. I have tuned prompts while the suite did extra I/O. The chart called me out. Free remote inference is a spotlight, not a trophy. A free server puts the wire in your lap. You see handshake tax. You see cold starts. You see what a laptop model will never show. Spend the free samples on the tail. Do not spend them on another debate about whether AI coding counts as engineering. The clocks do not care. This split does not grade models. It grades your wait. Do not use it to rank vendors. Do not use it as an SLA. Do not paste secrets into the payload. A free remote path is still a remote path. Redact prompts. Use a dummy repo. If your company forbids egress, stop. This is not for you. Do not treat dry-run numbers as evidence. They teach shape. Live numbers are yours. If you need guaranteed latency, this is the wrong lab. If you need eval scores, this is the wrong lab. If you just want to ship a patch tonight, skip the CDF. Write the patch. The method also assumes you can name phases. If your SDK hides HTTP, you will blame the SDK. That is still useful. Wrap lower. If you cannot log bytes, you will confuse upload with decode. Log bytes. I will not quote a speedup factor. I already learned that lesson elsewhere. I will not name models I did not pin. I will not invent quotas. I will not pretend a free server is permanent. Availability changes. The split still works on the next URL. I will not tell you AI coding is engineering. I will not tell you it is not. I will tell you where my seconds went. You can pick the slogan later. Bring a CDF to that argument or stay home. The spinner was a liar. The CDF was not. Split the wait. Keep p99. Change one variable. Blame tokens last.