The generate call was not the slow part. The wait around it was.
I stopped saying the model felt slow. That sentence is a shrug. It hides DNS, TLS, queueing, and decode together.
You ever stare at a spinner and blame tokens? I did, for too long. Then I split the wait into clocks I could name. P99 jumped out of the handshake. The body looked almost polite.
This is not a take on AI slogans. I do not need another slogan. I need a phase I can point at. If you cannot name the phase, you are guessing.
I work in a tight loop. Prompt, remote generate, apply a patch, run one test. The happy path looks fine in demos. The ugly path lives in p99. That is the graph I kept.
A mean latency is a lullaby. It soothes a standup. It also hides the stall that kills flow. I wanted a CDF, not a vibe. Fifty samples first. Then two hundred. I watched p50 and p99 diverge. That gap is the whole story.
Think of a subway commute. Average wait looks civil on paper. One jammed train ruins the morning. Remote inference has that jammed train. It is often TLS or a cold socket. Sometimes it is a queue you never see.
Local metal will not show this tax. A laptop model skips the wire. That is why a remote path belongs in this lab. You need the network in the picture. Hide the wire and you will blame tokens forever.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option. A free remote path is enough to put the wire on the chart. Point this harness at that option, or at any URL you already trust.
I am not selling a bake-off. I am selling a habit. Stamp the phases. Keep the CDF. Change one variable. Blame tokens last.
I picked a small coding task on purpose. Explain a failing test. Ask for a patch. Apply it. Run pytest on one file. I did not chase quality scores. Quality is another lab. This lab is time. If the patch is nonsense, the test phase still counts. Failed tests are still clocks.
Each run wrote one JSON line. Phase stamps. Status code. Byte count. Test exit. No model names in the log. The log is about clocks, not brands. I ran a dry-run first. Then a live endpoint. The mock taught me my client was sloppy. Connection reuse was off. That bug would have blamed the model. Sound familiar?
I use two tools, not one dashboard. curl splits the handshake. Python keeps the CDF. urllib will not give you honest DNS on every box. I refuse to invent that column. curl -w still can. So I let curl name TLS. I let Python name the tail across many loops.
Save the Python as split_wait.py. Stdlib only. No extra install.
#!/usr/bin/env python3
"""Phase-split timer for a generate call plus a local test.
Dry-run prints a shape. Live mode needs --url.
This is a lab harness, not a vendor benchmark.
"""
from __future__ import annotations
import argparse
import json
import random
import statistics
import sys
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass
@dataclass
class Sample:
connect_ms: float
ttfb_ms: float
body_ms: float
apply_ms: float
pytest_ms: float
bytes_in: int
status: int
dry_run: bool
def pct(xs, q):
if not xs:
return 0.0
ys = sorted(xs)
i = min(len(ys) - 1, max(0, int(round((q / 100) * (len(ys) - 1)))))
return ys[i]
def dry_sample():
def jitter(mid, spread):
return max(0.5, random.gauss(mid, spread))
ttfb = jitter(130, 35)
if random.random() < 0.05:
ttfb += 180 # fake tail, dry-run only
return Sample(
connect_ms=jitter(18, 8),
ttfb_ms=ttfb,
body_ms=jitter(95, 20),
apply_ms=jitter(7, 2),
pytest_ms=jitter(90, 15),
bytes_in=2048,
status=200,
dry_run=True,
)
def live_sample(url, payload, timeout):
t0 = time.perf_counter()
req = urllib.request.Request(url, data=payload, method="POST")
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
ttfb = time.perf_counter()
data = resp.read()
t1 = time.perf_counter()
status = getattr(resp, "status", 200)
except urllib.error.URLError as exc:
raise SystemExit(f"request failed: {exc}") from exc
t_apply0 = time.perf_counter()
time.sleep(0.005) # stand-in for patch apply
apply_ms = (time.perf_counter() - t_apply0) * 1000
t_py0 = time.perf_counter()
time.sleep(0.02) # replace with subprocess pytest
pytest_ms = (time.perf_counter() - t_py0) * 1000
return Sample(
connect_ms=(ttfb - t0) * 1000,
ttfb_ms=(ttfb - t0) * 1000,
body_ms=(t1 - ttfb) * 1000,
apply_ms=apply_ms,
pytest_ms=pytest_ms,
bytes_in=len(data),
status=status,
dry_run=False,
)
def summarize(path):
rows = []
with open(path, encoding="utf-8") as f:
for line in f:
line = line.strip()
if line:
rows.append(json.loads(line))
if not rows:
print("no rows")
return
phases = ["connect_ms", "ttfb_ms", "body_ms", "apply_ms", "pytest_ms"]
print(f"n={len(rows)} dry_run={rows[0].get('dry_run')}")
print(f"{'phase':<12} {'p50_ms':>8} {'p99_ms':>8}")
for p in phases:
xs = [r[p] for r in rows]
print(f"{p:<12} {pct(xs, 50):8.1f} {pct(xs, 99):8.1f}")
def main():
ap = argparse.ArgumentParser(description="Split wait time. Blame tokens last.")
ap.add_argument("--url")
ap.add_argument("--n", type=int, default=50)
ap.add_argument("--out", default="wait.jsonl")
ap.add_argument("--dry-run", action="store_true")
ap.add_argument("--summarize")
ap.add_argument("--timeout", type=float, default=30.0)
args = ap.parse_args()
if args.summarize:
summarize(args.summarize)
return
payload = json.dumps({"prompt": "lab: do not send secrets"}).encode()
with open(args.out, "w", encoding="utf-8") as out:
for i in range(args.n):
if args.dry_run or not args.url:
sample = dry_sample()
else:
sample = live_sample(args.url, payload, args.timeout)
out.write(json.dumps(asdict(sample)) + "\n")
print(f"wrote {i + 1}/{args.n}", file=sys.stderr)
summarize(args.out)
if __name__ == "__main__":
main()
Run the dry-run until the file format bores you. Then point it at a real URL.
python split_wait.py --dry-run --n 50 --out wait.jsonl
python split_wait.py --summarize wait.jsonl
Live mode is one flag more. Keep secrets out of the payload.
python split_wait.py --url "$ENDPOINT" --n 50 --out wait.jsonl
The summarize command prints a CDF slice. P50 and p99 side by side. I keep that pair. The mean can wait in the hallway.
Handshake detail still belongs to curl. Python told me the tail existed. Curl told me where the handshake sat. I keep this next to the harness.
curl -s -o /dev/null \
-w "dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n" \
"$ENDPOINT"
Run that twenty times on a cold client. Then run a reused session. Compare the tls field. You will feel the subway metaphor in your hands.
Here is a dry-run shape from the harness. Treat it as a shape, not a benchmark. I generated it with --dry-run. It is not a product claim. It is not my laptop's truth either.
n=50 dry_run=True
phase p50_ms p99_ms
connect_ms 17.4 41.2
ttfb_ms 128.0 312.7
body_ms 94.1 148.6
apply_ms 6.8 12.1
pytest_ms 89.4 129.0
Look at TTFB p99 in that shape. That is the jammed train. Tokens did not do that. Body did not do that. Pytest did not do that. I kept that table anyway, as a reminder of the form. Live numbers have to come from your endpoint. Mine will not travel.
What did I change after I trusted the split? First I enabled keep-alive. One session. Many calls. Connect tails shrank. P50 barely moved. P99 fell hard. That is the lesson in one move. Averages hide reuse. Tails reveal it.
Then I stopped opening a new client per prompt. Sounds obvious in a code review. Watch your agent loop anyway. Many wrappers build a fresh client every turn. They look clean. They tax the handshake. The spinner still smiles.
I added a cheap probe too. A tiny POST with no real prompt. If the probe is already fat, skip the model blame. Fix the path. I also logged bytes. A fat prompt can look like a slow decode. It is often upload. Split TTFB from body or you will misread it.
I do not keep a pretty dashboard for this. I keep a two-line CDF. P50 as a calm line. P99 as a rude one. When they kiss, the path is honest. When they split, I hunt the phase with the bigger gap. For a cold remote path, the gap often sits before the first byte. Decode can win on another day. Then the CDF would say so. I would believe the CDF. I would not believe the spinner.
Here is the decision I actually use after the chart settles. If p99 lives in connect or TLS, reuse the client. If p99 lives in TTFB, suspect queueing or a cold remote. If p99 lives in body, then you may talk about tokens. If p99 lives in pytest, stop tuning prompts. Fix the test. That last one stings. I have tuned prompts while the suite did extra I/O. The chart called me out.
Free remote inference is a spotlight, not a trophy. A free server puts the wire in your lap. You see handshake tax. You see cold starts. You see what a laptop model will never show. Spend the free samples on the tail. Do not spend them on another debate about whether AI coding counts as engineering. The clocks do not care.
This split does not grade models. It grades your wait. Do not use it to rank vendors. Do not use it as an SLA. Do not paste secrets into the payload. A free remote path is still a remote path. Redact prompts. Use a dummy repo. If your company forbids egress, stop. This is not for you.
Do not treat dry-run numbers as evidence. They teach shape. Live numbers are yours. If you need guaranteed latency, this is the wrong lab. If you need eval scores, this is the wrong lab. If you just want to ship a patch tonight, skip the CDF. Write the patch.
The method also assumes you can name phases. If your SDK hides HTTP, you will blame the SDK. That is still useful. Wrap lower. If you cannot log bytes, you will confuse upload with decode. Log bytes.
I will not quote a speedup factor. I already learned that lesson elsewhere. I will not name models I did not pin. I will not invent quotas. I will not pretend a free server is permanent. Availability changes. The split still works on the next URL.
I will not tell you AI coding is engineering. I will not tell you it is not. I will tell you where my seconds went. You can pick the slogan later. Bring a CDF to that argument or stay home.
The spinner was a liar. The CDF was not. Split the wait. Keep p99. Change one variable. Blame tokens last.