Your p50 Is a Lie: Four Free-Tier Myths You Can Verify in One Hour

A developer's probe of free-tier AI endpoints debunks four common myths, showing that free tiers share the same model weights as paid tiers but suffer from queue delays, making p50 latency misleading. The developer recommends monitoring p95 and p99, using jittered exponential backoff for retries, and tracking first-token vs. total time to accurately assess service health.

Your request timed out. What's your first move? Retry immediately? Blame the model? Check the p50? All three instincts are wrong. On free tiers, all three. I keep seeing the same four myths in issue trackers, Discord threads, and code reviews. So here's a myth-busting FAQ with a reproducible probe. The probe is standard-library Python. One file. Any OpenAI-compatible endpoint. AI promoted everyone to reviewer. Almost nobody reviews the queue in front of the model. That's the gap this post covers. I build small apps on free model endpoints. When I test harnesses against MonkeyCode's free model access and free server, I watch the same myths appear on day one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow works on any endpoint, not just theirs. Run the same prompt with temperature 0. Free endpoint or paid endpoint, same text. Compare the outputs. They match token for token. The weights are the same. Scheduling is different. Correct mental model: the free tier is a shared queue, not a downgraded brain. Your request waits for an inference slot. Then it runs on the same model as everyone else. Queue delay is bimodal. Half your requests land on a warm slot. Half sit behind a cold start or a crowded queue. p50 blends two populations into one number that means nothing. I've seen p50 look great while one request in five timed out. That's not a healthy service. That's a queue measured wrong. Correct mental model: watch p95, p99, and the stall rate. Split first-token time from total time. They answer different questions. A timeout usually means the queue is crowded at that exact microsecond. Retrying now buys the same ticket to the same line. You're not recovering a request. You're concentrating load. Worse: five clients retrying in sync create a thundering herd. The queue gets busier. The next timeout becomes more likely. Correct mental model: retry with jittered exponential backoff. Random sleep that grows each attempt. Spread retries across time instead of stacking them. With streaming, 200 means the gate opened. Nothing more. The first token can still be seconds away. Some servers send headers, then stall. I probe until data: DONE . That's the only honest completion signal. A 200 with no data is a stall. Set a read timeout so the probe can't hang forever. Correct mental model: first token and last token are separate events. Track both. A small gap means generation. A big gap after the 200 means queue. The script checks myths 2, 3, and 4 in one run. Myth 1 needs no script: just diff two responses at temperature 0. If your endpoint ignores stream , the probe will count every call as a stall. Confirm streaming support first. bash /usr/bin/env python3 '''myth probe.py - check free-tier myths on any OpenAI-style endpoint. Usage: python myth probe.py --url https://host/v1/chat/completions \ --model your-model --key '' --burst 20 --workers 5 --timeout 30 Standard library only. Python 3.9+. ''' import argparse import json import random import time from concurrent.futures import ThreadPoolExecutor from urllib.request import Request, urlopen def pct vals, p : if not vals: return float 'nan' s = sorted vals return s min len s - 1, len s p // 100 def one call url, key, model, timeout : body = { 'model': model, 'stream': True, 'temperature': 0, 'messages': {'role': 'user', 'content': 'List the numbers 1 to 20, one per line.'} , } req = Request url, data=json.dumps body .encode , method='POST', headers={ 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + key, } t0 = time.monotonic first at = total at = None try: with urlopen req, timeout=timeout as r: saw first = False while True: raw = r.readline if not raw: break line = raw.decode 'utf-8', 'ignore' .strip if not line.startswith 'data:' : continue if not saw first: first at = time.monotonic - t0 saw first = True if line == 'data: DONE ': total at = time.monotonic - t0 break if first at is None or total at is None: return {'ok': False} return {'ok': True, 'first': first at, 'total': total at} except Exception: return {'ok': False} def burst url, key, model, workers, n, timeout : with ThreadPoolExecutor max workers=workers as ex: futures = ex.submit one call, url, key, model, timeout for in range n results = f.result for f in futures ok = r for r in results if r 'ok' print f'completed {len ok }/{n} stall rate {1 - len ok / n:.0%}' for label, field in 'first token', 'first' , 'total time', 'total' : vals = r field for r in ok print f'{label:11s} p50 {pct vals, 50 :5.2f}s ' f'p95 {pct vals, 95 :5.2f}s p99 {pct vals, 99 :5.2f}s' def client work url, key, model, timeout, mode, budget : sent = 0 for i in range budget : sent += 1 if one call url, key, model, timeout 'ok' : return True, sent if mode == 'backoff': time.sleep random.uniform 0, 0.4 1.6 i return False, sent def herd url, key, model, timeout, clients, mode : t0 = time.monotonic with ThreadPoolExecutor max workers=clients as ex: out = list ex.map lambda : client work url, key, model, timeout, mode, 8 , range clients done = sum 1 for ok, in out if ok sent = sum s for , s in out print f'{mode:9s} done {done}/{clients} requests {sent:3d} ' f'wall {time.monotonic - t0:5.1f}s' if name == ' main ': ap = argparse.ArgumentParser ap.add argument '--url', required=True, help='OpenAI-style /chat/completions URL' ap.add argument '--model', required=True ap.add argument '--key', default='' ap.add argument '--timeout', type=float, default=30.0 ap.add argument '--burst', type=int, default=20 ap.add argument '--workers', type=int, default=5 ap.add argument '--clients', type=int, default=10 args = ap.parse args print '== burst ==' burst args.url, args.key, args.model, args.workers, args.burst, args.timeout print '== herd retry storm ==' herd args.url, args.key, args.model, args.timeout, args.clients, 'immediate' herd args.url, args.key, args.model, args.timeout, args.clients, 'backoff' Example output — numbers are illustrative. Your queue will look different. == burst == completed 18/20 stall rate 10% first token p50 0.82s p95 6.41s p99 11.02s total time p50 1.24s p95 7.73s p99 12.90s == herd retry storm == immediate done 8/10 requests 47 wall 34.2s backoff done 10/10 requests 31 wall 28.6s What each line tells you: completed 18/20 — 10% of requests never finished. That's your stall rate. first token p95 — six seconds is not a slow model. It's a busy queue. total time p99 — eleven seconds to finish twenty tokens. The queue dominates, not the weights. immediate vs backoff — the herd test shows the crowd paying for instant retries. Fewer requests, better completion, faster wall time. data: DONE is the answer.No. Free tiers are great for agents, batch jobs, and prototypes. They're wrong where a stall costs real money. Know which lane you're in before you wire the retry loop. That's the whole FAQ. The free tier is not a toy. It's a queue wearing a model costume. Once you accept that, retries, monitoring, and expectations all fall into place. Grab any free endpoint. Run the probe. Look at your p99. Then you'll know whether you're fighting a model or a queue. One hour. No dashboards. That's the whole test.