# Your p50 Is a Lie: Four Free-Tier Myths You Can Verify in One Hour > Source: > Published: 2026-08-28 08:32:49+00:00 Your request timed out. What's your first move? Retry immediately? Blame the model? Check the p50? All three instincts are wrong. On free tiers, all three. I keep seeing the same four myths in issue trackers, Discord threads, and code reviews. So here's a myth-busting FAQ with a reproducible probe. The probe is standard-library Python. One file. Any OpenAI-compatible endpoint. AI promoted everyone to reviewer. Almost nobody reviews the queue in front of the model. That's the gap this post covers. I build small apps on free model endpoints. When I test harnesses against MonkeyCode's free model access and free server, I watch the same myths appear on day one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow works on any endpoint, not just theirs. Run the same prompt with temperature 0. Free endpoint or paid endpoint, same text. Compare the outputs. They match token for token. The weights are the same. Scheduling is different. **Correct mental model:** the free tier is a shared queue, not a downgraded brain. Your request waits for an inference slot. Then it runs on the same model as everyone else. Queue delay is bimodal. Half your requests land on a warm slot. Half sit behind a cold start or a crowded queue. p50 blends two populations into one number that means nothing. I've seen p50 look great while one request in five timed out. That's not a healthy service. That's a queue measured wrong. **Correct mental model:** watch p95, p99, and the stall rate. Split first-token time from total time. They answer different questions. A timeout usually means the queue is crowded at that exact microsecond. Retrying now buys the same ticket to the same line. You're not recovering a request. You're concentrating load. Worse: five clients retrying in sync create a thundering herd. The queue gets busier. The next timeout becomes more likely. **Correct mental model:** retry with jittered exponential backoff. Random sleep that grows each attempt. Spread retries across time instead of stacking them. With streaming, 200 means the gate opened. Nothing more. The first token can still be seconds away. Some servers send headers, then stall. I probe until `data: [DONE]` . That's the only honest completion signal. A 200 with no data is a stall. Set a read timeout so the probe can't hang forever. **Correct mental model:** first token and last token are separate events. Track both. A small gap means generation. A big gap after the 200 means queue. The script checks myths 2, 3, and 4 in one run. Myth 1 needs no script: just diff two responses at temperature 0. If your endpoint ignores `stream` , the probe will count every call as a stall. Confirm streaming support first. ``` bash #!/usr/bin/env python3 '''myth_probe.py - check free-tier myths on any OpenAI-style endpoint. Usage: python myth_probe.py --url https://host/v1/chat/completions \ --model your-model [--key ''] [--burst 20] [--workers 5] [--timeout 30] Standard library only. Python 3.9+. ''' import argparse import json import random import time from concurrent.futures import ThreadPoolExecutor from urllib.request import Request, urlopen def pct(vals, p): if not vals: return float('nan') s = sorted(vals) return s[min(len(s) - 1, len(s) * p // 100)] def one_call(url, key, model, timeout): body = { 'model': model, 'stream': True, 'temperature': 0, 'messages': [{'role': 'user', 'content': 'List the numbers 1 to 20, one per line.'}], } req = Request(url, data=json.dumps(body).encode(), method='POST', headers={ 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + key, }) t0 = time.monotonic() first_at = total_at = None try: with urlopen(req, timeout=timeout) as r: saw_first = False while True: raw = r.readline() if not raw: break line = raw.decode('utf-8', 'ignore').strip() if not line.startswith('data:'): continue if not saw_first: first_at = time.monotonic() - t0 saw_first = True if line == 'data: [DONE]': total_at = time.monotonic() - t0 break if first_at is None or total_at is None: return {'ok': False} return {'ok': True, 'first': first_at, 'total': total_at} except Exception: return {'ok': False} def burst(url, key, model, workers, n, timeout): with ThreadPoolExecutor(max_workers=workers) as ex: futures = [ex.submit(one_call, url, key, model, timeout) for _ in range(n)] results = [f.result() for f in futures] ok = [r for r in results if r['ok']] print(f'completed {len(ok)}/{n} stall rate {1 - len(ok) / n:.0%}') for label, field in (('first_token', 'first'), ('total_time', 'total')): vals = [r[field] for r in ok] print(f'{label:11s} p50 {pct(vals, 50):5.2f}s ' f'p95 {pct(vals, 95):5.2f}s p99 {pct(vals, 99):5.2f}s') def client_work(url, key, model, timeout, mode, budget): sent = 0 for i in range(budget): sent += 1 if one_call(url, key, model, timeout)['ok']: return True, sent if mode == 'backoff': time.sleep(random.uniform(0, 0.4) * (1.6 ** i)) return False, sent def herd(url, key, model, timeout, clients, mode): t0 = time.monotonic() with ThreadPoolExecutor(max_workers=clients) as ex: out = list(ex.map(lambda _: client_work( url, key, model, timeout, mode, 8), range(clients))) done = sum(1 for ok, _ in out if ok) sent = sum(s for _, s in out) print(f'{mode:9s} done {done}/{clients} requests {sent:3d} ' f'wall {time.monotonic() - t0:5.1f}s') if __name__ == '__main__': ap = argparse.ArgumentParser() ap.add_argument('--url', required=True, help='OpenAI-style /chat/completions URL') ap.add_argument('--model', required=True) ap.add_argument('--key', default='') ap.add_argument('--timeout', type=float, default=30.0) ap.add_argument('--burst', type=int, default=20) ap.add_argument('--workers', type=int, default=5) ap.add_argument('--clients', type=int, default=10) args = ap.parse_args() print('== burst ==') burst(args.url, args.key, args.model, args.workers, args.burst, args.timeout) print('== herd (retry storm) ==') herd(args.url, args.key, args.model, args.timeout, args.clients, 'immediate') herd(args.url, args.key, args.model, args.timeout, args.clients, 'backoff') ``` Example output — numbers are illustrative. Your queue will look different. ``` == burst == completed 18/20 stall rate 10% first_token p50 0.82s p95 6.41s p99 11.02s total_time p50 1.24s p95 7.73s p99 12.90s == herd (retry storm) == immediate done 8/10 requests 47 wall 34.2s backoff done 10/10 requests 31 wall 28.6s ``` What each line tells you: `completed 18/20` — 10% of requests never finished. That's your stall rate.`first_token p95` — six seconds is not a slow model. It's a busy queue.`total_time p99` — eleven seconds to finish twenty tokens. The queue dominates, not the weights.`immediate` vs `backoff` — the herd test shows the crowd paying for instant retries. Fewer requests, better completion, faster wall time.`data: [DONE]` is the answer.No. Free tiers are great for agents, batch jobs, and prototypes. They're wrong where a stall costs real money. Know which lane you're in before you wire the retry loop. That's the whole FAQ. The free tier is not a toy. It's a queue wearing a model costume. Once you accept that, retries, monitoring, and expectations all fall into place. Grab any free endpoint. Run the probe. Look at your p99. Then you'll know whether you're fighting a model or a queue. One hour. No dashboards. That's the whole test.