{"slug": "finding-and-fixing-asyncio-event-loop-stalls-in-a-streaming-gateway-a-stack-per", "title": "Finding and fixing asyncio event-loop stalls in a streaming gateway: a stack-capturing watchdog, per-process shared resources, and the benchmark behind the numbers.", "summary": "A developer built a toy LLM streaming gateway that reproduces asyncio event-loop stalls under many concurrent token streams, pairing it with a stack-capturing watchdog that samples the loop thread's frames when heartbeats run late. The fix moves per-request setup work—tokenizer loading, TLS context creation, and retrieval scoring—out of the request path and into once-per-process initialization, and the accompanying benchmark quantifies the resulting stall reduction.", "body_md": "|  | \"\"\"A toy LLM streaming gateway that reproduces event-loop stalls. | \n|  |  | \n|  | Many clients hold open token streams on one asyncio loop. New requests keep | \n|  | arriving, and each one does some setup before its own stream starts. In | \n|  | \"naive\" mode the setup does the usual per-request things on the loop; in | \n|  | \"fixed\" mode the same work is built once per process before traffic. | \n|  |  | \n|  | Run:  python gateway.py naive   \\|   python gateway.py fixed   \\|   python gateway.py both | \n|  | \"\"\" | \n|  |  | \n|  | from __future__ import annotations | \n|  |  | \n|  | import asyncio | \n|  | import gc | \n|  | import json | \n|  | import os | \n|  | import random | \n|  | import ssl | \n|  | import statistics | \n|  | import sys | \n|  | import tempfile | \n|  | import threading | \n|  | import time | \n|  | import traceback | \n|  | import zlib | \n|  | from collections import Counter | \n|  |  | \n|  | import certifi | \n|  | import numpy as np | \n|  |  | \n|  | STREAMS = 40 # long-lived client streams | \n|  | CHUNK_EVERY = 0.020 # each stream sends a token chunk every 20 ms | \n|  | ARRIVAL_MEAN = 0.25 # a new request every 250 ms on average | \n|  | DURATION = 20.0 # seconds of traffic per run | \n|  | VOCAB_SIZE = 200_000 | \n|  | DOCS, DIMS = 400, 384 # retrieval candidates scored per request | \n|  | PAYLOAD = 1_000_000 # bytes of already-compressed data per request | \n|  | STARTUP_OBJECTS = 1_500_000 | \n|  |  | \n|  | rng = random.Random(7) | \n|  |  | \n|  |  | \n|  | # ---------- watchdog (same design as the article) ---------- | \n|  |  | \n|  | class LoopWatchdog: | \n|  | def __init__(self, loop, tick=0.010, margin=0.040, poll=0.005, report=None): | \n|  | self.loop, self.report = loop, report | \n|  | self.tick, self.limit, self.poll = tick, tick + margin, poll | \n|  | self.loop_thread = threading.get_ident() | \n|  | self.heartbeat = self.seen = time.monotonic() | \n|  | self.stacks = [] | \n|  | self._stop = False | \n|  | threading.Thread(target=self._watch, daemon=True).start() | \n|  |  | \n|  | async def beat(self): | \n|  | while True: | \n|  | self.heartbeat = time.monotonic() | \n|  | await asyncio.sleep(self.tick) | \n|  |  | \n|  | def stop(self): | \n|  | self._stop = True | \n|  |  | \n|  | def _watch(self): | \n|  | while not self._stop: | \n|  | time.sleep(self.poll) | \n|  | beat = self.heartbeat | \n|  | if beat != self.seen: | \n|  | gap, self.seen = beat - self.seen, beat | \n|  | stacks, self.stacks = self.stacks, [] | \n|  | if gap > self.limit: | \n|  | blocked_ms = (gap - self.tick) * 1000 | \n|  | self.loop.call_soon_threadsafe(self.report, blocked_ms, stacks) | \n|  | continue | \n|  | overdue = time.monotonic() - beat > self.limit | \n|  | if overdue and len(self.stacks) < 2: | \n|  | frame = sys._current_frames().get(self.loop_thread) | \n|  | if self.heartbeat == beat: | \n|  | self.stacks.append(traceback.extract_stack(frame, limit=15)) | \n|  |  | \n|  |  | \n|  | # ---------- the expensive things a request start does ---------- | \n|  |  | \n|  | def make_vocab_file() -> str: | \n|  | path = os.path.join(tempfile.gettempdir(), \"toy_vocab.json\") | \n|  | if not os.path.exists(path): | \n|  | vocab = {f\"tok_{i:06d}_{i * 7919 % 100003}\": i for i in range(VOCAB_SIZE)} | \n|  | with open(path, \"w\") as f: | \n|  | json.dump(vocab, f) | \n|  | return path | \n|  |  | \n|  |  | \n|  | def load_tokenizer(path: str) -> dict: | \n|  | with open(path) as f: | \n|  | vocab = json.load(f) | \n|  | return {v: k for k, v in vocab.items()} | \n|  |  | \n|  |  | \n|  | def make_tls_context() -> ssl.SSLContext: | \n|  | return ssl.create_default_context(cafile=certifi.where()) | \n|  |  | \n|  |  | \n|  | def score_python(query: list[float], docs: list[list[float]]) -> list[float]: | \n|  | out = [] | \n|  | for doc in docs: | \n|  | dot = qn = dn = 0.0 | \n|  | for q, d in zip(query, doc): | \n|  | dot += q * d | \n|  | qn += q * q | \n|  | dn += d * d | \n|  | out.append(dot / ((qn ** 0.5) * (dn ** 0.5))) | \n|  | return out | \n|  |  | \n|  |  | \n|  | def score_numpy(query: np.ndarray, docs: np.ndarray) -> np.ndarray: | \n|  | return (docs @ query) / (np.linalg.norm(docs, axis=1) * np.linalg.norm(query)) | \n|  |  | \n|  |  | \n|  | def compress_payload(data: bytes) -> bytes: | \n|  | return zlib.compress(data, 6) | \n|  |  | \n|  |  | \n|  | def make_garbage(): | \n|  | # Per-request bookkeeping: lots of small containers, some cyclic. | \n|  | items = [] | \n|  | for i in range(20_000): | \n|  | d = {\"i\": i, \"meta\": [i, str(i)]} | \n|  | d[\"self\"] = d | \n|  | items.append(d) | \n|  | return len(items) | \n|  |  | \n|  |  | \n|  | # ---------- the gateway ---------- | \n|  |  | \n|  | class Gateway: | \n|  | def __init__(self, mode: str): | \n|  | self.mode = mode | \n|  | self.vocab_path = make_vocab_file() | \n|  | self.docs_list = [[rng.random() for _ in range(DIMS)] for _ in range(DOCS)] | \n|  | self.docs_np = np.array(self.docs_list) | \n|  | self.payload = os.urandom(PAYLOAD) | \n|  | # A big long-lived heap, like a prompt/template catalog loaded at boot. | \n|  | self.catalog = [(i, str(i), [i]) for i in range(STARTUP_OBJECTS)] | \n|  | self.setup_ms: list[float] = [] | \n|  | self.step_ms: dict[str, list[float]] = {} | \n|  | if mode == \"fixed\": | \n|  | self.tls = make_tls_context() | \n|  | self.tokenizer = load_tokenizer(self.vocab_path) | \n|  | gc.collect() | \n|  | gc.freeze() | \n|  |  | \n|  | def _step(self, name, fn): | \n|  | t = time.perf_counter() | \n|  | out = fn() | \n|  | self.step_ms.setdefault(name, []).append((time.perf_counter() - t) * 1000) | \n|  | return out | \n|  |  | \n|  | def start_request(self): | \n|  | t0 = time.perf_counter() | \n|  | step = self._step | \n|  | if self.mode == \"naive\": | \n|  | clients = step(\"TLS context\", lambda: [make_tls_context() for _ in range(3)]) | \n|  | tokenizer = step(\"tokenizer load\", lambda: load_tokenizer(self.vocab_path)) | \n|  | q = [rng.random() for _ in range(DIMS)] | \n|  | step(\"pure-Python scoring\", lambda: score_python(q, self.docs_list)) | \n|  | step(\"compressing incompressible data\", lambda: compress_payload(self.payload)) | \n|  | else: | \n|  | clients = [self.tls] * 3 | \n|  | tokenizer = self.tokenizer | \n|  | q = np.array([rng.random() for _ in range(DIMS)]) | \n|  | step(\"numpy scoring\", lambda: score_numpy(q, self.docs_np)) | \n|  | # payload is already compressed: send as is | \n|  | step(\"request bookkeeping\", make_garbage) | \n|  | self.setup_ms.append((time.perf_counter() - t0) * 1000) | \n|  | return clients, tokenizer | \n|  |  | \n|  |  | \n|  | TIMELINE: list[tuple[float, float]] = []  # (seconds since start, lateness) for --dump | \n|  |  | \n|  |  | \n|  | async def stream(lateness: list[float], stop_at: float): | \n|  | next_due = time.monotonic() | \n|  | while True: | \n|  | next_due += CHUNK_EVERY | \n|  | await asyncio.sleep(max(0.0, next_due - time.monotonic())) | \n|  | now = time.monotonic() | \n|  | if now >= stop_at: | \n|  | return | \n|  | lateness.append(now - next_due) | \n|  | TIMELINE.append((now - (stop_at - DURATION), now - next_due)) | \n|  | if now - next_due > CHUNK_EVERY:  # fell behind: resync like a real sender | \n|  | next_due = now | \n|  |  | \n|  |  | \n|  | def arrival_offsets() -> list[float]: | \n|  | r, t, out = random.Random(11), 0.0, [] | \n|  | while True: | \n|  | t += r.expovariate(1 / ARRIVAL_MEAN) | \n|  | if t >= DURATION - 0.5: | \n|  | return out | \n|  | out.append(t) | \n|  |  | \n|  |  | \n|  | async def arrivals(gw: Gateway, start: float): | \n|  | # Same arrival times in both modes, even if the loop falls behind. | \n|  | for offset in arrival_offsets(): | \n|  | await asyncio.sleep(max(0.0, start + offset - time.monotonic())) | \n|  | gw.start_request() | \n|  |  | \n|  |  | \n|  | def _bucket_worst() -> list[float]: | \n|  | buckets = [0.0] * int(DURATION * 10) | \n|  | for t, late in TIMELINE: | \n|  | i = min(int(t * 10), len(buckets) - 1) | \n|  | buckets[i] = max(buckets[i], late * 1000) | \n|  | TIMELINE.clear() | \n|  | return [round(b, 1) for b in buckets] | \n|  |  | \n|  |  | \n|  | CATEGORIES = [ | \n|  | (\"make_tls_context\", \"TLS context\"), | \n|  | (\"load_tokenizer\", \"tokenizer load\"), | \n|  | (\"score_python\", \"pure-Python scoring\"), | \n|  | (\"compress_payload\", \"compressing incompressible data\"), | \n|  | ] | \n|  |  | \n|  |  | \n|  | def categorize(stack) -> str: | \n|  | names = {f.name for f in stack} | \n|  | for fn, label in CATEGORIES: | \n|  | if fn in names: | \n|  | return label | \n|  | return \"other\" | \n|  |  | \n|  |  | \n|  | async def run(mode: str) -> dict: | \n|  | gw = Gateway(mode) | \n|  | gc_pauses: list[float] = [] | \n|  | gc_start = [0.0] | \n|  |  | \n|  | def on_gc(phase, info): | \n|  | if phase == \"start\": | \n|  | gc_start[0] = time.perf_counter() | \n|  | else: | \n|  | gc_pauses.append((time.perf_counter() - gc_start[0]) * 1000) | \n|  |  | \n|  | gc.callbacks.append(on_gc) | \n|  | stalls = [] | \n|  |  | \n|  | def report(blocked_ms, stacks): | \n|  | stalls.append((blocked_ms, categorize(stacks[0]) if stacks else \"no stack (GIL held)\")) | \n|  |  | \n|  | wd = LoopWatchdog(asyncio.get_running_loop(), report=report) | \n|  | beat = asyncio.create_task(wd.beat()) | \n|  | stop_at = time.monotonic() + DURATION | \n|  | lateness: list[list[float]] = [[] for _ in range(STREAMS)] | \n|  | await asyncio.gather(arrivals(gw, stop_at - DURATION), *(stream(l, stop_at) for l in lateness)) | \n|  | wd.stop() | \n|  | beat.cancel() | \n|  | gc.callbacks.remove(on_gc) | \n|  | gc.unfreeze() | \n|  |  | \n|  | all_late = [x * 1000 for l in lateness for x in l] | \n|  |  | \n|  | def share_late(threshold_ms): | \n|  | return 100 * sum(x > threshold_ms for x in all_late) / len(all_late) | \n|  |  | \n|  | causes = Counter(c for _, c in stalls) | \n|  | return { | \n|  | \"mode\": mode, | \n|  | \"requests\": len(gw.setup_ms), | \n|  | \"setup_p50_ms\": statistics.median(gw.setup_ms), | \n|  | \"stalls_per_request\": len(stalls) / len(gw.setup_ms), | \n|  | \"stall_p50_ms\": statistics.median([s for s, _ in stalls]) if stalls else 0.0, | \n|  | \"stall_max_ms\": max([s for s, _ in stalls], default=0.0), | \n|  | \"loop_blocked_pct\": 100 * sum(s for s, _ in stalls) / 1000 / DURATION, | \n|  | \"chunk_late_p99_ms\": float(np.percentile(all_late, 99)), | \n|  | \"chunk_late_max_ms\": max(all_late), | \n|  | \"chunks_late_50ms_pct\": share_late(50), | \n|  | \"chunks_late_100ms_pct\": share_late(100), | \n|  | \"gc_collections\": len(gc_pauses), | \n|  | \"gc_pause_max_ms\": max(gc_pauses, default=0.0), | \n|  | \"gc_pause_total_ms\": sum(gc_pauses), | \n|  | \"setup_step_p50_ms\": {k: round(statistics.median(v), 2) for k, v in gw.step_ms.items()}, | \n|  | \"timeline_100ms_worst_late_ms\": _bucket_worst(), | \n|  | \"stall_causes\": {k: round(100 * v / len(stalls), 1) for k, v in causes.most_common()} if stalls else {}, | \n|  | } | \n|  |  | \n|  |  | \n|  | def main(): | \n|  | args = sys.argv[1:] | \n|  | dump = None | \n|  | if \"--dump\" in args: | \n|  | i = args.index(\"--dump\") | \n|  | dump = args[i + 1] | \n|  | del args[i:i + 2] | \n|  | modes = args or [\"both\"] | \n|  | if modes == [\"both\"]: | \n|  | modes = [\"naive\", \"fixed\"] | \n|  | results = [] | \n|  | for mode in modes: | \n|  | result = asyncio.run(run(mode)) | \n|  | timeline = result.pop(\"timeline_100ms_worst_late_ms\") | \n|  | print(json.dumps(result, indent=2)) | \n|  | result[\"timeline_100ms_worst_late_ms\"] = timeline | \n|  | results.append(result) | \n|  | gc.collect() | \n|  | if dump: | \n|  | with open(dump, \"w\") as f: | \n|  | json.dump(results, f) | \n|  |  | \n|  |  | \n|  | if __name__ == \"__main__\": | \n|  | main() |", "url": "https://wpnews.pro/news/finding-and-fixing-asyncio-event-loop-stalls-in-a-streaming-gateway-a-stack-per", "canonical_source": "https://gist.github.com/swayamg20/8eeeedae23aa05fe2dd124ee8c463866", "published_at": "2026-10-25 19:24:34+00:00", "updated_at": "2026-10-26 07:44:02.466153+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "mlops", "developer-tools", "ai-agents"], "entities": ["asyncio", "numpy", "certifi"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/finding-and-fixing-asyncio-event-loop-stalls-in-a-streaming-gateway-a-stack-per", "markdown": "https://wpnews.pro/news/finding-and-fixing-asyncio-event-loop-stalls-in-a-streaming-gateway-a-stack-per.md", "text": "https://wpnews.pro/news/finding-and-fixing-asyncio-event-loop-stalls-in-a-streaming-gateway-a-stack-per.txt", "jsonld": "https://wpnews.pro/news/finding-and-fixing-asyncio-event-loop-stalls-in-a-streaming-gateway-a-stack-per.jsonld"}}