{"slug": "vllm-ios-88-faster-multi-agent-inference-on-ios", "title": "vllm-ios: 88% Faster Multi-Agent Inference on iOS", "summary": "Developer Jon Ready released vllm-ios, a native Swift implementation of vLLM-style continuous batching for MLX on iOS, achieving 88% faster multi-agent inference than llama.cpp at 8 concurrent streams with identical weights. In benchmarks on an iPhone 16 Pro with Qwen3.5-0.8B, llama.cpp delivered 78 tokens/s aggregate across 8 streams versus 103 tokens/s for single-stream MLX, while vllm-ios enabled 8 agents to generate 385 tokens in 2.9 seconds on-device. The engine, about 300 lines of Swift, uses uniform-offset batching to share KV cache offsets and prefills shared prompt prefixes once, addressing iOS thermal throttling that limits full-speed GPU to 15-20 seconds per minute.", "body_md": "TL;DR: I improved multi-agent batching on iOS by implementing vLLM-style continuous batching in native Swift on MLX.\n\nI've written a lot about [the tokenomics of local LLMs and agent swarms](agent-swarms-are-great-for-local-ai.html). The short version: every forward pass uses the same memory bandwidth whether you're decoding one stream or eight, so concurrent agent work should share a batch. I ported vLLM's continuous-batching architecture to pure Swift for iOS — [vllm-ios](https://github.com/jonready/vllm-ios), native on MLX, 88% faster than llama.cpp at 8 concurrent streams with identical weights.\n\n[\nOpen source\njonready/vllm-ios\nView on GitHub →\n](https://github.com/jonready/vllm-ios)\n\nvLLM-style continuous batching for iPhone. Native Swift on MLX, no Python.\n\nThe end result first. SwarmBench, the repo's demo app: one question fans out to eight specialist agents (key facts, plan, risks, contrarian, and so on), all answering at once.\n\nEvery card you see streaming is a row in a single batched decode:\n\n- All eight agents share one weight read per token\n- The shared prompt prefix is prefilled once, cached, and reused\n- Tokens stream to each card as they materialize\n- Agents finish independently and exit the batch early\n\nIn this clip: 8 agents, 385 tokens, 2.9 seconds, on the phone's silicon alone. The rest of this post is how the engine behind it got built, and the benchmarks it exists to win.\n\nTo be clear, 8 subagents answering the same question isn't terribly useful, but it demonstrates the multi-agent inference performance that other workflows can leverage: agents each reading a different document, map-reduce over a corpus, a planner and its critics running side by side, speculative branches you throw away. The benchmarks below use exactly that shape: my travel app's research agents, each reading a different 1,000-token blog excerpt and extracting ~128 tokens of structured JSON.\n\n## Batched inference on iOS is surprisingly bad\n\nThe two serious runtimes fail in opposite ways. MLX has the fastest kernels but no multi-sequence serving on iOS — one stream per model. (Python mlx-lm has batching; a Swift port has been [stuck in review since May](https://github.com/ml-explore/mlx-swift-lm/pull/263).) llama.cpp has excellent continuous batching, but its Metal kernels barely reward it on A-series silicon. Measured with Qwen3.5-0.8B, greedy decoding, on an iPhone 16 Pro, thermally controlled:\n\n| Runtime, 4-bit | 1 stream | 8 streams (aggregate) | Speedup |\n|---|---|---|---|\n| llama.cpp (Q4_K_M) | 54 tok/s | 78 tok/s | 1.4x |\n| MLX (stock) | 103 tok/s | n/a — no batching | 1.0x |\n\nA 1.4x return on 8x the parallelism, or none at all. The same idea on a Mac ([vllm-mlx](https://arxiv.org/abs/2601.19139)) gets 4.3x at 16 requests.\n\n## iphones get hot fast\n\nOn iOS you want inference over as soon as possible. A full-tilt GPU pulls ~10 watts, heats the phone, drains the battery, and dies when the user backgrounds the app. I measure roughly **15-20 seconds of full-speed GPU per minute** before throttling — a 20-25 second burst takes the phone from cool to throttled. That's the real argument for batching: sixteen requests in one 25-second burn fit a single thermal budget; poor concurrency burns two or three, with later requests throttled.\n\n## So I rewrote vLLM's scheduler in Swift\n\nFast kernels with no scheduler (MLX), or a great scheduler with slow kernels (llama.cpp). Kernels are hard; schedulers are a few hundred lines. Easy call. [The engine](https://github.com/jonready/vllm-ios) is ~300 lines of Swift on [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm): requests join at token boundaries, finished ones exit immediately, stock kernels underneath. The trick that avoids touching model code is **uniform-offset batching** — every sequence shares the same KV offset so the stock causal mask works. Late arrivals get left-padded, prefilled solo, then spliced in by KV cache surgery on the public API. With greedy decoding, several steps chain into one lazy on-GPU graph so you sync once per chunk instead of per token.\n\n## What the phone can actually do\n\nWith a real scheduler on the fast kernels, scaling shows up (thermally clean, zero measurable scheduling overhead):\n\n| Batch size | Per-stream | Aggregate decode | Speedup |\n|---|---|---|---|\n| 1 | 103 tok/s | 103 tok/s | 1.0x |\n| 2 | 84 tok/s | 168 tok/s | 1.6x |\n| 8 | 25 tok/s | 199 tok/s | 1.9x |\n\nAt 8-bit the gap widens: 51 → 169 tok/s (3.3x) vs llama.cpp's 45 → 90 (2.0x) on identical weights. Batch 2 is the sweet spot — 84% of max aggregate while each stream stays 3.4x faster than at batch 8. And for this prefill-heavy workload, wall-clock barely moves with batch size (26.6s at B=2 vs 25.2s at B=8); small batches just return the first result sooner.\n\nAn aside worth pulling out: across everything we measured, quantization and batching are the two levers that actually matter for local inference speed, and they're the same lever twice. Decode is memory-bandwidth-bound, so quantization shrinks the bytes every token must stream (8-bit → 4-bit took single-stream from 51 to 103 tok/s, tracking file size almost exactly), and batching splits those bytes across streams (103 → 199 aggregate at batch 8). They multiply: 51 → 199 tok/s, a 3.9x swing on the same silicon, from just those two knobs. Everything else we tuned — flash attention, KV layouts, chunk sizes, sync elimination — moved the numbers by single-digit percentages.\n\nBottom line: **16 research requests, ~17k prompt tokens in and 2k tokens of JSON out, in 25 seconds, thermally stable**. The llama.cpp baseline needed 47 seconds for half as many requests — two thermal budgets instead of one.\n\n## The scoreboard\n\nSame phone, weight-identical 8-bit models, 1–8 concurrent streams. The gap between \"batching exists\" and \"batching works\":\n\nLocal AI is less hardware-limited than we think: this phone's silicon had 4x more to give than any off-the-shelf stack would extract, and every gain came from co-designing the harness with the hardware. Batching, because decode is bandwidth-bound. Bursts, because the thermal budget is 15-20 seconds a minute. Prefix caching, because agent prompts share structure. And the table is still full: kernel fusion, calibrated quants, a Neural Engine sitting idle. The phone in your pocket runs sixteen research agents in one thermal budget with [vllm-ios](https://github.com/jonready/vllm-ios). The next multiple is waiting on software shaped like the silicon.", "url": "https://wpnews.pro/news/vllm-ios-88-faster-multi-agent-inference-on-ios", "canonical_source": "https://jonready.com/blog/posts/continuous-batching-on-an-iphone.html", "published_at": "2026-08-24 00:00:00+00:00", "updated_at": "2026-08-25 21:13:40.115680+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Jon Ready", "vllm-ios", "MLX", "llama.cpp", "Qwen3.5-0.8B", "iPhone 16 Pro", "SwarmBench", "mlx-swift-lm"], "alternates": {"html": "https://wpnews.pro/news/vllm-ios-88-faster-multi-agent-inference-on-ios", "markdown": "https://wpnews.pro/news/vllm-ios-88-faster-multi-agent-inference-on-ios.md", "text": "https://wpnews.pro/news/vllm-ios-88-faster-multi-agent-inference-on-ios.txt", "jsonld": "https://wpnews.pro/news/vllm-ios-88-faster-multi-agent-inference-on-ios.jsonld"}}