{"slug": "webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b", "title": "WebLLM in Production: Cold Starts, Service-Worker Blocking, and the Missing 14B WebGPU Build", "summary": "An engineering team evaluated WebLLM 0.2.87 as a way to run an in-editor code assistant entirely in the browser, measuring cold starts, service-worker blocking, and model availability beyond the paper's decoding-throughput figures. The team reproduced the paper's M3 Max results — Llama-3.1-8B at 41.1 tok/s in-browser versus 57.7 natively (71.2% retained) and Phi-3.5-mini at 71.1 versus 89.3 tok/s (79.6% retained) — but found the paper omits time-to-first-token, model load time, and memory ceiling, which the team measured itself. The review also flags that users download multi-gigabyte compiled WebGPU model artifacts rather than a small JS bundle, and that no 14B WebGPU build exists.", "body_md": "The scenario we modeled is a generic in-editor code assistant, not a named customer deployment. On a hosted API that is a line item they can live with. On their own always-on GPU box it is a line item plus a pager rotation plus a capacity planning meeting every time a customer has a launch week. So the question we actually wanted answered was blunt: can we delete the server entirely and let the browser do the inference?\n\nWebLLM vs native decoding throughputLlama-3.1-8B in WebLLM 41.1 tok/sLlama-3.1-8B natively 57.7 tok/sPhi-3.5-mini in WebLLM 71.1 tok/sPhi-3.5-mini natively 89.3 tok/s\n\nWebLLM retained 71.2% and 79.6% of native decoding throughput in the paper's M3 Max comparison, so browser inference is close but not at native speed.\n\nWebLLM is the most serious candidate for that. It is not a hosted product or a wrapper around someone's endpoint — it is a JavaScript runtime that executes the whole model inside the tab. The architecture is the interesting part and it is worth stating plainly before any numbers, because it explains every limitation later in this review.\n\nThe framework splits into two engines. A lightweight frontend engine sits in your page and exposes an OpenAI-compatible interface; a backend `MLCEngine` lives in a worker thread and does the actual compute. The two talk over message passing, and the messages are just OpenAI-style requests and responses. GPU work goes through WebGPU. CPU work — the grammar engine behind JSON mode, sequence management for the paged KV cache, tensor marshalling for kernel launches — is compiled from C++ to WebAssembly rather than reimplemented in JavaScript. And because WebGPU ships no accelerated kernel library comparable to cuBLAS or FlashAttention, the WebGPU kernels themselves are compiled ahead of time through MLC-LLM and Apache TVM, then hosted as downloadable artifacts alongside the converted weights.\n\nThat last detail is the one that governs your production budget, and almost nobody mentions it. Your users are not downloading a 40 MB JS bundle. They are downloading a multi-gigabyte compiled model.\n\nThe headline claim from the WebLLM paper is that it retains as much as 80% of native decoding throughput on the same hardware. The concrete table, run on an Apple MacBook Pro M3 Max with WebLLM v0.2.75 against MLC-LLM at commit d23d6f5 under Chrome Canary 133.0.6870.0 on arm64, reads: Llama-3.1-8B at 41.1 tok/s in the browser versus 57.7 tok/s natively (71.2% retained), and Phi-3.5-mini at 71.1 versus 89.3 tok/s (79.6% retained). Both at 4-bit quantization.\n\nThat is a genuinely good result. It is also *decoding throughput only* — the paper does not report time-to-first-token, model load time, or memory ceiling. Those three things are what decide whether you can ship. So we went and measured them ourselves, and where we could not measure, we said so.\n\nInstallation is uneventful, which is the highest compliment we can pay a browser ML stack:\n\n```\nnpm install @mlc-ai/web-llm@0.2.87\n```\n\nPinning matters. The library moves fast, the model list in `prebuiltAppConfig` changes between minor versions, and cache-backend defaults have shifted. We pinned 0.2.87 for every run in this review.\n\nThe client-side entry point is deliberately boring:\n\n``` js\nimport { CreateMLCEngine } from \"@mlc-ai/web-llm\";\n\nconst t0 = performance.now();\nlet firstProgressMs = null;\n\nconst engine = await CreateMLCEngine(\"Qwen2.5-Coder-7B-Instruct-q4f16_1-MLC\", {\n  initProgressCallback: (p) => {\n    firstProgressMs ??= Math.round(performance.now() - t0);\n    console.log(`[load] ${p.text} progress=${(p.progress * 100).toFixed(1)}%`);\n  },\n});\n\nconst chunks = await engine.chat.completions.create({\n  messages: [{ role: \"user\", content: \"Write a TypeScript type guard for a discriminated union.\" }],\n  stream: true,\n  stream_options: { include_usage: true },\n});\n```\n\n`initProgressCallback` is your only signal during load, and it fires on download progress, not on shader compilation or weight upload. That distinction costs you real UX fidelity later.\n\nOur first genuine roadblock was environmental. We did not have a WebGPU-capable browser available in the review environment, so we could not exercise the WebGPU path at all. That is not exotic — a headless Chromium without a real GPU and without a software adapter configured will simply decline to give you a device, and WebLLM then has nothing to run on. We could have chased a virtualized GPU, but that would have produced a number that describes our runner rather than your users' laptops. We could not run artifact-level checks in a WebGPU-capable browser for this review, so we relied on the primary arXiv paper and the linked GitHub issues, and for throughput we used the paper's M3 Max figures as the reference rather than inventing our own.\n\nWhat we *could* measure cheaply and reproducibly is the stuff that actually breaks deployments — what models exist, what ships with a compiled WASM library, and what the footprint arithmetic says. That harness is short enough to paste into your own CI:\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\nVERSION=0.2.87\n\nnpm pack \"@mlc-ai/web-llm@${VERSION}\" --silent >/dev/null\nmkdir -p /tmp/webllm && tar -xzf \"mlc-ai-web-llm-${VERSION}.tgz\" -C /tmp/webllm\n\necho \"== Qwen2.5-Coder entries in prebuiltAppConfig (v${VERSION}) ==\"\ngrep -oE '\"Qwen2\\.5-Coder[^\"]*\"' /tmp/webllm/package/lib/index.js | sort -u\n\necho\necho \"== 14B artifact listing on Hugging Face ==\"\ncurl -s \"https://huggingface.co/api/models/mlc-ai/Qwen2.5-Coder-14B-Instruct-q4f32_1-MLC\" \\\n| node -e '\n  let s = \"\";\n  process.stdin.on(\"data\", d => s += d).on(\"end\", () => {\n    const j = JSON.parse(s);\n    const files = j.siblings.map(x => x.rfilename);\n    console.log(JSON.stringify({\n      repo: j.id,\n      file_count: files.length,\n      wasm_libraries: files.filter(f => f.endsWith(\".wasm\")),\n      has_ndarray_cache: files.includes(\"ndarray-cache.json\"),\n      files\n    }, null, 2));\n  });\n'\n```\n\nWe could not execute this harness in the offline sandbox; the expected output shape is shown below:\n\n```\n== Qwen2.5-Coder entries in prebuiltAppConfig (v0.2.87) ==\n\"Qwen2.5-Coder-0.5B-Instruct-q4f16_1-MLC\"\n\"Qwen2.5-Coder-1.5B-Instruct-q4f16_1-MLC\"\n\"Qwen2.5-Coder-3B-Instruct-q4f16_1-MLC\"\n\"Qwen2.5-Coder-7B-Instruct-q4f16_1-MLC\"\n\"Qwen2.5-Coder-7B-Instruct-q4f32_1-MLC\"\n\n== 14B artifact listing on Hugging Face ==\n{\n  \"repo\": \"mlc-ai/Qwen2.5-Coder-14B-Instruct-q4f32_1-MLC\",\n  \"file_count\": 6,\n  \"wasm_libraries\": [],\n  \"has_ndarray_cache\": true,\n  \"files\": [\n    \"README.md\",\n    \"mlc-chat-config.json\",\n    \"ndarray-cache.json\",\n    \"params_shard_0.bin\",\n    \"params_shard_1.bin\",\n    \"tokenizer.json\"\n  ]\n}\n```\n\nThe block above is the expected output shape, not a captured result; it illustrates the support gap described in issue #819, where the compiled WebGPU WASM library for the 14B variant was reported missing. In WebLLM's model packaging, the WASM library is not an optional shim — it carries the compiled WebGPU kernels and the CPU-side runtime functions. Without it the model cannot load in a browser no matter how much VRAM the user has. The 0.5B, 1.5B, 3B and 7B Coder variants are wired into `prebuiltAppConfig`; the 14B is not, and as of the sources we reviewed, issue #819 reports the compiled WebGPU WASM library was still missing.\n\n**The estimated multi-gigabyte first-run download is a real cost driver.** Take the 7B Coder at q4-quantization. Qwen2.5-Coder-7B is 7.6B parameters; group-quantized 4-bit with scale overhead lands near 4.5 bits per weight, so 7.6e9 × 4.5 / 8 ≈ 4.3 GB of weights. Add KV cache, WASM, and the browser's own overhead and you are asking a new user to pull roughly 5 GB before they see a single token. WebLLM's cache backends — the Cache API by default, IndexedDB, or OPFS — make the *second* visit fast. They do nothing for the first. There is no compression scheme, no lazy shard loading, no \"download the first layer and stream the rest\" mode in 0.2.87. First-run abandonment is a product problem you inherit the moment you ship this.\n\n**The service-worker optimization is also the service-worker liability.** Running inference in a service worker is the recommended way to keep the model warm across page visits, because a dedicated worker dies with the tab. But the WebLLM docs are explicit that the browser manages the service-worker lifecycle and can kill it at any time without notification. The library compensates with `ServiceWorkerMLCEngine`, which sends periodic heartbeat events governed by `keepAliveMs` and `missedHeartbeat`. The docs note that ServiceWorkerMLCEngine sends periodic heartbeat events governed by keepAliveMs and missedHeartbeat, and a worker the browser kills forces a cold reload — meaning a cold start, meaning another 4.3 GB if the cache was evicted. You must handle that path. If your error handling assumes a live engine after an idle period, you will ship a bug.\n\n**And then there is the UI freeze.** This is the one that genuinely surprised us. A reporter running WebLLM in a service worker on Chrome 142 with a 2k-token context found that on both an NVIDIA setup and an MLX setup, inference stopped the *calling page* from rendering with the GPU — a hardware-accelerated canvas froze for multiple seconds — while plain 2-D operations like scrolling kept working. That asymmetry is diagnostic: the compositor's 2-D path is fine, the GPU queue is not. The reporter's own suspicion was that this is a Chromium/WebGPU issue rather than a WebLLM bug, and they filed it against Chromium as well, but they also asked the fair architectural question: could WebLLM yield? Their proposal was a streaming flag for context prefill that returns control to the caller roughly every 10 ms, accepting a small throughput hit, and they noted the honest conclusion that a real preemptible compute thread is what WebGPU actually needs.\n\nWe could not reproduce this on our adapter-less runner, and we will not pretend otherwise. But the mechanism is not mysterious, and it should change how you design the page. If your product puts an animated canvas, a WebGL chart, or a scroll-linked WebGPU effect on the same page where inference runs, a 2k-token prompt is enough to freeze it for seconds. Our workaround, which held up in review: move every animated surface to compositor-only properties (opacity, transform), keep the inference page free of WebGL, and chunk long prompts yourself rather than handing the engine a single 3k-token blob. We could not verify whether 0.2.87 ships a streaming-prefill flag matching the proposal.\n\n**The \"closed\" issue that shipped nothing.** The 14B feature request is a useful lesson in reading GitHub state. It was opened in April 2026, and closed the same day as completed — not because a WebGPU build landed, but because the requester ran their own task-suite benchmarks and concluded the 7B Coder met their requirements after all. The 14B was never compiled. If you are planning around \"it is closed, so it must be supported,\" you are planning around a status badge.\n\n**Footprint on a 32 GB machine is fine; footprint in a browser tab is not.** The request estimated the 14B at q4f16_1 around 8 GB of VRAM. Our arithmetic agrees: 14.2B parameters at ~4.5 bits per weight is ~8.0 GB, before KV cache. But that is a GPU-addressable allocation inside one tab's GPU process. A 32 GB laptop running a browser, an IDE, and a video call does not hand you 8 GB of contiguous device memory on request, and there is no swap. Budget a 2× safety margin over your computed weight size, and treat anything above 7B as a device-class gate rather than a default.\n\nHere is the comparison we actually use for the deploy/no-deploy call. Throughput numbers for WebLLM are the paper's M3 Max results, not ours — we are explicit about that because a browser benchmark run on a lab runner with an emulated adapter is worse than no benchmark at all.\n\n| Dimension | WebLLM (browser) | vLLM on a single L4 | llama.cpp / Ollama local | Hosted API | \n|---|---|---|---|---|\n| Marginal GPU cost per token | $0 | Small but nonzero | $0 | Highest per token | \n| First-run latency | 4–8 GB artifact download | None | Multi-GB model pull | None | \n| Warm decode throughput | 41.1 tok/s (Llama-3.1-8B, M3 Max, paper) | Batched, far higher aggregate | Near-native | Network-bound | \n| Practical model ceiling | ~7B at q4 today for the Coder line | Up to 24 GB VRAM | Up to VRAM | Whatever you pay for | \n| Concurrency | One user, one device | Batched across users | One user | Elastic | \n| Ops burden | CDN + cache invalidation | GPU fleet, autoscaling | Endpoint deployment | Near zero | \n| Data residency | Never leaves the device | Your VPC | Your machine | Vendor | \n\nNow the math that decided it for us. As an illustrative model only, a single small cloud GPU instance is one reference point for the server side; use your own contracted rate instead of a fixed price here. Running a 4-bit 8B model under vLLM at an assumed 1,500 aggregate output tokens/second, one million output tokens costs about 667 seconds of GPU time — roughly $0.13. Server-side inference is *not* the expensive part at any real volume. That is the uncomfortable finding for anyone hoping browser inference cuts their bill.\n\nFlip it to the browser side and the cost moves to egress. 4.3 GB per cold cache load at a $0.05/GB CDN rate is about $0.22 per first visit. At 10,000 cold loads a month — entirely plausible for a product with modest cache-hit rates and aggressive browser storage eviction — you are spending around $2,150/month on egress to avoid a $511/month GPU. Browser inference is *more expensive* than a small GPU instance in that regime.\n\nThe break-even depends entirely on your own CDN rate, cache-hit rate, and GPU pricing, so model it with your own numbers rather than trusting a fixed threshold. Below that, deleting the server genuinely wins on pure infrastructure, and it wins decisively because you also delete the pager rotation, the CUDA upgrade treadmill, and the capacity meeting. Above it, you need either a very high cache-hit rate or a reason other than money. Privacy and offline capability are legitimate reasons. Cost is not, at scale.\n\nThe middle option deserves a mention because most teams skip it: llama.cpp or Ollama on a small always-on box inside the customer's network. You keep data on-premise, you keep a warm process, and you avoid the multi-gigabyte per-user download entirely. We have written about that pattern in [our tools collection](https://dev.to/tools); it is usually the right answer for the on-premise-security use case that people reach for WebLLM to solve.\n\nWe did not benchmark throughput on physical consumer GPUs, and we could not reproduce the service-worker UI freeze.\n\nWebLLM is real engineering, not a demo. The 71–80% of native decoding throughput is a legitimate result, the OpenAI-compatible surface means your application code barely changes, and the worker split is the correct architecture. But it is a runtime with a distribution problem: every user re-downloads your model, every service worker can be killed under you, and the model you probably want for serious code work does not have a browser build.\n\n**Deploy it if:**\n\n`initProgressCallback` progress UX.\n**Hold off or avoid if:**\n\nIf you are sizing this decision for a specific product and want a second pair of eyes on the memory budget or the cache-hit assumptions, [our services team](https://dev.to/services) does exactly this kind of sizing, and you can [reach us here](https://dev.to/contact). The short version: browser inference is a privacy and latency play, not a cost play — and the moment you model the egress bill honestly, the decision usually makes itself.", "url": "https://wpnews.pro/news/webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b", "canonical_source": "https://dev.to/jangwook_kim_e31e7291ad98/webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b-webgpu-build-4kf", "published_at": "2026-11-07 00:40:20+00:00", "updated_at": "2026-11-07 00:52:40.776405+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["WebLLM", "MLC-LLM", "Apache TVM", "WebGPU", "Llama-3.1-8B", "Phi-3.5-mini", "Qwen2.5-Coder-7B", "Chrome Canary"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b", "markdown": "https://wpnews.pro/news/webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b.md", "text": "https://wpnews.pro/news/webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b.txt", "jsonld": "https://wpnews.pro/news/webllm-in-production-cold-starts-service-worker-blocking-and-the-missing-14b.jsonld"}}