{"slug": "running-llms-inside-a-chrome-extension-webllm-transformers-js-and-the-mv3-traps", "title": "Running LLMs inside a Chrome extension: WebLLM, transformers.js and the MV3 traps", "summary": "AWFlow, a Chrome extension for browser automation workflows, runs local LLMs inside a Manifest V3 extension using three runtimes — WebLLM for chat, transformers.js for task pipelines, and TensorFlow.js for legacy models — with inference split across two Web Workers to keep the Svelte side-panel UI responsive. The build required a WebGPU probe with a WASM fallback and surfaced a Content Security Policy rule that silently blocks the runtime, according to the developer's account. The extension defaults to 1–4B parameter local models so text never leaves the machine, while letting users switch individual nodes to OpenAI, Anthropic, Google or a local Ollama for deeper reasoning.", "body_md": "Running LLMs inside a Chrome extension: WebLLM, transformers.js and the MV3 traps\n\nWhat it took to run AI models inside a Manifest V3 extension: three runtimes, two Web Workers, a WebGPU probe with a WASM fallback, and the CSP rule that silently blocks your runtime.\n\nI build AWFlow, a Chrome extension that runs automation workflows in your browser. One of its node types is “summarize, classify or extract this text with AI”. I wanted that step to work without an API key and without sending the page anywhere, which meant running the model inside the extension itself.\n\nThis post covers what that took: three runtimes, two Web Workers, one CSP surprise, and a fallback path for machines without WebGPU. If you want the gentler tour of what WebLLM and Chrome’s built-in AI are, read Run AI nodes directly in Chrome with WebLLM and Chrome AI API first. This one is about the plumbing.\n\nWhy run the model in the extension\n\nLook at what most “AI automation” steps actually do: summarize a page, tag a sentiment, pull five fields out of some HTML. You don’t need a frontier model in a data center for that. A 1–4B parameter model running locally does it well enough, for free, and the text never leaves the machine.\n\nWhen a step really does need deep reasoning, the user can switch that one node to OpenAI, Anthropic, Google or a local Ollama. The point is to make local the default, not the only option.\n\nThree runtimes, one job each\n\nThere’s no single “run a model in the browser” library that covers everything, so the extension ships three:\n\nTask pipelines: summarization, classification, embeddings, speech, small chat models\n\nTensorFlow.js\n\nA few classic models (for example legacy embeddings)\n\nThere’s also a node for Chrome’s built-in AI when the browser exposes it. A Local Models page lists what’s installed for every runtime and lets you download or delete models, so users never have to care which engine a model runs on. (If you are choosing a model rather than building one in, this guide is the practical side.)\n\nKeep inference off the main thread\n\nThe extension’s UI is a Svelte app in the side panel and in a tab. If a 2 GB model compiles on that thread, the canvas freezes. So the first rule is simple: every runtime runs in a dedicated worker.\n\nHere is the whole layout. Two workers do the heavy work, and the UI only sends requests and draws results.\n\nFive parts\n\nThe UI draws the canvas. Two workers do the heavy work, on the graphics card when they can and on WASM when they can’t.\n\nChat goes to the WebLLM worker\n\nDownloading, compiling and generating all happen in the worker, so the side panel stays responsive.\n\nTasks go to a second worker\n\ntransformers.js and TensorFlow.js share one worker. It caches one pipeline per model id, so the second run of a workflow doesn’t reload anything, and it streams download progress back to the UI.\n\nWebLLM runs on the graphics card\n\nIts runtime is WebAssembly, and the heavy math runs as WebGPU kernels.\n\nTasks use the GPU when it is safe\n\nA task pipeline goes to WebGPU only if the adapter exists and the task is on the list of tasks known to work there.\n\nOtherwise they use WASM\n\nSlower, but it runs on any machine.\n\nThe WebLLM worker is three lines\n\nWebLLM makes the worker side fairly clean:\n\n``` js\n// worker.ts\nimport { WebWorkerMLCEngineHandler } from '@mlc-ai/web-llm';\nconst handler = new WebWorkerMLCEngineHandler();\nself.onmessage = (msg: MessageEvent) => handler.onmessage(msg);\n```\n\nThe catch: I use LangChain’s ChatWebLLM wrapper so the local model plugs into the same agent code as the cloud models (the same agent that builds your workflows). ChatWebLLM builds its own main-thread engine in its constructor and has no way to accept one from outside. But WebWorkerMLCEngine implements the same MLCEngineInterface, so you can build the worker engine yourself and swap it in.\n\nIn code, with the fallback that matters most:\n\n``` js\nconst worker = new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' });\ntry {\n  const engine = await CreateWebWorkerMLCEngine(worker, modelId, {\n    appConfig: model.appConfig,\n    initProgressCallback: onProgress,\n  });\n  // `engine` is protected and typed as the concrete MLCEngine,\n  // but the runtime contract is the interface, which the worker engine satisfies.\n  (model as unknown as { engine: MLCEngineInterface }).engine = engine;\n} catch (error) {\n  worker.terminate();                 // don't orphan the thread\n  await model.initialize(onProgress); // fall back to the main-thread engine\n}\n```\n\nIf the worker path fails (no WebGPU, a download error), it falls back to the old behavior instead of breaking the node.\n\nProbe WebGPU once, keep a WASM path\n\nNot every machine has a usable WebGPU adapter. The inference worker probes it once and remembers the answer:\n\n``` js\nlet webgpu: Promise<boolean> | undefined;\nfunction hasWebGPU() {\n  webgpu ??= (async () => {\n    try {\n      const gpu = (navigator as any).gpu;\n      return !!(gpu && (await gpu.requestAdapter()));\n    } catch { return false; }\n  })();\n  return webgpu;\n}\n```\n\nThen each pipeline picks its device. Some pipelines are fine on WASM and flaky on WebGPU, so the allowlist is per task, not global. (For the trade-offs between the backends themselves, see WebGPU vs WebGL vs WebAssembly.)\n\nTwo MV3 rules nobody warns you about\n\nThe first two rules are about performance. The next two are about Manifest V3, and the first of them cost me an afternoon.\n\nThe CSP will block your WASM runtime\n\nBy default, ONNX Runtime Web (inside transformers.js) loads its WASM loader script and binary from a CDN. A Manifest V3 extension page has this CSP:\n\n```\nscript-src 'self' 'wasm-unsafe-eval'; object-src 'self'\n```\n\nYou can’t add a remote host to script-src in MV3. So the import is blocked, and no local model runs. The error doesn’t make the cause obvious.\n\nThe fix: ship the ONNX Runtime assets inside the extension (a small Vite plugin copies them to /assets/ort/ at build time) and point the runtime at the same-origin copy before the first pipeline() call:\n\n``` js\nconst tf = await import('@huggingface/transformers'); // lazy: keep it out of the main bundle\ntf.env.backends.onnx.wasm.wasmPaths = '/assets/ort/';\n```\n\n'wasm-unsafe-eval' is what allows WebAssembly compilation at all. Without it in your manifest CSP, none of this works. (Why WebAssembly is what makes a browser tab capable of this in the first place is the subject of WebAssembly, explained.)\n\nStorage is the user’s, so show it\n\nModels are big. Users need to see what’s on their disk and delete it. transformers.js models live in the Cache API, so that part is easy. WebLLM is harder: it doesn’t expose a per-model byte size. The Local Models page falls back to navigator.storage.estimate() for the total and shows per-model sizes only where a runtime reports them. That’s an honest approximation, not a made-up number.\n\nWhat small local models are good at\n\nAfter a few months of this in real workflows, the split is clear. Small local models are good at narrow, well-defined jobs:\n\nThen there are the jobs where they need help, or a bigger model:\n\nOkay with help: tool use. Small models have no tool-calling API, so you have to prompt and parse (more on that in a later post).\n\nNot good: long multi-step reasoning, and anything where a wrong answer is expensive. That’s when you switch the node to a bigger model.\n\nThe honest cost is the first run: it pays a download, from tens of MB for a classifier to a couple of GB for a chat model. After that it’s cached and starts fast. If you want to test whether a small model is enough for your own inputs before committing, this post walks through how.\n\nTo see the local path end to end, here is the simplest workflow that uses it: read the current page, summarize it with a local model, show the result. The model downloads once, then works offline.\n\nIf you’re doing this in your own extension, here is the short version.\n\nAWFlow is free to install. I’m happy to go deeper on any of this, especially if you’ve solved the WebLLM model-size problem better than I have.", "url": "https://wpnews.pro/news/running-llms-inside-a-chrome-extension-webllm-transformers-js-and-the-mv3-traps", "canonical_source": "https://awflow.io/blog/running-llms-inside-a-chrome-extension", "published_at": "2026-10-15 00:00:00+00:00", "updated_at": "2026-10-21 22:12:36.854304+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["AWFlow", "WebLLM", "transformers.js", "TensorFlow.js", "Chrome", "LangChain", "WebGPU", "Ollama"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-llms-inside-a-chrome-extension-webllm-transformers-js-and-the-mv3-traps", "markdown": "https://wpnews.pro/news/running-llms-inside-a-chrome-extension-webllm-transformers-js-and-the-mv3-traps.md", "text": "https://wpnews.pro/news/running-llms-inside-a-chrome-extension-webllm-transformers-js-and-the-mv3-traps.txt", "jsonld": "https://wpnews.pro/news/running-llms-inside-a-chrome-extension-webllm-transformers-js-and-the-mv3-traps.jsonld"}}