WebLLM: high-performance in-browser LLM inference engine

MLC AI's WebLLM is a high-performance in-browser LLM inference engine that runs entirely in web browsers with WebGPU hardware acceleration, eliminating the need for server-side processing. It is fully compatible with the OpenAI API, supports streaming, JSON-mode, and function-calling, and natively supports models including Llama 3, Phi 3, Gemma, Mistral, and Qwen. The engine enables privacy-preserving AI assistants with GPU acceleration and can be integrated via npm, Yarn, or CDN.

High-Performance In-Browser LLM Inference Engine. Documentation https://webllm.mlc.ai/docs/ | Blogpost https://blog.mlc.ai/2024/06/13/webllm-a-high-performance-in-browser-llm-inference-engine | Paper https://arxiv.org/abs/2412.15803 | Examples /mlc-ai/web-llm/blob/main/examples WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU. WebLLM is fully compatible with OpenAI API. That is, you can use the same OpenAI API on any open source models locally, with functionalities including streaming, JSON-mode, function-calling WIP , etc. We can bring a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration. You can use WebLLM as a base npm package https://www.npmjs.com/package/@mlc-ai/web-llm and build your own web application on top of it by following the examples below. This project is a companion project of MLC LLM https://github.com/mlc-ai/mlc-llm , which enables universal deployment of LLM across hardware environments. - In-Browser Inference : WebLLM is a high-performance, in-browser language model inference engine that leverages WebGPU for hardware acceleration, enabling powerful LLM operations directly within web browsers without server-side processing. - : Seamlessly integrate your app with WebLLM using OpenAI API with functionalities such as streaming, JSON-mode, logit-level control, seeding, and more. Full OpenAI API Compatibility - Structured JSON Generation : WebLLM supports state-of-the-art JSON mode structured generation, implemented in the WebAssembly portion of the model library for optimal performance. Check WebLLM JSON Playground https://huggingface.co/spaces/mlc-ai/WebLLM-JSON-Playground on HuggingFace to try generating JSON output with custom JSON schema. - : WebLLM natively supports a range of models including Llama 3, Phi 3, Gemma, Mistral, Qwen 通义千问 , and many others, making it versatile for various AI tasks. For the complete supported model list, check Extensive Model Support MLC Models https://mlc.ai/models . - : Easily integrate and deploy custom models in MLC format, allowing you to adapt WebLLM to specific needs and scenarios, enhancing flexibility in model deployment. Custom Model Integration - Plug-and-Play Integration : Easily integrate WebLLM into your projects using package managers like NPM and Yarn, or directly via CDN, complete with comprehensive examples /mlc-ai/web-llm/blob/main/examples and a modular design for connecting with UI components. - Streaming & Real-Time Interactions : Supports streaming chat completions, allowing real-time output generation which enhances interactive applications like chatbots and virtual assistants. - Web Worker & Service Worker Support : Optimize UI performance and manage the lifecycle of models efficiently by offloading computations to separate worker threads or service workers. - Chrome Extension Support : Extend the functionality of web browsers through custom Chrome extensions using WebLLM, with examples available for building both basic and advanced extensions. Check the complete list of available models on MLC Models https://mlc.ai/models . WebLLM supports a subset of these available models and the list can be accessed at prebuiltAppConfig.model list https://github.com/mlc-ai/web-llm/blob/main/src/config.ts L293 . Here are the primary families of models currently supported: Llama : Llama 3, Llama 2, Hermes-2-Pro-Llama-3 Phi : Phi 3, Phi 2, Phi 1.5 Gemma : Gemma-2B Mistral : Mistral-7B-v0.3, Hermes-2-Pro-Mistral-7B, NeuralHermes-2.5-Mistral-7B, OpenHermes-2.5-Mistral-7B Qwen 通义千问 : Qwen2 0.5B, 1.5B, 7B If you need more models, request a new model via opening an issue https://github.com/mlc-ai/web-llm/issues/new/choose or check Custom Models custom-models for how to compile and use your own models with WebLLM. Learn how to use WebLLM to integrate large language models into your application and generate chat completions through this simple Chatbot example: For an advanced example of a larger, more complicated project, check WebLLM Chat https://github.com/mlc-ai/web-llm-chat/blob/main/app/client/webllm.ts . More examples for different use cases are available in the examples /mlc-ai/web-llm/blob/main/examples folder. WebLLM offers a minimalist and modular interface to access the chatbot in the browser. The package is designed in a modular way to hook to any of the UI components. npm npm install @mlc-ai/web-llm yarn yarn add @mlc-ai/web-llm or pnpm pnpm install @mlc-ai/web-llm Then import the module in your code. // Import everything import as webllm from "@mlc-ai/web-llm"; // Or only import what you need import { CreateMLCEngine } from "@mlc-ai/web-llm"; Thanks to jsdelivr.com https://www.jsdelivr.com/package/npm/@mlc-ai/web-llm , WebLLM can be imported directly through URL and work out-of-the-box on cloud development platforms like jsfiddle.net https://jsfiddle.net/ , Codepen.io https://codepen.io/ , and Scribbler https://scribbler.live : import as webllm from "https://esm.run/@mlc-ai/web-llm"; It can also be dynamically imported as: js const webllm = await import "https://esm.run/@mlc-ai/web-llm" ; Most operations in WebLLM are invoked through the MLCEngine interface. You can create an MLCEngine instance and loading the model by calling the CreateMLCEngine factory function. Note that loading models requires downloading and it can take a significant amount of time for the very first run without caching previously. You should properly handle this asynchronous call. js import { CreateMLCEngine } from "@mlc-ai/web-llm"; // Callback function to update model loading progress const initProgressCallback = initProgress = { console.log initProgress ; }; const selectedModel = "Llama-3.1-8B-Instruct-q4f32 1-MLC"; const engine = await CreateMLCEngine selectedModel, { initProgressCallback: initProgressCallback }, // engineConfig ; Under the hood, this factory function does the following steps for first creating an engine instance synchronous and then loading the model asynchronous . You can also do them separately in your application. js import { MLCEngine } from "@mlc-ai/web-llm"; // This is a synchronous call that returns immediately const engine = new MLCEngine { initProgressCallback: initProgressCallback, } ; // This is an asynchronous call and can take a long time to finish await engine.reload selectedModel ; WebLLM supports four cache backends through AppConfig.cacheBackend : "cache" : browser Cache API https://developer.mozilla.org/en-US/docs/Web/API/Cache default . "indexeddb" : browser IndexedDB https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB API . "opfs" : browser Origin Private File System OPFS https://developer.mozilla.org/en-US/docs/Web/API/File System API/Origin private file system . "cross-origin" : experimental Chrome Cross-Origin Storage API https://github.com/WICG/cross-origin-storage extension backend. Install the Cross-Origin Storage extension https://chromewebstore.google.com/detail/cross-origin-storage/denpnpcgjgikjpoglpjefakmdcbmlgih to use it. If the extension isn't installed, WebLLM falls back to the default cache automatically. Example: js import { CreateMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm"; const appConfig = { ...prebuiltAppConfig, cacheBackend: "cross-origin" }; const engine = await CreateMLCEngine "Llama-3.1-8B-Instruct-q4f32 1-MLC", { appConfig, } ; Notes: - If "opfs" is selected in an environment without OPFS support, cache operations fail with an OPFS availability error. - When using "opfs" , appConfig.opfsAccessMode can be set to "auto" to use OPFS sync access handles where supported, or "sync" to require sync access handles. The default is "async" . - The "cross-origin" backend requires installing and enabling a compatible browser extension. - Cross-origin backend currently does not support programmatic tensor-cache deletion; clearing is extension-managed. After successfully initializing the engine, you can now invoke chat completions using OpenAI style chat APIs through the engine.chat.completions interface. For the full list of parameters and their descriptions, check section below full-openai-compatibility and OpenAI API reference https://platform.openai.com/docs/api-reference/chat/create . Note: The model parameter is not supported and will be ignored here. Instead, call CreateMLCEngine model or engine.reload model instead as shown in the Create MLCEngine create-mlcengine above. js const messages = { role: "system", content: "You are a helpful AI assistant." }, { role: "user", content: "Hello " }, ; const reply = await engine.chat.completions.create { messages, } ; console.log reply.choices 0 .message ; console.log reply.usage ; WebLLM also supports streaming chat completion generating. To use it, simply pass stream: true to the engine.chat.completions.create call. js const messages = { role: "system", content: "You are a helpful AI assistant." }, { role: "user", content: "Hello " }, ; // Chunks is an AsyncGenerator object const chunks = await engine.chat.completions.create { messages, temperature: 1, stream: true, // <-- Enable streaming stream options: { include usage: true }, } ; let reply = ""; for await const chunk of chunks { reply += chunk.choices 0 ?.delta.content || ""; console.log reply ; if chunk.usage { console.log chunk.usage ; // only last chunk has usage } } const fullReply = await engine.getMessage ; console.log fullReply ; You can put the heavy computation in a worker script to optimize your application performance. To do so, you need to: - Create a handler in the worker thread that communicates with the frontend while handling the requests. - Create a Worker Engine in your main application, which under the hood sends messages to the handler in the worker thread. For detailed implementations of different kinds of Workers, check the following sections. WebLLM comes with API support for WebWorker so you can hook the generation process into a separate worker thread so that the computing in the worker thread won't disrupt the UI. We create a handler in the worker thread that communicates with the frontend while handling the requests. js // worker.ts import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm"; // A handler that resides in the worker thread const handler = new WebWorkerMLCEngineHandler ; self.onmessage = msg: MessageEvent = { handler.onmessage msg ; }; In the main logic, we create a WebWorkerMLCEngine that implements the same MLCEngineInterface . The rest of the logic remains the same. js // main.ts import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm"; async function main { // Use a WebWorkerMLCEngine instead of MLCEngine here const engine = await CreateWebWorkerMLCEngine new Worker new URL "./worker.ts", import.meta.url , { type: "module", } , selectedModel, { initProgressCallback }, // engineConfig ; // everything else remains the same } WebLLM comes with API support for ServiceWorker so you can hook the generation process into a service worker to avoid reloading the model in every page visit and optimize your application's offline experience. Note, Service Worker's life cycle is managed by the browser and can be killed any time without notifying the webapp. ServiceWorkerMLCEngine will try to keep the service worker thread alive by periodically sending heartbeat events, but your application should also include proper error handling. Check keepAliveMs and missedHeatbeat in ServiceWorkerMLCEngine https://github.com/mlc-ai/web-llm/blob/main/src/service worker.ts L234 for more details. We create a handler in the worker thread that communicates with the frontend while handling the requests. Instantiate the handler at the top level of the worker script so its message listener is registered during initial script evaluation. Do not instantiate it from an activate or message listener: the browser can restart an already-active worker without dispatching another activate event. js // sw.ts import { ServiceWorkerMLCEngineHandler } from "@mlc-ai/web-llm"; new ServiceWorkerMLCEngineHandler ; console.log "Service Worker is ready" ; Then in the main logic, we register the service worker and create the engine using CreateServiceWorkerMLCEngine function. The rest of the logic remains the same. js // main.ts import { MLCEngineInterface, CreateServiceWorkerMLCEngine, } from "@mlc-ai/web-llm"; if "serviceWorker" in navigator { navigator.serviceWorker.register new URL "sw.ts", import.meta.url , // worker script { type: "module" }, ; } const engine: MLCEngineInterface = await CreateServiceWorkerMLCEngine selectedModel, { initProgressCallback }, // engineConfig ; You can find a complete example on how to run WebLLM in service worker in examples/service-worker /mlc-ai/web-llm/blob/main/examples/service-worker . You can also find examples of building Chrome extension with WebLLM in examples/chrome-extension /mlc-ai/web-llm/blob/main/examples/chrome-extension and examples/chrome-extension-webgpu-service-worker /mlc-ai/web-llm/blob/main/examples/chrome-extension-webgpu-service-worker . The latter one leverages service worker, so the extension is persistent in the background. Additionally, you can explore another full project of a Chrome extension, WebLLM Assistant, which leverages WebLLM here https://github.com/mlc-ai/web-llm-assistant . WebLLM is designed to be fully compatible with OpenAI API https://platform.openai.com/docs/api-reference/chat . Thus, besides building a simple chatbot, you can also have the following functionalities with WebLLM: streaming /mlc-ai/web-llm/blob/main/examples/streaming : return output as chunks in real-time in the form of an AsyncGenerator json-mode /mlc-ai/web-llm/blob/main/examples/json-mode : efficiently ensure output is in JSON format, see OpenAI Reference https://platform.openai.com/docs/guides/text-generation/chat-completions-api for more. seed-to-reproduce /mlc-ai/web-llm/blob/main/examples/seed-to-reproduce : use seeding to ensure a reproducible output with fields seed . function-calling /mlc-ai/web-llm/blob/main/examples/function-calling WIP : function calling with fields tools and tool choice with preliminary support ; or manual function calling without tools or tool choice keeps the most flexibility . WebLLM supports optional integrity verification for model artifacts using SRI Subresource Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subresource Integrity hashes. When the integrity field is set on a ModelRecord , WebLLM will verify the downloaded config, WASM, and tokenizer files against the provided hashes before loading. js import { CreateMLCEngine } from "@mlc-ai/web-llm"; const appConfig = { model list: { model: "https://huggingface.co/mlc-ai/Llama-3.2-1B-Instruct-q4f16 1-MLC", model id: "Llama-3.2-1B-Instruct-q4f16 1-MLC", model lib: "https://raw.githubusercontent.com/user/model-libs/main/model.wasm", integrity: { config: "sha256-<base64-hash-of-mlc-chat-config.json ", model lib: "sha256-<base64-hash-of-wasm-file ", tokenizer: { "tokenizer.json": "sha256-<base64-hash-of-tokenizer.json ", }, onFailure: "error", // "error" default throws IntegrityError, "warn" logs and continues }, }, , }; const engine = await CreateMLCEngine "Llama-3.2-1B-Instruct-q4f16 1-MLC", { appConfig, } ; You can generate SRI hashes for model files with: SHA-256 openssl dgst -sha256 -binary <file | openssl base64 -A | sed 's/^/sha256-/' SHA-384 openssl dgst -sha384 -binary <file | openssl base64 -A | sed 's/^/sha384-/' SHA-512 openssl dgst -sha512 -binary <file | openssl base64 -A | sed 's/^/sha512-/' The openssl commands require a Unix-like shell macOS/Linux . On Windows, run openssl via Git Bash or WSL . If a hash does not match, an IntegrityError is thrown or a warning is logged when onFailure: "warn" . All fields in integrity are optional — only specified artifacts will be verified. When the integrity field is omitted entirely, WebLLM behaves exactly as before no verification . See the integrity-verification example /mlc-ai/web-llm/blob/main/examples/integrity-verification for a complete working demo. WebLLM works as a companion project of MLC LLM https://github.com/mlc-ai/mlc-llm and it supports custom models in MLC format. It reuses the model artifact and builds the flow of MLC LLM. To compile and use your own models with WebLLM, please check out MLC LLM document https://llm.mlc.ai/docs/deploy/webllm.html on how to compile and deploy new model weights and libraries to WebLLM. Here, we go over the high-level idea. There are two elements of the WebLLM package that enable new models and weight variants. model : Contains a URL to model artifacts, such as weights and meta-data. model lib : A URL to the web assembly library i.e. wasm file that contains the executables to accelerate the model computations. Both are customizable in the WebLLM. js import { CreateMLCEngine } from "@mlc-ai/web-llm"; async main { const appConfig = { "model list": { "model": "/url/to/my/llama", "model id": "MyLlama-3b-v1-q4f32 0", "model lib": "/url/to/myllama3b.wasm", } , }; // override default const chatOpts = { "repetition penalty": 1.01 }; // load a prebuilt model // with a chat option override and app config // under the hood, it will load the model from myLlamaUrl // and cache it in the browser cache // The chat will also load the model library from "/url/to/myllama3b.wasm", // assuming that it is compatible to the model in myLlamaUrl. const engine = await CreateMLCEngine "MyLlama-3b-v1-q4f32 0", { appConfig }, // engineConfig chatOpts, ; } In many cases, we only want to supply the model weight variant, but not necessarily a new model e.g. NeuralHermes-Mistral can reuse Mistral 's model library . For examples of how a model library can be shared by different model variants, see webllm.prebuiltAppConfig . NOTE: you don't need to build from source unless you would like to modify the WebLLM package. To use the npm, simply follow Get Started get-started or any of the examples /mlc-ai/web-llm/blob/main/examples instead. To build from source, simply run: npm install npm run build Then, to test the effects of your code change in an example, inside examples/get-started/package.json , change from "@mlc-ai/web-llm": "^0.2.84" to "@mlc-ai/web-llm": ../.. . Then run: cd examples/get-started npm install npm start Note that sometimes you would need to switch between file:../.. and ../.. to trigger npm to recognize new changes. In the worst case, you can run: cd examples/get-started rm -rf node modules dist package-lock.json .parcel-cache npm install npm start WebLLM's runtime largely depends on TVMjs: https://github.com/apache/tvm/tree/main/web https://github.com/apache/tvm/tree/main/web While it is also available as an npm package: https://www.npmjs.com/package/@mlc-ai/web-runtime https://www.npmjs.com/package/@mlc-ai/web-runtime , you can build it from source if needed by following the steps below. - Install emscripten https://emscripten.org . It is an LLVM-based compiler that compiles C/C++ source code to WebAssembly.- Follow the installation instruction https://emscripten.org/docs/getting started/downloads.html installation-instructions-using-the-emsdk-recommended to install the latest emsdk. - Source emsdk env.sh by source path/to/emsdk env.sh , so that emcc is reachable from PATH and the command emcc works. We can verify the successful installation by trying out emcc terminal.Note: We recently found that using the latest emcc version may run into issues during runtime. Use ./emsdk install 3.1.56 instead of ./emsdk install latest for now as a workaround. The error may look like Init error, LinkError: WebAssembly.instantiate : Import 6 module="wasi snapshot preview1" function="proc exit": function import requires a callable - Follow the - In ./package.json , change from "@mlc-ai/web-runtime": "0.18.0-dev2", to "@mlc-ai/web-runtime": "file:./tvm home/web", . - Setup necessary environment Prepare all the necessary dependencies for web build: ./scripts/prep deps.sh In this step, if $TVM SOURCE DIR is not defined in the environment, we will execute the following line to build tvmjs dependency: git clone https://github.com/mlc-ai/relax 3rdparty/tvm-unity --recursive This clones the current HEAD of mlc-ai/relax . However, it may not always be the correct branch or commit to clone. To build a specific npm version from source, refer to the version bump PR, which states which branch i.e. mlc-ai/relax or apache/tvm and which commit the current WebLLM version depends on. For instance, version 0.2.52, according to its version bump PR 521 https://github.com/mlc-ai/web-llm/pull/521 , is built by checking out the following commit https://github.com/apache/tvm/commit/e6476847753c80e054719ac47bc2091c888418b6 https://github.com/apache/tvm/commit/e6476847753c80e054719ac47bc2091c888418b6 in apache/tvm , rather than the HEAD of mlc-ai/relax .Besides, --recursive is necessary and important. Otherwise, you may encounter errors like fatal error: 'dlpack/dlpack.h' file not found . - Build WebLLM Package npm run build - Validate some of the sub-packages You can then go to the subfolders in examples /mlc-ai/web-llm/blob/main/examples to validate some of the sub-packages. We use Parcelv2 for bundling. Although Parcel is not very good at tracking parent directory changes sometimes. When you make a change in the WebLLM package, try to edit the package.json of the subfolder and save it, which will trigger Parcel to rebuild. Demo App: WebLLM Chat https://chat.webllm.ai/ - If you want to run LLM on native runtime, check out MLC-LLM https://github.com/mlc-ai/mlc-llm - You might also be interested in Web Stable Diffusion https://github.com/mlc-ai/web-stable-diffusion/ . This project is initiated by members from CMU Catalyst, UW SAMPL, SJTU, OctoML, and the MLC community. We would love to continue developing and supporting the open-source ML community. This project is only possible thanks to the shoulders open-source ecosystems that we stand on. We want to thank the Apache TVM community and developers of the TVM Unity effort. The open-source ML community members made these models publicly available. PyTorch and Hugging Face communities make these models accessible. We would like to thank the teams behind Vicuna, SentencePiece, LLaMA, and Alpaca. We also would like to thank the WebAssembly, Emscripten, and WebGPU communities. Finally, thanks to Dawn and WebGPU developers. If you find this project to be useful, please cite: @misc{ruan2026webllmhighperformanceinbrowserllm, title={WebLLM: A High-Performance In-Browser LLM Inference Engine}, author={Charlie F. Ruan and Yucheng Qin and Akaash R. Parthasarathy and Xun Zhou and Ruihang Lai and Hongyi Jin and Yixin Dong and Bohan Hou and Meng-Shiun Yu and Yiyan Zhai and Sudeep Agarwal and Hangrui Cao and Siyuan Feng and Tianqi Chen}, year={2026}, eprint={2412.15803}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2412.15803}, }