Private & Fast: Building a Local-First Mental Health Assistant with WebLLM and WebGPU 🧠💻

A developer has built a local-first mental health assistant using WebLLM, TVM Unity, and React, which runs Llama 3 models directly in the browser via WebGPU. The assistant, designed for Cognitive Behavioral Therapy, keeps all sensitive data on the user's device, ensuring privacy and low latency. The project demonstrates the feasibility of edge AI for sensitive applications.

Privacy is no longer just a feature; it's a human right—especially when it comes to mental health. Imagine a Cognitive Behavioral Therapy CBT assistant that lives entirely in your browser, never sends a single byte of your conversation to a remote server, and runs at lightning speed. Thanks to the explosion of Edge AI and the maturing WebGPU standard, this isn't science fiction anymore. In this tutorial, we will explore how to use WebLLM , TVM Unity , and React to build a high-performance, offline-capable mental health assistant. By leveraging WebLLM and Local-first AI principles, we can provide low-latency support while keeping sensitive user data exactly where it belongs: on the user's device. Mental health data is incredibly sensitive. Using traditional LLM APIs like OpenAI or Claude means sending private thoughts to the cloud. By using WebGPU acceleration , we can run models like Llama 3 or Mistral directly on the client's GPU via the browser. The magic happens through TVM Unity , which compiles machine learning models into high-performance kernels that the browser can execute via the WebGPU API. php graph TD A User Input -- B React UI State B -- C WebLLM Worker subgraph Browser Environment C -- D TVM Runtime D -- E WebGPU API E -- F Local GPU / VRAM F -- G Model Inference G -- D end D -- H Streaming Response H -- B I IndexedDB Cache -.- C To follow along, you’ll need: React , WebLLM , and Vite .First, let's install the core dependencies: npm install @mlc-ai/web-llm react The heart of our application is the ChatWorker . We want to run the LLM in a Web Worker to ensure the UI remains responsive during heavy inference. js // engine.ts import { CreateWebWorkerEngine, type ChatOptions } from "@mlc-ai/web-llm"; const SELECTED MODEL = "Llama-3-8B-Instruct-q4f16 1-MLC"; export async function initializeEngine onProgress: p: any = void { // This downloads the model and initializes the WebGPU pipeline const engine = await CreateWebWorkerEngine new Worker new URL "./worker.ts", import.meta.url , { type: "module" } , SELECTED MODEL, { initProgressCallback: onProgress } ; return engine; } For a mental health assistant, the system prompt is everything. We need to steer the model toward Cognitive Behavioral Therapy techniques—identifying cognitive distortions and suggesting reframing exercises. js const CBT SYSTEM PROMPT = You are a supportive Mental Health Assistant specialized in Cognitive Behavioral Therapy CBT . Your goal is to help users identify negative thought patterns. Rules: 1. Be empathetic and non-judgmental. 2. If a user mentions self-harm, immediately provide emergency resources. 3. Use Socratic questioning to help users reach their own conclusions. 4. Keep data privacy top-of-mind: remind users this is a local-only session. ; Now, let's build a custom hook to manage the chat state and the engine lifecycle. 🚀 js // useWebLLM.ts import { useState, useEffect } from 'react'; import as webllm from "@mlc-ai/web-llm"; export function useWebLLM { const engine, setEngine = useState<webllm.EngineInterface | null null ; const progress, setProgress = useState "" ; useEffect = { async function init { const instance = await webllm.CreateMLCEngine "Llama-3-8B-Instruct-q4f16 1-MLC", { initProgressCallback: p = setProgress p.text } ; setEngine instance ; } init ; }, ; const chat = async messages: webllm.ChatCompletionMessageParam = { if engine return; const chunks = await engine.chat.completions.create { messages: { role: "system", content: CBT SYSTEM PROMPT }, ...messages , stream: true, } ; return chunks; }; return { chat, progress, ready: engine }; } While this demo gets you up and running, deploying Edge AI at scale requires deep optimization regarding model quantization, caching strategies, and cross-browser compatibility. If you are looking for advanced patterns on model sharding or more production-ready examples of local-first architectures, you should definitely check out the deep-dives at WellAlly Tech Blog https://www.wellally.tech/blog . They cover the nuances of TVM Unity and memory management that are crucial for high-traffic applications. Finally, we connect our hook to a simple chat interface. Note how we handle the streaming response to create that "typing" effect users love. js // App.tsx import { useState } from 'react'; import { useWebLLM } from './useWebLLM'; function App { const { chat, progress, ready } = useWebLLM ; const input, setInput = useState "" ; const messages, setMessages = useState ; const handleSend = async = { const userMsg = { role: "user", content: input }; setMessages prev = ...prev, userMsg ; setInput "" ; const stream = await chat ...messages, userMsg ; let fullReply = ""; for await const chunk of stream { fullReply += chunk.choices 0 ?.delta?.content || ""; // Update UI with the stream setMessages prev = ...prev.slice 0, -1 , { role: "assistant", content: fullReply } ; } }; if ready return <div Loading AI Engine: {progress}</div ; return <div className="chat-container" <h2 Local CBT Assistant 🌿</h2 <div className="messages" {messages.map m, i = <div key={i} className={m.role} {m.content}</div } </div <input value={input} onChange={ e = setInput e.target.value } / <button onClick={handleSend} Talk</button </div ; } By combining WebLLM and WebGPU , we've built a tool that is: Edge AI is transforming how we think about sensitive applications like mental health, finance, and personal journaling. By moving the "brain" to the browser, we empower users to own their data without sacrificing the power of modern LLMs. What are you planning to build with WebGPU? Let me know in the comments And don't forget to head over to wellally.tech/blog https://www.wellally.tech/blog for more technical guides on the future of decentralized AI 🚀🔥