Privacy First: Run Your Own Health Assistant LLM Entirely in the Browser (No Backend Required!)

Have you ever wondered why your most personal health queries need to travel across the globe to a centralized server just to get a simple answer? In an era where privacy-preserving AI is becoming a necessity rather than a luxury, the paradigm of Edge AI is shifting the landscape.

By leveraging WebLLM and the raw power of WebGPU, we can now execute high-performance Large Language Models (LLMs) directly within the browser sandbox. No API keys, no server costs, and most importantly—zero data leakage. Today, we are building a private health consultation bot that runs 100% client-side.

Before we dive into the code, let’s talk about why this matters. Traditional AI architectures rely on heavy GPU clusters. However, with the advent of the WebGPU API, we can tap into the user's local hardware. This approach offers:

If you are interested in more production-ready examples and advanced architectural patterns for decentralized AI, I highly recommend checking out the deep dives over at WellAlly Tech Blog.

To make this work, we use TVM (Apache TVM) as the compilation stack, which allows models to run on different backends, and WebLLM as the high-level interface for the browser.

graph TD
    A[User Input] --> B[React Frontend]
    B --> C[WebLLM Worker]
    C --> D{WebGPU Support?}
    D -- Yes --> E[TVM.js Runtime]
    D -- No --> F[Fallback/Error]
    E --> G[IndexedDB Model Cache]
    G --> H[Local GPU Inference]
    H --> I[Streamed Response]
    I --> B

To follow this tutorial, ensure you have:

tech_stack

: First, we need to initialize the MLCEngine

. Since LLMs are heavy, we should run the inference engine inside a Web Worker to keep the UI thread buttery smooth. 🚀

// engine.ts
import { CreateMLCEngine, MLCEngine } from "@mlc-ai/web-llm";

const modelId = "Llama-3-8B-Instruct-q4f16_1-MLC"; // Quantized for browser use

export async function initializeEngine(onProgress: (p: number) => void) {
  const engine = await CreateMLCEngine(modelId, {
    initProgressCallback: (report) => {
      onProgress(Math.round(report.progress * 100));
    },
  });
  return engine;
}

We want a clean way to interact with our local model. Let's wrap the logic into a custom hook.

// useHealthAI.ts
import { useState } from 'react';
import { initializeEngine } from './engine';

export const useHealthAI = () => {
  const [engine, setEngine] = useState<any>(null);
  const [, set] = useState(false);
  const [progress, setProgress] = useState(0);

  const boot = async () => {
    set(true);
    const inst = await initializeEngine((p) => setProgress(p));
    setEngine(inst);
    set(false);
  };

  const askHealthQuestion = async (prompt: string) => {
    const messages = [
      { role: "system", content: "You are a private health assistant. Provide concise, empathetic advice. Always suggest seeing a doctor for serious issues." },
      { role: "user", content: prompt }
    ];

    const reply = await engine.chat.completions.create({ messages });
    return reply.choices[0].message.content;
  };

  return { boot, askHealthQuestion, , progress, ready: !!engine };
};

Now, we integrate this into our React component. Notice how we handle the " Weights" phase—the model is about 4GB-5GB, so clear feedback is key!

// App.tsx
import React, { useState } from 'react';
import { useHealthAI } from './useHealthAI';

function App() {
  const { boot, askHealthQuestion, , progress, ready } = useHealthAI();
  const [input, setInput] = useState("");
  const [answer, setAnswer] = useState("");

  const handleConsult = async () => {
    const res = await askHealthQuestion(input);
    setAnswer(res);
  };

  return (
    <div className="p-8 max-w-2xl mx-auto">
      <h1 className="text-3xl font-bold">🩺 LocalHealth AI</h1>
      {!ready && ! && (
        <button onClick={boot} className="bg-blue-500 text-white p-2 rounded mt-4">
          Initialize Private Model (WebGPU)
        </button>
      )}

      { && <p>Down Model Weights: {progress}%</p>}

      {ready && (
        <div className="mt-6">
          <textarea 
            className="w-full border p-2" 
            placeholder="How can I help you today?"
            onChange={(e) => setInput(e.target.value)}
          />
          <button onClick={handleConsult} className="bg-green-600 text-white p-2 mt-2">
            Ask Locally
          </button>
          <div className="mt-4 p-4 bg-gray-100 rounded italic">
            {answer || "Response will appear here..."}
          </div>
        </div>
      )}
    </div>
  );
}

When running LLMs in the browser, you must be aware of device constraints.

For a deeper dive into securing Edge AI workloads and optimizing TVM runtimes for production environments, don't forget to visit the WellAlly Tech Engineering Blog. It’s the source of inspiration for this architecture!

By moving the "brain" of our application to the user's device, we've eliminated latency, server costs, and privacy risks in one fell swoop. While WebGPU and WebLLM are still evolving, the ability to run a "Llama" in a browser tab is nothing short of magic.

What will you build next? A private journal? A local-first coding assistant? Let me know in the comments! 👇