{"slug": "i-put-a-real-llm-in-my-portfolio-it-runs-on-your-gpu-not-my-server", "title": "I put a real LLM in my portfolio. It runs on your GPU, not my server.", "summary": "Engineer Rahul Agarwal built a portfolio site that runs a 0.5B-parameter Qwen2.5-Instruct model entirely in the visitor's browser via WebGPU and MLC's WebLLM, with no server or API key involved. The chat answers questions about his work through a keyword-scoring retrieval step over a dozen hand-written notes, constrained by a strict system prompt to avoid hallucination, and includes a warm-up generation to hide WebGPU's lazy shader compilation. He also fixed a Cumulative Layout Shift of 1.41 caused by a text-scramble headline animation by locking each word to its real width inside inline-block spans.", "body_md": "Every AI-engineer portfolio says \"I build with LLMs.\" I wanted mine to *prove* it in the first ten seconds.\n\nSo the chat box on my homepage has a button: **Run a real LLM on your GPU**. Click it and a 0.5B-parameter model downloads into your browser, boots on your graphics card through WebGPU, and answers questions about my work. No server. No API key. Nothing you type leaves your machine.\n\n👉 **Try it:** [rahul-agarwal.in](https://www.rahul-agarwal.in) (desktop Chrome or Edge works best)\n\nHere's how it works, and the four things I got wrong before it worked well.\n\n``` js\nconst adapter = await navigator.gpu.requestAdapter();\nconst f16 = adapter.features.has('shader-f16');\nconst model = f16\n ? 'Qwen2.5-0.5B-Instruct-q4f16_1-MLC'\n : 'Qwen2.5-0.5B-Instruct-q4f32_1-MLC'; // fallback for GPUs without fp16\n\nconst webllm = await import('https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.86/+esm');\nconst engine = await webllm.CreateMLCEngine(model, {\n initProgressCallback: r => updateProgressBar(r.progress, r.text),\n});\n```\n\nThat's it. WebLLM exposes an OpenAI-style API, so streaming looks familiar:\n\n``` js\nconst chunks = await engine.chat.completions.create({\n messages, stream: true, temperature: 0.2, max_tokens: 160,\n});\nfor await (const c of chunks) {\n const token = c.choices[0]?.delta?.content || '';\n output.textContent += token;\n window.neural.fire(1); // light up the hero network\n}\n```\n\nTwo details matter: **pin the CDN version** (an unpinned import can change under you), and **check for `shader-f16`** instead of assuming it. Plenty of laptop GPUs don't have it.\n\nA half-billion-parameter model knows nothing about me and will happily invent things. So it never answers from memory. Every question goes through a retrieval step first:\n\n```\n// Simplified from the real code, which also weights longer keywords higher\nfunction retrieve(question) {\n const q = question.toLowerCase();\n return NOTES\n .map(note => ({ note, score: note.keys.filter(k => q.includes(k)).length }))\n .filter(x => x.score > 0)\n .sort((a, b) => b.score - a.score)\n .slice(0, 2)\n .map(x => x.note);\n}\n\nconst SYSTEM = `Answer in 1 to 3 short sentences using ONLY the notes provided.\nIf the notes don't contain the answer, say you don't know.\nNever invent companies, numbers, dates or links.`;\n```\n\nThe notes are a dozen short paragraphs I wrote (projects, stack, experience). Keyword scoring sounds crude next to embeddings, but with a dozen documents it's instant, deterministic and easy to debug. Under every answer the UI shows what it used:\n\n**Q:** Where does he work now?\n\n**A:** Rahul is the sole engineer at Aryan News Agency, where he builds and develops websites and apps.\n\n↳ retrieved 1 note (intro) · 22 tokens in 2.8s · generated on your GPU\n\nAnd the test I cared about most:\n\n**Q:** What is his favourite food?\n\n**A:** Rahul is not specifically known for his favorite food.\n\nNo hallucination. Low temperature plus a strict \"only the notes\" prompt does most of the work.\n\nThe model loaded fine, but the first question sat there \"thinking.\" WebGPU compiles its shaders lazily, on the first real inference. The fix is a throwaway warm-up generation while the progress bar is still visible:\n\n```\nawait engine.chat.completions.create({\n messages: [{ role: 'user', content: 'Hi' }], max_tokens: 4,\n});\n```\n\nNow the slow part happens during loading, where users expect waiting, not after they've asked a question.\n\nMy headline \"decodes\" from random glyphs into the real text. It looked great, and it gave the page a **Cumulative Layout Shift of 1.41** (anything above 0.25 is \"poor\"). Random glyphs have different widths, so the line re-wrapped about 60 times a second and shoved everything below it around.\n\nThe fix: wrap each word in an inline-block, lock it to its real width, scramble inside it, then unwrap.\n\n``` js\nfor (const w of words) {\n w.span.style.display = 'inline-block';\n w.span.style.width = w.span.getBoundingClientRect().width + 'px';\n}\n// ...animate...then restore plain text nodes\nwords.forEach(w => w.span.replaceWith(document.createTextNode(w.text)));\n```\n\nSame effect, layout shift **0.00**. One more catch: measure *after* `document.fonts.ready`, or you lock in the fallback font's widths.\n\nI added a strict Content-Security-Policy (hash-based, no `unsafe-inline`). Locally: zero violations. In production: errors. My domain was proxied through Cloudflare *in front of* Vercel, and Cloudflare was injecting its own inline scripts (analytics and bot detection) and rewriting my email address into an obfuscated link. That put an `` inside my \"copy email\" `