{"slug": "100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-js", "title": "100% Private Skin Screening: Building an Edge AI Vision App with WebGPU and Transformers.js", "summary": "A developer built a browser-based skin lesion screening app that runs entirely on-device using WebGPU acceleration, Transformers.js, and WebLLM, so no images are uploaded to a server. The app loads a quantized Vision Transformer locally for image classification and uses an in-browser Llama-3-8B model to explain results in plain language, with the author noting that production deployment requires quantization to cut model size from roughly 300MB to about 80MB.", "body_md": "What if you could screen for skin health issues without ever uploading a single photo to a corporate server? In the era of massive data breaches and privacy concerns, \"sending data to the cloud\" is becoming a liability, especially for sensitive medical imagery.\n\nToday, we are diving deep into the world of **Edge AI** and **Privacy-First Machine Learning**. We will build a skin lesion screening application that runs entirely in the browser using **WebGPU acceleration**, **Transformers.js**, and **WebLLM**. By leveraging on-device computation, we ensure that user data stays strictly within the browser sandbox.\n\nKeywords: *Edge AI*, *WebGPU Acceleration*, *Privacy-Preserving AI*, *Transformers.js Tutorial*, *On-device Machine Learning*.\n\nTraditional AI apps send images to a Python backend. Our approach flips the script. We download the model weights once and execute the inference locally using the user's GPU.\n\n``` php\ngraph TD\n    A[User Uploads Image] --> B{Browser Environment}\n    B --> C[WebGPU Tensors]\n    C --> D[Transformers.js Vision Model]\n    D --> E[Skin Lesion Classification]\n    E --> F[WebLLM Assistant]\n    F --> G[Local Privacy-First Report]\n    B -.->|No Data Transmitted| H[External Internet]\n    style H fill:#f96,stroke:#333,stroke-dasharray: 5 5\n```\n\nBefore we start, ensure your browser (Chrome 113+ or Edge) supports WebGPU.\n\nFirst, we need to initialize our image classification model. We'll use a pre-trained Vision Transformer (ViT) fine-tuned on medical datasets.\n\n``` js\nimport { pipeline, env } from '@xenova/transformers';\n\n// Enable WebGPU if available\nenv.allowLocalModels = false;\nenv.useBrowserCache = true;\n\nconst useSkinClassifier = () => {\n  const [classifier, setClassifier] = useState(null);\n\n  useEffect(() => {\n    const initModel = async () => {\n      // Initialize the pipeline with WebGPU execution provider\n      const pipe = await pipeline('image-classification', 'Xenova/vit-base-patch16-224', {\n        device: 'webgpu', \n      });\n      setClassifier(() => pipe);\n    };\n    initModel();\n  }, []);\n\n  return classifier;\n};\n```\n\nWhen a user selects a file, we convert it into a format `Transformers.js` understands without any multipart/form-data uploads.\n\n``` js\nconst handleUpload = async (event: React.ChangeEvent<HTMLInputElement>) => {\n  const file = event.target.files?.[0];\n  if (!file || !classifier) return;\n\n  const url = URL.createObjectURL(file);\n\n  // Running inference 100% locally!\n  const output = await classifier(url);\n\n  console.log(\"Classification Results:\", output);\n  // Example output: [{ label: 'Melanocytic nevi', score: 0.98 }]\n};\n```\n\nTo make the screening \"human-readable,\" we use **WebLLM** to explain the results. This allows the app to provide context while keeping the \"AI logic\" on the edge.\n\n``` js\nimport { CreateMLCEngine } from \"@mlc-ai/web-llm\";\n\nasync function explainResults(label: string) {\n  const engine = await CreateMLCEngine(\"Llama-3-8B-Instruct-q4f16_1-MLC\");\n  const response = await engine.chat.completions.create({\n    messages: [\n      { role: \"system\", content: \"You are a helpful medical assistant. Explain what this skin condition label means in simple terms.\" },\n      { role: \"user\", content: `Explain the label: ${label}` }\n    ]\n  });\n  return response.choices[0].message.content;\n}\n```\n\nWhile building local-first apps is exciting, deploying medical-grade AI requires rigorous version control, model quantization, and robust fallback mechanisms.\n\nFor more production-ready examples and advanced patterns on optimizing WebGPU shaders for mobile browsers, check out the detailed guides at **[WellAlly Tech Blog](https://www.wellally.tech/blog)**. It’s my go-to resource for scaling Edge AI applications beyond simple prototypes.\n\nStandard Vision Transformers can be ~300MB. For a production app, use **Quantization** (Int8 or O4) to reduce the model size to ~80MB without significant accuracy loss.\n\nThe first time the model runs, WebGPU compiles the shaders. \n\n*Tip*: Run a \"dummy inference\" with a blank 1x1 pixel image as soon as the app loads to prevent UI lag during actual usage.\n\nWe’ve just built a foundation for a 100% private, browser-based medical screening tool. By combining **Transformers.js** and **WebGPU**, we respect user privacy while providing high-performance AI capabilities.\n\nThe future of AI isn't just in the cloud—it's right there in your browser's console. 🛠️\n\n**What’s next?**\n\n`MobileNetV3` for even faster speeds.\nIf you enjoyed this tutorial, drop a comment below and let me know what Edge AI project you're working on! Happy coding! 🚀\n\n*Disclaimer: This tool is for educational purposes and is not a substitute for professional medical advice. Always consult a dermatologist.*", "url": "https://wpnews.pro/news/100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-js", "canonical_source": "https://dev.to/beck_moulton/100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-transformersjs-c6c", "published_at": "2026-09-27 00:41:00+00:00", "updated_at": "2026-09-27 01:31:04.343236+00:00", "lang": "en", "topics": ["computer-vision", "ai-tools", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Transformers.js", "WebGPU", "WebLLM", "Llama-3-8B-Instruct", "Xenova", "WellAlly Tech Blog"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-js", "markdown": "https://wpnews.pro/news/100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-js.md", "text": "https://wpnews.pro/news/100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-js.txt", "jsonld": "https://wpnews.pro/news/100-private-skin-screening-building-an-edge-ai-vision-app-with-webgpu-and-js.jsonld"}}