The 7 MB Embedding Model Bringing Semantic Search to the Browser

Ternlight, a 7 MB embedding model using ternary quantization and WebAssembly, enables fully client-side semantic search in browsers, bypassing server round-trips and API costs. The model runs locally via a lightweight WASM engine with SIMD support, offering practical vector embeddings for applications like retrieval-augmented generation and real-time similarity features.

AI https://sourcefeed.dev/c/ai Article The 7 MB Embedding Model Bringing Semantic Search to the Browser Ternlight proves that client-side vector embeddings are finally practical, bypassing server round-trips and API costs entirely. Priya Nair https://sourcefeed.dev/u/priya nair We have been promised local, browser-native AI for years. But downloading a 2 GB quantized language model over a mobile connection is a hard sell for most web applications. It is slow, eats data caps, and hogs device memory. While running a full generative model locally is often impractical, generating vector embeddings is a completely different story. Enter Ternlight Demo https://ternlight-demo.vercel.app/ , an embedding model that runs entirely client-side. At just 7 MB, the model is smaller than many modern hero images, yet it opens the door to fully local semantic search, client-side retrieval-augmented generation RAG chunking, and real-time similarity features. By combining ternary quantization with a lightweight WebAssembly engine, it bypasses the latency, cost, and privacy concerns of traditional API-based embedding pipelines. The Math Behind the Shrinkage: Ternary Weights How does an embedding model shrink to 7 MB without becoming completely useless? The secret lies in ternary quantization, specifically using BitLinear layers. Traditional deep learning models represent weights using 16-bit or 32-bit floating-point numbers. Ternary quantization restricts these weights to just three possible values: -1, 0, and 1. This architecture requires only 1.58 bits per parameter. Beyond the massive reduction in storage size, ternary weights fundamentally alter the compute profile. Instead of expensive floating-point multiplications, the processor can perform matrix multiplication using simple additions and subtractions. For a browser running on a user's CPU, this is a massive win. It bypasses the need for heavy GPU acceleration for basic text representation tasks. The Browser Runtime Stack: WASM, SIMD, and Caching To run this model efficiently, Ternlight relies on @ternlight/mini , a lightweight engine compiled to WebAssembly https://webassembly.org . While WebGPU is the gold standard for heavy parallel workloads, WebAssembly remains the universal fallback for CPU-bound execution. Modern browsers support WebAssembly SIMD Single Instruction, Multiple Data . If WebAssembly.validate returns true for SIMD, the engine can run quantized operations on the CPU roughly 2 to 4 times faster than standard FP32 WASM. This makes the latency of generating a single embedding negligible on modern devices. To make this practical for production, developers must implement an aggressive caching strategy. You do not want to fetch the 7 MB model payload on every page load. The standard pattern is to check IndexedDB or the Cache API first, download the model once, and cache it for subsequent sessions. flowchart TD subgraph Traditional API-Based Search Client1 Client Browser -- |1. Send Query| Server Backend Server Server -- |2. Get Embedding| OpenAI Embedding API OpenAI -- |3. Return Vector| Server Server -- |4. Query DB| VectorDB Vector Database VectorDB -- |5. Return Results| Server Server -- |6. Send Results| Client1 end subgraph Local-First Search Client2 Client Browser -- |1. Load Cached Model| LocalEngine Local WASM Engine Client2 -- |2. Generate Embedding| LocalEngine LocalEngine -- |3. Local Vector Search| LocalIndex Local Index / IndexedDB end Developer Angle: Architecture and Implementation To keep the user interface responsive while generating embeddings, you should never run inference on the main browser thread. Instead, offload the execution to a WebWorker. This architecture, championed by projects like Mozilla AI https://mozilla.ai , ensures that the main thread remains free to handle user interactions. Here is a practical implementation of how you can set up a WebWorker to load the Ternlight model, cache it in IndexedDB, and handle embedding queries. js // worker.js import { TernlightEngine } from '@ternlight/mini'; let engine = null; async function getModelBuffer { const cacheName = 'ternlight-model-cache'; const modelUrl = '/models/ternlight-mini.wasm'; const cache = await caches.open cacheName ; let response = await cache.match modelUrl ; if response { // Fetch and cache on first load await cache.add modelUrl ; response = await cache.match modelUrl ; } return await response.arrayBuffer ; } self.onmessage = async event = { const { type, payload } = event.data; if type === 'INIT' { try { const buffer = await getModelBuffer ; engine = new TernlightEngine buffer ; self.postMessage { type: 'READY' } ; } catch err { self.postMessage { type: 'ERROR', error: err.message } ; } } if type === 'EMBED' { if engine { self.postMessage { type: 'ERROR', error: 'Engine not initialized' } ; return; } const startTime = performance.now ; const vector = engine.embed payload.text ; const latency = performance.now - startTime; self.postMessage { type: 'EMBED COMPLETE', payload: { vector, latency } } ; } }; In your main application code, you instantiate the worker and communicate via standard message passing: js // app.js const worker = new Worker new URL './worker.js', import.meta.url ; worker.postMessage { type: 'INIT' } ; worker.onmessage = event = { const { type, payload } = event.data; if type === 'READY' { console.log 'Ternlight is ready for local inference.' ; worker.postMessage { type: 'EMBED', payload: { text: 'How do I share state across React components?' } } ; } if type === 'EMBED COMPLETE' { console.log Generated embedding in ${payload.latency.toFixed 2 }ms: , payload.vector ; } }; The Real-World Trade-offs Before ripping out your backend pgvector setup, you need to consider the constraints of a 7 MB model. First, the context window and output dimensionality are naturally limited compared to cloud-hosted models like OpenAI's text-embedding-3-small or Cohere's v3 models. Ternlight is designed for short-to-medium text chunks, such as documentation search, UI autocomplete, or local browser history indexing. If you are trying to embed entire PDF books at once, a 7 MB model will struggle with representation quality. Second, while 7 MB is incredibly small for an AI model, it is still a non-trivial payload for users on slow mobile connections. You should lazy-load the engine only when the user interacts with a feature that requires semantic search, rather than blocking the initial page bundle. For documentation sites, offline-first applications, and privacy-sensitive tools where data cannot leave the user's device, Ternlight represents a massive step forward. It proves that we do not always need massive GPU clusters to build intelligent search features. Sources & further reading - Ternlight – 7 MB embedding model that runs in browser WASM https://ternlight-demo.vercel.app/ — ternlight-demo.vercel.app - Linux Report | Latest Linux News https://linuxreport.net/ — linuxreport.net - 3W for In-Browser AI: WebLLM + WASM + WebWorkers https://blog.mozilla.ai/3w-for-in-browser-ai-webllm-wasm-webworkers/ — blog.mozilla.ai - Run AI Models in the Browser with WebGPU & WASM https://maddevs.io/writeups/running-ai-models-locally-in-the-browser/ — maddevs.io Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.