AIArticle
Ternlight proves that client-side vector embeddings are finally practical, bypassing server round-trips and API costs entirely.
We have been promised local, browser-native AI for years. But down a 2 GB quantized language model over a mobile connection is a hard sell for most web applications. It is slow, eats data caps, and hogs device memory. While running a full generative model locally is often impractical, generating vector embeddings is a completely different story.
Enter Ternlight Demo, an embedding model that runs entirely client-side. At just 7 MB, the model is smaller than many modern hero images, yet it opens the door to fully local semantic search, client-side retrieval-augmented generation (RAG) chunking, and real-time similarity features. By combining ternary quantization with a lightweight WebAssembly engine, it bypasses the latency, cost, and privacy concerns of traditional API-based embedding pipelines.
The Math Behind the Shrinkage: Ternary Weights #
How does an embedding model shrink to 7 MB without becoming completely useless? The secret lies in ternary quantization, specifically using BitLinear layers.
Traditional deep learning models represent weights using 16-bit or 32-bit floating-point numbers. Ternary quantization restricts these weights to just three possible values: -1, 0, and 1. This architecture requires only 1.58 bits per parameter. Beyond the massive reduction in storage size, ternary weights fundamentally alter the compute profile.
Instead of expensive floating-point multiplications, the processor can perform matrix multiplication using simple additions and subtractions. For a browser running on a user's CPU, this is a massive win. It bypasses the need for heavy GPU acceleration for basic text representation tasks.
The Browser Runtime Stack: WASM, SIMD, and Caching #
To run this model efficiently, Ternlight relies on @ternlight/mini
, a lightweight engine compiled to WebAssembly. While WebGPU is the gold standard for heavy parallel workloads, WebAssembly remains the universal fallback for CPU-bound execution.
Modern browsers support WebAssembly SIMD (Single Instruction, Multiple Data). If WebAssembly.validate()
returns true for SIMD, the engine can run quantized operations on the CPU roughly 2 to 4 times faster than standard FP32 WASM. This makes the latency of generating a single embedding negligible on modern devices.
To make this practical for production, developers must implement an aggressive caching strategy. You do not want to fetch the 7 MB model payload on every page load. The standard pattern is to check IndexedDB or the Cache API first, download the model once, and cache it for subsequent sessions.
flowchart TD
subgraph Traditional API-Based Search
Client1[Client Browser] -->|1. Send Query| Server[Backend Server]
Server -->|2. Get Embedding| OpenAI[Embedding API]
OpenAI -->|3. Return Vector| Server
Server -->|4. Query DB| VectorDB[(Vector Database)]
VectorDB -->|5. Return Results| Server
Server -->|6. Send Results| Client1
end
subgraph Local-First Search
Client2[Client Browser] -->|1. Load Cached Model| LocalEngine[Local WASM Engine]
Client2 -->|2. Generate Embedding| LocalEngine
LocalEngine -->|3. Local Vector Search| LocalIndex[Local Index / IndexedDB]
end
Developer Angle: Architecture and Implementation #
To keep the user interface responsive while generating embeddings, you should never run inference on the main browser thread. Instead, offload the execution to a WebWorker. This architecture, championed by projects like Mozilla AI, ensures that the main thread remains free to handle user interactions.
Here is a practical implementation of how you can set up a WebWorker to load the Ternlight model, cache it in IndexedDB, and handle embedding queries.
// worker.js
import { TernlightEngine } from '@ternlight/mini';
let engine = null;
async function getModelBuffer() {
const cacheName = 'ternlight-model-cache';
const modelUrl = '/models/ternlight-mini.wasm';
const cache = await caches.open(cacheName);
let response = await cache.match(modelUrl);
if (!response) {
// Fetch and cache on first load
await cache.add(modelUrl);
response = await cache.match(modelUrl);
}
return await response.arrayBuffer();
}
self.onmessage = async (event) => {
const { type, payload } = event.data;
if (type === 'INIT') {
try {
const buffer = await getModelBuffer();
engine = new TernlightEngine(buffer);
self.postMessage({ type: 'READY' });
} catch (err) {
self.postMessage({ type: 'ERROR', error: err.message });
}
}
if (type === 'EMBED') {
if (!engine) {
self.postMessage({ type: 'ERROR', error: 'Engine not initialized' });
return;
}
const startTime = performance.now();
const vector = engine.embed(payload.text);
const latency = performance.now() - startTime;
self.postMessage({
type: 'EMBED_COMPLETE',
payload: { vector, latency }
});
}
};
In your main application code, you instantiate the worker and communicate via standard message passing:
// app.js
const worker = new Worker(new URL('./worker.js', import.meta.url));
worker.postMessage({ type: 'INIT' });
worker.onmessage = (event) => {
const { type, payload } = event.data;
if (type === 'READY') {
console.log('Ternlight is ready for local inference.');
worker.postMessage({
type: 'EMBED',
payload: { text: 'How do I share state across React components?' }
});
}
if (type === 'EMBED_COMPLETE') {
console.log(`Generated embedding in ${payload.latency.toFixed(2)}ms:`, payload.vector);
}
};
The Real-World Trade-offs #
Before ripping out your backend pgvector setup, you need to consider the constraints of a 7 MB model.
First, the context window and output dimensionality are naturally limited compared to cloud-hosted models like OpenAI's text-embedding-3-small
or Cohere's v3 models. Ternlight is designed for short-to-medium text chunks, such as documentation search, UI autocomplete, or local browser history indexing. If you are trying to embed entire PDF books at once, a 7 MB model will struggle with representation quality.
Second, while 7 MB is incredibly small for an AI model, it is still a non-trivial payload for users on slow mobile connections. You should lazy-load the engine only when the user interacts with a feature that requires semantic search, rather than blocking the initial page bundle.
For documentation sites, offline-first applications, and privacy-sensitive tools where data cannot leave the user's device, Ternlight represents a massive step forward. It proves that we do not always need massive GPU clusters to build intelligent search features.
Sources & further reading #
Ternlight – 7 MB embedding model that runs in browser (WASM)— ternlight-demo.vercel.app - Linux Report | Latest Linux News— linuxreport.net - 3W for In-Browser AI: WebLLM + WASM + WebWorkers— blog.mozilla.ai - Run AI Models in the Browser with WebGPU & WASM— maddevs.io
Priya Nair· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.