{"slug": "building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0-query", "title": "Building a 3-tier on-device AI concierge: Gemini Nano -> MiniLM -> keyword, $0/query", "summary": "A developer built a three-tier on-device AI concierge for websites that runs entirely in the visitor's browser, using Google's Gemini Nano, MiniLM embeddings, and keyword matching, with zero server cost and full privacy. The system degrades gracefully through tiers, each labeled in the UI, and loads no models or data until the user clicks the chat launcher. The knowledge base is pre-embedded and shipped as a base64-encoded Float32Array, enabling local cosine similarity search without any network calls.", "body_md": "Most \"AI chat widget\" tutorials assume you're calling out to an LLM API. This post is about the version where you don't -- where the model runs in the visitor's own browser, the cost per conversation is exactly zero, and nothing about what someone asks ever leaves their machine.\n\nA server-side LLM call is simple to build and expensive/slow/privacy-leaky to run at scale on a marketing site that gets real traffic. For a small business's website, \"every visitor's chat questions get logged on someone's server\" is also just a worse default than it needs to be.\n\nEvery tier is visibly labeled in the UI -- the badge literally says which one answered. Degrading honestly beats pretending.\n\nThe whole widget is dead weight until someone clicks the launcher. No KB fetch, no model download, no Transformers.js import -- all of it is gated behind a boot() call that only fires on first open:\n\n```\nlaunch.addEventListener('click', function () {\n  if (panel.hasAttribute('hidden')) {\n    panel.removeAttribute('hidden');\n    boot(); // KB + tier detection only starts here\n  }\n});\n```\n\nboot() pulls a /concierge/kb-data.js script tag, then tries tier 1, then tier 2, falling through to tier 3 as a last resort:\n\n``` js\nfunction boot() {\n  var s = document.createElement('script');\n  s.src = '/concierge/kb-data.js';\n  s.onload = function () {\n    KB = window.LE_KB;\n    decodeVecs();\n    tryNano().then(function (ok) {\n      if (ok) return;\n      tryEmbed().then(function (ok2) {\n        if (!ok2) { tier = 3; setBadge('keyword match - on-device - private'); }\n      });\n    });\n  };\n  document.head.appendChild(s);\n}\n```\n\nChrome's on-device model API exposes an availability() check that can return 'available', 'downloadable', or 'unavailable'. It would be easy to treat 'downloadable' as a yes and kick off the download -- but that's a multi-gigabyte model pull with no progress UI in a 384px corner widget. So the code only takes a model that's already sitting ready:\n\n``` js\nvar avail = await LM.availability();\nif (avail !== 'available') return resolve(false); // no silent multi-GB download\nnanoSession = await LM.create({ initialPrompts: [{ role: 'system', content: SYSTEM }] });\n```\n\nWhen Nano answers, it's not just freeform generation -- it's grounded (RAG-style) in whatever the retrieval layer finds first, then asked to answer from that context:\n\n``` js\nvar hits = embedFn ? cosineTop(await embedFn(q)) : liteTop(q);\nvar ctx = hits.map(h => 'Q: ' + h.e.q + '\\nA: ' + h.e.a).join('\\n\\n');\nvar prompt = 'Knowledge:\\n' + ctx + '\\n\\nVisitor question: ' + q + '\\nAnswer as the concierge...';\n```\n\nNano's raw output also gets run through a markdown/URL stripper before it ever touches the DOM -- a free-text model will happily emit bullet points, bold, and links that don't belong in a plain chat bubble, and the widget's own rule is that it never pastes a URL out loud in a conversational reply.\n\nThe knowledge base ships pre-embedded. A build step (this component credits it as \"Labs' build-kb.mjs\") computes MiniLM embeddings for every KB entry ahead of time and packs them as a base64-encoded Float32Array baked directly into /public/concierge/kb-data.js. At runtime, that's just:\n\n``` js\nfunction decodeVecs() {\n  var bin = atob(KB.vecsB64), buf = new ArrayBuffer(bin.length), u8 = new Uint8Array(buf);\n  for (var i = 0; i < bin.length; i++) u8[i] = bin.charCodeAt(i);\n  VECS = new Float32Array(buf);\n}\n```\n\nNo vector database, no network call for retrieval -- just a flat array and a manual cosine similarity loop over it once the query itself gets embedded live in the browser:\n\n``` js\nfunction cosineTop(qv, k) {\n  var scores = [];\n  for (var i = 0; i < KB.entries.length; i++) {\n    var s = 0, off = i * KB.dims;\n    for (var d = 0; d < KB.dims; d++) s += qv[d] * VECS[off + d];\n    scores.push([s, i]);\n  }\n  scores.sort((a, b) => b[0] - a[0]);\n  return scores.slice(0, k).map(p => ({ score: p[0], e: KB.entries[p[1]] }));\n}\n```\n\nBecause the embeddings are normalized at encode time, that dot product *is* the cosine similarity -- no separate normalization step needed at query time.\n\nThe tricky part isn't the math, it's version drift: the build-time embedding model and the runtime one have to be the exact same version, or the vector space doesn't line up and every similarity score comes back meaningless. The fix is a version pin baked right into the KB data itself (KB.tfVer), so the runtime import is locked to whatever version actually produced the vectors it's searching:\n\n``` js\nvar mod = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@' + KB.tfVer);\nvar p = await mod.pipeline('feature-extraction', KB.model);\n```\n\nIf the cosine match is weak (score under 0.25), the widget doesn't return a low-confidence guess -- it falls through to tier 3 instead.\n\nThe floor is a stopword-filtered keyword scorer -- no model, no embeddings, works in any browser:\n\n``` js\nfunction liteTop(q, k) {\n  var toks = q.toLowerCase().replace(/[^a-z0-9\\s]/g, ' ').split(/\\s+/).filter(t => t && !STOP[t]);\n  var scores = KB.entries.map((e, i) => {\n    var hay = (e.q + ' ' + e.a + ' ' + e.tags.join(' ')).toLowerCase();\n    var s = 0;\n    toks.forEach(t => { if (hay.indexOf(t) !== -1) s += t.length > 5 ? 2 : 1; });\n    return [s, i];\n  });\n  return scores.sort((a, b) => b[0] - a[0]).slice(0, k).filter(p => p[0] > 0);\n}\n```\n\nLonger, more specific tokens (>5 characters) score higher than short ones -- a crude but effective way to weight \"concierge\" over \"the.\"\n\nThe one thing that can't happen is a visitor hitting a wall. If Nano's session creation succeeds but the actual prompt() call throws mid-conversation (it happens -- Nano is still an experimental API), the widget demotes itself to tier 2 or 3 in the middle of that same request and answers from retrieval instead of surfacing an error:\n\n```\ncatch (e) {\n  if (tier === 1) {\n    tier = embedFn ? 2 : 3;\n    nanoSession = null;\n    // ...retry via retrieveAnswer() instead of failing the turn\n  }\n}\n```\n\nAlongside the human-facing chat, every page also registers structured tools via document.modelContext.registerTool() -- get_services, get_pricing, start_free_audit, get_contact -- so a browser AI agent visiting the page programmatically doesn't have to scrape the DOM to find that information. Different surface, same underlying idea: the site should be legible to something other than a human with a mouse.\n\nOn-device inference is quietly becoming viable for real product surfaces, not just demos. If your product doesn't need a frontier model's full capability -- and most support/FAQ-style chat doesn't -- the on-device path is worth taking seriously before defaulting to a server call. Three tiers sounds like a lot of engineering for a corner-of-the-screen chat widget, but each one is genuinely simple in isolation; the value is entirely in never leaving a visitor stuck when the fanciest tier isn't available.\n\nLive example: [https://localenhance.com](https://localenhance.com) (bottom-right \"Ask AI\" widget).", "url": "https://wpnews.pro/news/building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0-query", "canonical_source": "https://dev.to/tony_hildn_26f6eb18f87d2/building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0query-30aj", "published_at": "2026-07-16 17:51:47+00:00", "updated_at": "2026-07-16 18:34:30.841475+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-products", "developer-tools"], "entities": ["Google", "Gemini Nano", "MiniLM", "Transformers.js"], "alternates": {"html": "https://wpnews.pro/news/building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0-query", "markdown": "https://wpnews.pro/news/building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0-query.md", "text": "https://wpnews.pro/news/building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0-query.txt", "jsonld": "https://wpnews.pro/news/building-a-3-tier-on-device-ai-concierge-gemini-nano-minilm-keyword-0-query.jsonld"}}