{"slug": "on-device-ai-new-knowledge-without-fine-tuning", "title": "On-device AI: new knowledge without fine-tuning", "summary": "Viper 1.5.34, now in App Store review, ships the open gpt-oss-20b model on Mac with a retrieval harness that feeds the model Viper's own in-app guides at question time, and on a 20-question internal test raised correct answers from 4 (model alone) to 19 (model with guides). Viper said training Viper's facts into model weights made results worse, so it left the model unchanged and built the program around it, keeping questions, knowledge and the model on the device with no API keys and no provider account. The company cautioned that 20 self-written questions is not proof, and reported that at the scale of its larger blind test — hundreds of new Russian and English questions graded by two blind judges — the 1.5.34 approach fell short on fresh situation-style questions.", "body_md": "The model is fixed. The program around it is not.\n\nViper has an AI chat that runs entirely on your Mac. No cloud. No AI provider account. Your questions never leave the device.\n\nIt knows Viper. Ask how keys work, how to move to a new phone, what TON Drop does — it answers from Viper's own guides and shows where the answer came from.\n\nIt can act, too. Ask it to message someone, create an event or save a note, and\nit prepares a draft. Nothing is sent until you tap **Send**.\n\n**A stock open model. Viper's harness. Your device.**\n\n**What you see.** AI Chat in Viper on a Mac, offline. Three everyday questions:\nwhat Viper is, how to turn on Face ID login, and what the clock and the single\ncheck next to a message mean. Each answer comes with **Sources**: the in-app\nguides it was taken from (*Enter via Face ID*, *Message marks*, *About Viper*).\nTap one to open that guide and check the answer yourself.\n\n**Why it matters.** A messenger's help should not need the internet, an account\nwith an AI company or a support chat. People ask about keys, delivery marks and\nprivacy when something feels wrong — often offline, often in a hurry. Viper\nanswers on the device, from Viper's own guides, and points to the source, so\nyou can trust the answer or check it in one tap.\n\nTwo pieces of work this November set us off:\n\n- **[Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming)** (MIT CSAIL, Nov 2026): frontier LLMs can read and edit a neural network's\nweights much like code. Editing a tiny share of a small model's weights\nremoved what held it back — but it did not add abilities the model lacked.\n- **[BotPsy](https://qvac.tether.io/blog/on-device-ai-agent-botpsy)** (Tether QVAC, Nov 2026): a 0.8B model trained to act through tool calls,\nsmall enough to run on a phone.\n\nBoth point at the same goal: a model on your own device that knows your app and can act in it. We tried both roads for Viper — editing weights and training a small model on Viper. Neither taught a model Viper's facts reliably. What worked was leaving the model alone and building the program around it.\n\nWe did not retrain the model to teach it Viper. We tried — training facts into the weights made it worse.\n\nInstead, Viper builds a *harness* around an unchanged open model: the knowledge\nit needs at the moment you ask, the actions it may propose, and checks that run\nbefore every build ships. Swap in a better model tomorrow, and everything Viper\ntaught it stays.\n\n- The model, the knowledge and your questions stay on your Mac.\n- No API keys, no usage logs on someone else's server.\n- The AI proposes; you decide. Every action is a draft until you confirm it.\n- You choose what it knows: turn Viper's built-in knowledge on or off, add your own notes.\n\nViper 1.5.34, now in App Store review, ships gpt-oss-20b on Mac with Viper's\nknowledge: when you ask about Viper, the app gives the model the About Viper\ncard and the two in-app guides that match your question, and the answer shows\nthose guides as **Sources** (the screenshot above is 1.5.34).\n\nOn a small test this worked well: 20 questions about Viper went from 4 right (the model alone) to 19 right (the model with the guides).\n\nTwenty questions we wrote ourselves is not proof. So we built a large blind test:\n\n- **Hundreds of new questions** , written by two different authors (another\nopen model and us), in Russian and English, in the ways people really ask:\nshort, vague, with typos, slang, a situation (\"my old phone broke…\"), several\nquestions at once — and questions about things Viper does not have.\n- **Two blind judges** grade every answer against Viper's knowledge. An answer\ncounts as*correct* only when both say so, and as*invented* when either one\nfinds a made-up feature, menu or number.\n- **Fresh questions decide.** A change counts only when it also wins on\nquestions no version of the harness has seen.\n\nAt this scale the 1.5.34 approach fell short. On fresh situation-style questions it was right about a third of the time, and invented something in about a third of its answers. Reading the failures showed why: the model often had the right guide in front of it and still added things that were not there.\n\nStill no training — the model's weights are untouched. Only the program around it changed:\n\n1. **Guides rewritten for the model.** Every guide page became short numbered\nsteps with the exact menu path from the app's own code, plus plain lines for\nwhat Viper does*not* have (no groups, no QR scanner, no cloud sync).\n2. **The question is taken apart.** A multi-part question is split into\nsub-questions; each finds its own guide, and the model first picks the guide\nlines that answer it.\n3. **Answers come from the guide page itself.** Instead of retelling the guide\nin its own words, the assistant shows the guide pages it picked. When nothing\nfits, it says the guides do not cover it, instead of guessing.\n4. **Checks in code** , not in the prompt: the answer's language must match the\nquestion, a refusal while a matching guide page exists falls back to a normal\nanswer, and actions stay drafts.\n\nThe same model in three setups, on **fresh questions no version has seen**\n(167 Viper questions, two blind judges):\n\n- **Model alone (no harness)** — stock gpt-oss-20b. It knows nothing about\nViper, and it cannot:**gpt-oss-20b was released in 2025, and Viper Messenger\nappeared on the App Store in January 2026.** The model was finished before\nViper existed in public, so everything it knows about Viper has to come from\nthe harness.\n- **1.5.34 (harness)** — now in App Store review: the About Viper card and two\nmatching guides.\n- **1.5.35 (harness)** — the patch that fixes what this test found in 1.5.34:\nthe changes listed above.\n\n|  | Model alone (no harness) | 1.5.34 (harness) | 1.5.35 (harness) | \n|---|---|---|---|\n| Correct | 2% | **41%** | **61%** | \n| Correct or partly correct | 11% | **68%** | **89%** | \n| Invented something | 57% | **24%** | **4%** | \n\nThe model alone cannot know Viper: gpt-oss-20b was released in 2025, and Viper Messenger appeared on the App Store in January 2026 — after the model was made. Its few \"correct\" answers are generic messenger advice that happens to match Viper (tap a photo, then Share → Save Image) or an honest \"I don't know\". On 33 questions about things Viper does not have, it invented an answer 76% of the time; 1.5.34 6%; 1.5.35 3%.\n\n**Why these numbers look low.** On simple questions 1.5.34 already works, as\nthe screenshot shows: the answer may not be exact in every detail, but it\npoints the right way. This test is harder on purpose: varied questions at\nscale — slang, typos, situations, several questions at once — and strict\ngrading: an answer counts as *correct* only when both judges agree that every\nViper detail in it is true.\n\nOn the set we studied the failures on (166 questions; higher, because we fixed the guides where these answers went wrong): 1.5.34 43% correct, 28% invented; 1.5.35 71% correct, 2% invented. We publish it, but count only the fresh set.\n\n*Draft note: 1.5.35 numbers are from the test bench, not yet from the app build.*\n\nFor now this needs a powerful Mac. Every number on this page was measured on one laptop:\n\n- **MacBook Pro, Apple M5 Pro** (15-core CPU: 5 super + 10 performance cores;\n16-core GPU),**24 GB** of memory, macOS 26.5.\n- The model is **gpt-oss-20b** (MLX, MXFP4), about 12 GB, loaded fully into\nmemory.\n- **Speed:** the model alone writes about 57 tokens a second and answers a\nquestion in about 2.6 s (median; 5.6 s for the slowest tenth). The new\nharness asks the model 3–4 times per question (split, find, pick, answer),\nso a full answer takes about 6.4 s (median; 9.9 s for the slowest tenth).\n\nThe model itself takes about 12 GB of memory while it runs (13 GB at peak; 11 GB on disk). With macOS and the app on top, 24 GB is comfortable; a 16 GB Mac is tight. iPhone is not there yet: the largest iPhones have 12 GB in total, less than the model alone needs.\n\n- Pick the right guide page more often — most remaining misses are a missing or wrong page, not a wrong answer.\n- Ship the new harness in the next build, then measure speed on real devices.\n- Publish the test questions and grading rules alongside the results.\n\nAvailable now on Mac with Apple silicon and 24 GB of memory.\n[Viper Messenger on the App Store](https://apps.apple.com/us/app/viper-messenger/id6754181968)", "url": "https://wpnews.pro/news/on-device-ai-new-knowledge-without-fine-tuning", "canonical_source": "https://github.com/viperfoundation/Viper/blob/main/on-device-ai.md", "published_at": "2026-11-18 22:44:39+00:00", "updated_at": "2026-11-18 23:15:47.070535+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-agents", "ai-tools"], "entities": ["Viper", "gpt-oss-20b", "MIT CSAIL", "Tether QVAC", "BotPsy", "Sparks of Weight-Space Programming", "App Store"], "also_reported_by": [], "takeaway": "Supply the model with the matching in-app guides at question time instead of fine-tuning facts into weights: Viper's 20-question test went from 4 correct with gpt-oss-20b alone to 19 correct with guides, while weight training made results worse.", "recommendation": {"advice": "Retrieve the matching guides into context at question time and cite them as sources rather than fine-tuning facts into weights, which Viper found made answers worse; validate on fresh, unseen questions before shipping.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/on-device-ai-new-knowledge-without-fine-tuning", "markdown": "https://wpnews.pro/news/on-device-ai-new-knowledge-without-fine-tuning.md", "text": "https://wpnews.pro/news/on-device-ai-new-knowledge-without-fine-tuning.txt", "jsonld": "https://wpnews.pro/news/on-device-ai-new-knowledge-without-fine-tuning.jsonld"}}