{"slug": "we-taught-a-230m-language-model-to-keep-learning-on-android", "title": "We Taught a 230M Language Model to Keep Learning on Android", "summary": "Engineers at an open-source project have developed Online-SDFT, a prototype that enables a 230M-parameter language model to continually learn on Android devices from delayed user interactions. The system uses hindsight distillation and reliability-conditioned soft targets to update the model locally without requiring explicit labels or rewards.", "body_md": "Small language models can now run directly on phones. But most of them stop learning the moment they ship.\n\nFor personal AI, that feels like a strange stopping point. Some of the most useful signals arrive only after the model acts:\n\nThese interactions contain useful information about the user, but they are delayed, private, and ambiguous. They are not clean labels, and they are not reliable scalar rewards.\n\nTo explore this problem, we built **Online-SDFT**, an open-source prototype that continually fine-tunes a small language model from delayed interactions while keeping the learning loop on the device.\n\nThe prototype uses:\n\nOnce the model has been provisioned, inference, interaction storage, replay, and adapter updates all happen locally.\n\nSuppose the model receives a notification and chooses one of three actions:\n\nSupervised fine-tuning would require a correct action for every notification. But the phone never observes what the ideal action was.\n\nReinforcement learning replaces the correct answer with a reward, but that reward is also difficult to define. Opening a notification does not necessarily mean it arrived at the right time. Ignoring it does not necessarily mean it was unimportant. The user may simply have been busy.\n\nThere is another complication: the model only observes the result of the action it actually took. If it archives a notification, it cannot know what would have happened had it shown the notification immediately.\n\nWhat the phone receives is not a label or reward. It receives **hindsight**.\n\nThe core idea is simple: let the model reconsider its decision after seeing what happened.\n\nAt decision time, the student sees only the current context:\n\n```\nnotification + time + local context\n```\n\nLater, the teacher sees the same context plus the observed outcome:\n\n```\nnotification + time + local context + what the user did afterward\n```\n\nBecause the teacher has more information, it can produce a better-informed distribution over the possible actions. We then distill that soft distribution into the student, which must make future decisions without access to the outcome.\n\nConceptually, the loop looks like this:\n\n```\nfor interaction in stream:\n    context = observe_context()\n\n    action = student.sample(context)\n    execute(action)\n\n    hindsight = wait_for_outcome(action)\n\n    with lora_disabled():\n        target = model(context, hindsight)\n\n    if causally_supported(action, hindsight):\n        replay.add(context, target)\n        update_lora(replay.sample_balanced())\n```\n\nThere is no separate teacher model. The deployed model uses its LoRA adapter to act, while the same frozen base model reviews the completed interaction with the adapter disabled.\n\nThe teacher knows what happened. The student learns to anticipate what the teacher would conclude.\n\nA tempting alternative is to turn every outcome into a hard label.\n\nFor example:\n\n```\ndismissed notification → archive\n```\n\nBut that inference is often too strong. A dismissal might mean the notification was irrelevant, badly timed, already understood from the preview, or simply interrupted by another task.\n\nOnline-SDFT therefore uses reliability-conditioned soft targets. Reliable outcomes can strongly support one action. Ambiguous outcomes redistribute probability only among the actions that remain plausible. Outcomes that reveal nothing useful produce no update.\n\nThis is important because it prevents the training loop from inventing counterfactual outcomes or treating every gesture as an explicit preference.\n\nVanilla self-distillation does not specify how to collect useful interactions from a live, action-dependent stream.\n\nA purely greedy model can lock into an early behavior and only collect evidence that confirms its own choices.\n\nWe add a small amount of exploration while the model is uncertain, then gradually taper it as confidence grows. This gives the system opportunities to observe outcomes from alternative actions without permanently making the serving policy random.\n\nFeedback is sparse and correlated. Several similar outcomes may arrive together, while rare but informative outcomes may occur only once.\n\nInstead of updating from only the newest interaction, we retain a bounded window of recent lessons. Sampling balances feedback categories while favoring newer examples. The newest lesson is always included so that the model remains responsive to changes.\n\nReplay is used only for training. It does not make the serving prompt longer.\n\nWe evaluated six approaches on three paired synthetic notification streams, with 240 decisions per stream.\n\nHere are selected results:\n\n| Method | Preference accuracy ↑ | Cumulative regret ↓ |\n|---|---|---|\n| Frozen base model | 28.2% | 164.7 |\n| RAG | 50.0% | 106.6 |\n| Rejection fine-tuning | 52.8% | 105.3 |\nOnline-SDFT |\n70.3% |\n44.8 |\n\nOnline-SDFT matched the hidden sampled preference on 506 of 720 decisions.\n\nThe comparison with rejection fine-tuning was particularly interesting. Rejection fine-tuning used the same LoRA capacity but required a verified hard target. It accepted only 75 of 311 hindsight-teacher candidates. Online-SDFT could preserve graded information from outcomes that were useful but not strong enough to justify a one-hot label.\n\nReplay also mattered substantially. Removing replay reduced preference accuracy from 70.3% to 39.6% and increased cumulative regret from 44.8 to 134.9.\n\nThese results are preliminary. There are only three synthetic streams, and the selected configuration was tuned on those same streams rather than confirmed on a separate held-out benchmark.\n\nThe repository includes a separate Android project that runs the continual-learning loop on a physical device.\n\nThe frozen 230M-parameter base model makes notification-routing decisions. Once an outcome becomes available, ONNX Runtime Training updates the LoRA adapter. Adapter checkpoints and replay state remain in app-private storage and survive application restarts.\n\nIn one physical-phone test, repeatedly dismissing a notification teaches the model to keep it quiet. Asking for the notification again then changes the learned behavior so the next one is shown.\n\nNo server performs the update.\n\nThis is still an engineering prototype rather than a production notification manager. The current graph uses FP32 and targets high-memory ARM64 devices. Model export and initial provisioning still require a Linux host. We have also not yet completed systematic profiling of latency, peak memory, battery consumption, or thermal throttling.\n\nAndroid notification-listener APIs also operate after a notification has been posted, so the current prototype demonstrates post-time routing rather than guaranteed suppression before an alert appears.\n\nThe broader question we are interested in is:\n\nWhat other applications could learn locally from delayed, action-dependent outcomes?\n\nNotification routing is one example, but the same structure appears in writing suggestions, shortcut recommendations, application ranking, email assistance, and other forms of personal AI.\n\nI would be especially interested in examples where the useful signal already exists on the device, but is too ambiguous to treat as a conventional label or reward.\n\n*Disclosure: I used AI assistance to help draft and edit this article. I reviewed the technical claims against the linked implementation and experiment artifacts before publication.*", "url": "https://wpnews.pro/news/we-taught-a-230m-language-model-to-keep-learning-on-android", "canonical_source": "https://dev.to/ijulin/we-taught-a-230m-language-model-to-keep-learning-on-android-36hl", "published_at": "2026-08-23 21:55:22+00:00", "updated_at": "2026-08-23 22:13:53.464410+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-agents"], "entities": ["Online-SDFT", "Android"], "alternates": {"html": "https://wpnews.pro/news/we-taught-a-230m-language-model-to-keep-learning-on-android", "markdown": "https://wpnews.pro/news/we-taught-a-230m-language-model-to-keep-learning-on-android.md", "text": "https://wpnews.pro/news/we-taught-a-230m-language-model-to-keep-learning-on-android.txt", "jsonld": "https://wpnews.pro/news/we-taught-a-230m-language-model-to-keep-learning-on-android.jsonld"}}