We Taught a 230M Language Model to Keep Learning on Android

Small language models can now run directly on phones. But most of them stop learning the moment they ship.

For personal AI, that feels like a strange stopping point. Some of the most useful signals arrive only after the model acts:

These interactions contain useful information about the user, but they are delayed, private, and ambiguous. They are not clean labels, and they are not reliable scalar rewards.

To explore this problem, we built Online-SDFT, an open-source prototype that continually fine-tunes a small language model from delayed interactions while keeping the learning loop on the device.

The prototype uses:

Once the model has been provisioned, inference, interaction storage, replay, and adapter updates all happen locally.

Suppose the model receives a notification and chooses one of three actions:

Supervised fine-tuning would require a correct action for every notification. But the phone never observes what the ideal action was.

Reinforcement learning replaces the correct answer with a reward, but that reward is also difficult to define. Opening a notification does not necessarily mean it arrived at the right time. Ignoring it does not necessarily mean it was unimportant. The user may simply have been busy.

There is another complication: the model only observes the result of the action it actually took. If it archives a notification, it cannot know what would have happened had it shown the notification immediately.

What the phone receives is not a label or reward. It receives hindsight.

The core idea is simple: let the model reconsider its decision after seeing what happened.

At decision time, the student sees only the current context:

notification + time + local context

Later, the teacher sees the same context plus the observed outcome:

notification + time + local context + what the user did afterward

Because the teacher has more information, it can produce a better-informed distribution over the possible actions. We then distill that soft distribution into the student, which must make future decisions without access to the outcome.

Conceptually, the loop looks like this:

for interaction in stream:
    context = observe_context()

    action = student.sample(context)
    execute(action)

    hindsight = wait_for_outcome(action)

    with lora_disabled():
        target = model(context, hindsight)

    if causally_supported(action, hindsight):
        replay.add(context, target)
        update_lora(replay.sample_balanced())

There is no separate teacher model. The deployed model uses its LoRA adapter to act, while the same frozen base model reviews the completed interaction with the adapter disabled.

The teacher knows what happened. The student learns to anticipate what the teacher would conclude.

A tempting alternative is to turn every outcome into a hard label.

For example:

dismissed notification → archive

But that inference is often too strong. A dismissal might mean the notification was irrelevant, badly timed, already understood from the preview, or simply interrupted by another task.

Online-SDFT therefore uses reliability-conditioned soft targets. Reliable outcomes can strongly support one action. Ambiguous outcomes redistribute probability only among the actions that remain plausible. Outcomes that reveal nothing useful produce no update.

This is important because it prevents the training loop from inventing counterfactual outcomes or treating every gesture as an explicit preference.

Vanilla self-distillation does not specify how to collect useful interactions from a live, action-dependent stream.

A purely greedy model can lock into an early behavior and only collect evidence that confirms its own choices.

We add a small amount of exploration while the model is uncertain, then gradually taper it as confidence grows. This gives the system opportunities to observe outcomes from alternative actions without permanently making the serving policy random.

Feedback is sparse and correlated. Several similar outcomes may arrive together, while rare but informative outcomes may occur only once.

Instead of updating from only the newest interaction, we retain a bounded window of recent lessons. Sampling balances feedback categories while favoring newer examples. The newest lesson is always included so that the model remains responsive to changes.

Replay is used only for training. It does not make the serving prompt longer.

We evaluated six approaches on three paired synthetic notification streams, with 240 decisions per stream.

Here are selected results:

Online-SDFT matched the hidden sampled preference on 506 of 720 decisions.

The comparison with rejection fine-tuning was particularly interesting. Rejection fine-tuning used the same LoRA capacity but required a verified hard target. It accepted only 75 of 311 hindsight-teacher candidates. Online-SDFT could preserve graded information from outcomes that were useful but not strong enough to justify a one-hot label.

Replay also mattered substantially. Removing replay reduced preference accuracy from 70.3% to 39.6% and increased cumulative regret from 44.8 to 134.9.

These results are preliminary. There are only three synthetic streams, and the selected configuration was tuned on those same streams rather than confirmed on a separate held-out benchmark.

The repository includes a separate Android project that runs the continual-learning loop on a physical device.

The frozen 230M-parameter base model makes notification-routing decisions. Once an outcome becomes available, ONNX Runtime Training updates the LoRA adapter. Adapter checkpoints and replay state remain in app-private storage and survive application restarts.

In one physical-phone test, repeatedly dismissing a notification teaches the model to keep it quiet. Asking for the notification again then changes the learned behavior so the next one is shown.

No server performs the update.

This is still an engineering prototype rather than a production notification manager. The current graph uses FP32 and targets high-memory ARM64 devices. Model export and initial provisioning still require a Linux host. We have also not yet completed systematic profiling of latency, peak memory, battery consumption, or thermal throttling.

Android notification-listener APIs also operate after a notification has been posted, so the current prototype demonstrates post-time routing rather than guaranteed suppression before an alert appears.

The broader question we are interested in is:

What other applications could learn locally from delayed, action-dependent outcomes?

Notification routing is one example, but the same structure appears in writing suggestions, shortcut recommendations, application ranking, email assistance, and other forms of personal AI.

I would be especially interested in examples where the useful signal already exists on the device, but is too ambiguous to treat as a conventional label or reward.

Disclosure: I used AI assistance to help draft and edit this article. I reviewed the technical claims against the linked implementation and experiment artifacts before publication.