Member-only story
When optimizing on-device AI, we often obsess over inference time — the raw milliseconds a model spends processing tokens. I spent days tuning prompts and finding fast quantized models, yet my input processing times were still screaming on low-end devices.
What I uncovered during a late-night profiling session completely shattered my assumptions and common knowledge on the internet.
It turns out, raw inference time wasn’t the bottleneck. It wasn’t even close. In this post, I’ll share the unexpected performance trap that derailed my app, the dead ends I hit, and the unconventional workarounds that finally saved it.
If you have been following me or have read my earlier post on this topic, where I discussed many tricks to get the most out of on-device AI SLMs, you know that I believe we can only achieve true democratization of AI only consumers when consumers can run AI models on their personal devices. This also gives us the much needed data privacy. To that end, for everything I build, the first attempt is to squeeze things on edge device. But it’s a path paved with thorns. Every single step is a struggle and heartache.
Limited Choices
If you want to run models on edge device, let’s say on Android, you are cornered with limited choices. Android (or any other edge device OS) can run only certain formats efficiently like .litertlm, .onnx etc. Compound that with the fact that…