{"slug": "i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing", "title": "I Thought My On-Device AI Model Was Slow. I Was Profiling the Wrong Thing.", "summary": "A developer profiling an on-device AI model discovered that raw inference time was not the bottleneck, contrary to common assumptions, and that input processing times remained high on low-end devices. The developer, Asutosh Nayak, shared the findings in a Medium post, noting that optimizing prompts and quantized models did not solve the issue, and that the real performance trap required unconventional workarounds. The post highlights the challenges of running AI models on edge devices like Android, where only certain formats such as .litertlm and .onnx are efficient.", "body_md": "Member-only story\n\n# I Thought My On-Device AI Model Was Slow. I Was Profiling the Wrong Thing.\n\nWhen optimizing on-device AI, we often obsess over **inference time** — the raw milliseconds a model spends processing tokens. I spent days tuning prompts and finding fast quantized models, yet my input processing times were still screaming on low-end devices.\n\nWhat I uncovered during a late-night profiling session completely shattered my assumptions and common knowledge on the internet.\n\nIt turns out, raw inference time wasn’t the bottleneck. It wasn’t even close. In this post, I’ll share the unexpected performance trap that derailed my app, the dead ends I hit, and the unconventional workarounds that finally saved it.\n\nIf you have been following me or have read my [earlier post on this topic](https://medium.com/towards-artificial-intelligence/how-to-squeeze-the-most-out-of-on-device-small-language-models-5f17bdba095d?sharedUserId=asutoshnayak), where I discussed many tricks to get the most out of on-device AI SLMs, you know that I believe we can only achieve true democratization of AI only consumers when consumers can run AI models on their personal devices. This also gives us the much needed data privacy. To that end, for everything I build, the first attempt is to squeeze things on edge device. But it’s a path paved with thorns. Every single step is a struggle and heartache.\n\n### Limited Choices\n\nIf you want to run models on edge device, let’s say on Android, you are cornered with limited choices. Android (or any other edge device OS) can run only certain formats efficiently like .litertlm, .onnx etc. Compound that with the fact that…", "url": "https://wpnews.pro/news/i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing", "canonical_source": "https://pub.towardsai.net/i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing-a357377d2801?source=rss----98111c9905da---4", "published_at": "2026-08-03 13:01:02+00:00", "updated_at": "2026-08-03 13:22:08.107044+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-products", "ai-infrastructure"], "entities": ["Asutosh Nayak", "Medium", "Android"], "alternates": {"html": "https://wpnews.pro/news/i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing", "markdown": "https://wpnews.pro/news/i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing.md", "text": "https://wpnews.pro/news/i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing.txt", "jsonld": "https://wpnews.pro/news/i-thought-my-on-device-ai-model-was-slow-i-was-profiling-the-wrong-thing.jsonld"}}