{"slug": "memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone", "title": "Memory, not speed, is the hard part of running an LLM on a phone", "summary": "A developer building Onira, an Android app that generates personalized hypnosis and relaxation scripts on-device with Gemma 4 E2B via LiteRT-LM, found that memory management rather than generation speed is the hard part of running an LLM on a phone. Because the runtime loads the 2.6GB model into the native heap without mmap, the app thrashes on mid-range devices and gets killed by Android's low-memory killer, so the developer unloads the model immediately after the last block is generated and relies on a mediaPlayback foreground service to keep the process alive during narration. The developer also replaced fixed word-count budgeting with duration-based budgeting after discovering that narration speed and pause settings can swing spoken duration by more than 3x.", "body_md": "I ship an Android app (Onira) that generates a personalized hypnosis/relaxation\n\nscript on-device with Gemma 4 E2B, through LiteRT-LM, then narrates it with\n\non-device TTS. Nothing the user types, and nothing the model generates, ever\n\nleaves the phone. This is the part that was actually hard to get right, and it\n\nwas not the part I expected.\n\nA mid-range phone writes 300-500 words in tens of seconds — acceptable for a\n\nrelaxation app where narration masks generation latency. The real problem: the\n\nruntime loads the 2.6GB model into the native heap and does not mmap it.\n\nMeasured on a Pixel 7 (7.6GB RAM), mid-generation:\n\nThe process thrashes, generation crawls, and Android's low-memory killer takes\n\nthe app the instant it leaves the foreground. On a 6GB device it's worse.\n\nThe obvious mistake would be keeping the `Engine`/` Conversation` alive for the\n\nwhole session \"just in case.\" But my output is consumed by TTS over the next\n\n30-40 minutes, and for all of that time nothing needs the model resident. So:\n\nclose it the instant the last block is generated, and let the rest of the\n\nsession narrate with the model unloaded. That window is also, not\n\ncoincidentally, exactly when a user is most likely to background the app to do\n\nsomething else — which is exactly when the low-memory killer was taking it.\n\nThe one deliberate exception: a `mediaPlayback` foreground service keeps the\n\nprocess alive for the whole narration, started right when the user taps\n\n\"Begin Session.\" Without it, backgrounding the app during the first few\n\ngenerating minutes (model still resident) got the whole session killed\n\noutright. Android 12+ also refuses a foreground-service start from the\n\nbackground — exactly the moment that request usually comes — so it has to\n\nstart earlier than you'd want, while the model is still loaded. The accepted\n\ncost: the device may thrash and lmkd kills *other* background apps instead,\n\nuntil the model unloads a few minutes in.\n\nA session is an ordered sequence of blocks — induction, deepening, metaphor,\n\nsuggestions, anchoring, repeat, emergence. Each is generated as its own turn\n\non the *same* `Engine`/` Conversation`, so later blocks stay thematically\n\nconsistent with earlier ones without re-stating prior text in the prompt.\n\nTwo consequences that matter more than the consistency:\n\nThe body is capped at 6 blocks — a repeating cycle of\n\n`[deepening, metaphor, suggestions, anchoring]`. This bound is what keeps any\n\nsingle generation call short regardless of total session length, and it's\n\nalso exactly what the context window is sized against:\n\n``` js\nprivate const val MAX_BODY_BLOCKS = 6\nWhen the target word budget doesn't fill a whole cycle, the scheduler keeps\nthe highest-priority block types and drops the rest — suggestions and\nanchoring survive, metaphor goes first:\n\nfun buildBodyPlan(targetBodyWords: Int): List<SessionBlockType> {\n    val bodyBlocks = ((targetBodyWords + WORDS_PER_BLOCK_TARGET / 2) / WORDS_PER_BLOCK_TARGET)\n        .coerceIn(1, MAX_BODY_BLOCKS)\n    return buildList {\n        repeat(bodyBlocks / CORE_CYCLE.size) { addAll(CORE_CYCLE) }\n        addAll(partialCycle(bodyBlocks % CORE_CYCLE.size))\n    }\n}\n```\n\nRounded to the nearest block, not floored — at the short end of the range,\n\nflooring to whole cycles costs four blocks at once and undershoots the target by far more than rounding up overshoots it.\n\nThe thing I got wrong for a while: sizing by word count\n\nMy first version budgeted a fixed word count for the whole script — \"4200\n\nwords ≈ 30 minutes at 140 wpm.\" It shipped ~90-minute sessions at default\n\nsettings. The bug: narration speed and the user's \"pause between phrases\"\n\nsetting (3-15s, applied at every clause boundary) swing real spoken duration\n\nby more than 3x. A fixed word count cannot hold duration steady when the\n\nseconds-per-word ratio itself varies that much per user.\n\nThe fix budgets from a target duration, converted to words using the user's\n\nown playback settings:\n\n```\nprivate fun secondsPerWord(speechRate: Float, sentencePauseSeconds: Float): Double {\n    val userScale = (speechRate.coerceAtLeast(0.1f) / REFERENCE_USER_RATE).toDouble()\n    val wordsPerMinute = BASE_WORDS_PER_MINUTE * NARRATION_BASE_RATE * userScale\n    val speakingSeconds = 60.0 / wordsPerMinute\n    val pause = sentencePauseSeconds.coerceIn(3f, 15f)\n    val pausePerClause =\n        (SENTENCE_BOUNDARY_SHARE * pause + (1 - SENTENCE_BOUNDARY_SHARE) * NON_SENTENCE_PAUSE_SECONDS) / userScale\n    return speakingSeconds + pausePerClause / AVG_WORDS_PER_CLAUSE\n}\n\nfun wordsForSeconds(targetSeconds: Int, speechRate: Float, sentencePauseSeconds: Float = 10f): Int =\n    (targetSeconds / secondsPerWord(speechRate, sentencePauseSeconds)).toInt().coerceAtLeast(1)\n```\n\nAVG_WORDS_PER_CLAUSE and SENTENCE_BOUNDARY_SHARE aren't guesses — they're\n\nmeasured off the shipped template corpus (~10k words): one clause boundary\n\nevery ~17 words, ~85% of them sentence ends that take the user's pause\n\nsetting; the rest (ellipsis, semicolon) stay on a fixed, shorter pause. Only\n\nsentence boundaries scale with the user's setting, because that's the only\n\npause the setting is actually supposed to control.\n\nThe result: total session length is the thing held constant across every\n\nspeech-rate and pause-length combination a user can pick, instead of word\n\ncount — which is what actually matters for a relaxation session that's\n\nsupposed to run 30-40 minutes, not 90.\n\nThe honest costs of on-device\n\nA 2.6GB first-run model download — by a wide margin the largest drop-off in\n\nthe funnel.\n\nNo server-side moderation layer in front of a model writing\n\npsychologically-framed text for someone who typed in whatever's actually\n\non their mind. Handled with a deliberately high-recall keyword gate that\n\nruns before the model is ever invoked — false positives (blocking a\n\nbenign message) are cheap here; false negatives are not.\n\nHappy to go deeper on the safety gate design or the LiteRT-LM integration\n\nspecifically, if there's interest. The app is [Onira](https://play.google.com/store/apps/details?id=com.oytaub.mindease)\n\non the Play Store if anyone wants to see the output — but the above is the\n\npart I think is worth discussing.", "url": "https://wpnews.pro/news/memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone", "canonical_source": "https://dev.to/cfournel/memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone-53mk", "published_at": "2026-09-22 12:12:35+00:00", "updated_at": "2026-09-22 12:23:12.318864+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "ai-tools", "mlops", "developer-tools"], "entities": ["Onira", "Gemma 4 E2B", "LiteRT-LM", "Android", "Pixel 7", "Google"], "alternates": {"html": "https://wpnews.pro/news/memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone", "markdown": "https://wpnews.pro/news/memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone.md", "text": "https://wpnews.pro/news/memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone.txt", "jsonld": "https://wpnews.pro/news/memory-not-speed-is-the-hard-part-of-running-an-llm-on-a-phone.jsonld"}}