Memory, not speed, is the hard part of running an LLM on a phone

A developer building Onira, an Android app that generates personalized hypnosis and relaxation scripts on-device with Gemma 4 E2B via LiteRT-LM, found that memory management rather than generation speed is the hard part of running an LLM on a phone. Because the runtime loads the 2.6GB model into the native heap without mmap, the app thrashes on mid-range devices and gets killed by Android's low-memory killer, so the developer unloads the model immediately after the last block is generated and relies on a mediaPlayback foreground service to keep the process alive during narration. The developer also replaced fixed word-count budgeting with duration-based budgeting after discovering that narration speed and pause settings can swing spoken duration by more than 3x.

I ship an Android app Onira that generates a personalized hypnosis/relaxation script on-device with Gemma 4 E2B, through LiteRT-LM, then narrates it with on-device TTS. Nothing the user types, and nothing the model generates, ever leaves the phone. This is the part that was actually hard to get right, and it was not the part I expected. A mid-range phone writes 300-500 words in tens of seconds — acceptable for a relaxation app where narration masks generation latency. The real problem: the runtime loads the 2.6GB model into the native heap and does not mmap it. Measured on a Pixel 7 7.6GB RAM , mid-generation: The process thrashes, generation crawls, and Android's low-memory killer takes the app the instant it leaves the foreground. On a 6GB device it's worse. The obvious mistake would be keeping the Engine / Conversation alive for the whole session "just in case." But my output is consumed by TTS over the next 30-40 minutes, and for all of that time nothing needs the model resident. So: close it the instant the last block is generated, and let the rest of the session narrate with the model unloaded. That window is also, not coincidentally, exactly when a user is most likely to background the app to do something else — which is exactly when the low-memory killer was taking it. The one deliberate exception: a mediaPlayback foreground service keeps the process alive for the whole narration, started right when the user taps "Begin Session." Without it, backgrounding the app during the first few generating minutes model still resident got the whole session killed outright. Android 12+ also refuses a foreground-service start from the background — exactly the moment that request usually comes — so it has to start earlier than you'd want, while the model is still loaded. The accepted cost: the device may thrash and lmkd kills other background apps instead, until the model unloads a few minutes in. A session is an ordered sequence of blocks — induction, deepening, metaphor, suggestions, anchoring, repeat, emergence. Each is generated as its own turn on the same Engine / Conversation , so later blocks stay thematically consistent with earlier ones without re-stating prior text in the prompt. Two consequences that matter more than the consistency: The body is capped at 6 blocks — a repeating cycle of deepening, metaphor, suggestions, anchoring . This bound is what keeps any single generation call short regardless of total session length, and it's also exactly what the context window is sized against: js private const val MAX BODY BLOCKS = 6 When the target word budget doesn't fill a whole cycle, the scheduler keeps the highest-priority block types and drops the rest — suggestions and anchoring survive, metaphor goes first: fun buildBodyPlan targetBodyWords: Int : List<SessionBlockType { val bodyBlocks = targetBodyWords + WORDS PER BLOCK TARGET / 2 / WORDS PER BLOCK TARGET .coerceIn 1, MAX BODY BLOCKS return buildList { repeat bodyBlocks / CORE CYCLE.size { addAll CORE CYCLE } addAll partialCycle bodyBlocks % CORE CYCLE.size } } Rounded to the nearest block, not floored — at the short end of the range, flooring to whole cycles costs four blocks at once and undershoots the target by far more than rounding up overshoots it. The thing I got wrong for a while: sizing by word count My first version budgeted a fixed word count for the whole script — "4200 words ≈ 30 minutes at 140 wpm." It shipped ~90-minute sessions at default settings. The bug: narration speed and the user's "pause between phrases" setting 3-15s, applied at every clause boundary swing real spoken duration by more than 3x. A fixed word count cannot hold duration steady when the seconds-per-word ratio itself varies that much per user. The fix budgets from a target duration, converted to words using the user's own playback settings: private fun secondsPerWord speechRate: Float, sentencePauseSeconds: Float : Double { val userScale = speechRate.coerceAtLeast 0.1f / REFERENCE USER RATE .toDouble val wordsPerMinute = BASE WORDS PER MINUTE NARRATION BASE RATE userScale val speakingSeconds = 60.0 / wordsPerMinute val pause = sentencePauseSeconds.coerceIn 3f, 15f val pausePerClause = SENTENCE BOUNDARY SHARE pause + 1 - SENTENCE BOUNDARY SHARE NON SENTENCE PAUSE SECONDS / userScale return speakingSeconds + pausePerClause / AVG WORDS PER CLAUSE } fun wordsForSeconds targetSeconds: Int, speechRate: Float, sentencePauseSeconds: Float = 10f : Int = targetSeconds / secondsPerWord speechRate, sentencePauseSeconds .toInt .coerceAtLeast 1 AVG WORDS PER CLAUSE and SENTENCE BOUNDARY SHARE aren't guesses — they're measured off the shipped template corpus ~10k words : one clause boundary every ~17 words, ~85% of them sentence ends that take the user's pause setting; the rest ellipsis, semicolon stay on a fixed, shorter pause. Only sentence boundaries scale with the user's setting, because that's the only pause the setting is actually supposed to control. The result: total session length is the thing held constant across every speech-rate and pause-length combination a user can pick, instead of word count — which is what actually matters for a relaxation session that's supposed to run 30-40 minutes, not 90. The honest costs of on-device A 2.6GB first-run model download — by a wide margin the largest drop-off in the funnel. No server-side moderation layer in front of a model writing psychologically-framed text for someone who typed in whatever's actually on their mind. Handled with a deliberately high-recall keyword gate that runs before the model is ever invoked — false positives blocking a benign message are cheap here; false negatives are not. Happy to go deeper on the safety gate design or the LiteRT-LM integration specifically, if there's interest. The app is Onira https://play.google.com/store/apps/details?id=com.oytaub.mindease on the Play Store if anyone wants to see the output — but the above is the part I think is worth discussing.