Wiring Android's MediaPipe LLM Inference API to a Streaming UI

A developer published a pattern for streaming on-device LLM token output from MediaPipe's LLM Inference API into a Jetpack Compose Text component without dropped tokens or jank. The approach bridges MediaPipe's non-suspending callback into a Kotlin Flow via a bounded Channel with capacity 64 and BufferOverflow.SUSPEND, then batches emissions in an 80ms time window to cap recompositions at roughly 12 per second, keeping budget devices such as the Helio G88 within their frame budget at 25 tokens/sec.

--- title: "Streaming LLM Tokens in Android with Jetpack Compose: Back-Pressure and the Frame Budget" published: true description: "Wire MediaPipe's LLM Inference API to Jetpack Compose using Kotlin Flow back-pressure, correct buffer sizing, and a 16ms recomposition budget that keeps mid-range devices jank-free at 20+ tokens/sec." tags: kotlin, android, mobile, architecture canonical url: https://mvpfactory.co/blog/streaming-llm-tokens-android-compose --- By the end of this tutorial you will have a production-ready pattern for streaming on-device LLM token output into a Compose Text component — without dropped tokens, without jank on mid-range hardware, and without the recomposition storm that trips up most first implementations. Here is the gotcha that will save you hours: MediaPipe does not give you a suspend function. It gives you a callback on a thread you do not own. The naive bridge will work fine on your Pixel 9 and silently drop frames on your users' Helio G88. MediaPipe exposes token output like this: inferenceSession.addQueryChunk prompt inferenceSession.generateAsync { partialResult, done - onToken partialResult, done } Two constraints that matter: Ignore these two constraints and you get frame drops that only appear under thermal throttling on mid-range hardware. That is exactly the device your users own. The cleanest bridge is a Channel with a bounded buffer. Here is the minimal setup to get this working: fun LlmInferenceSession.tokenFlow prompt: String : Flow<String = channelFlow { val buffer = Channel<String capacity = 64, onBufferOverflow = BufferOverflow.SUSPEND addQueryChunk prompt generateAsync { partial, done - runBlocking { buffer.send partial } if done buffer.close } for token in buffer send token }.flowOn Dispatchers.Default A capacity of 64 covers roughly 2–3 seconds of burst at 25 tok/sec before SUSPEND kicks in. runBlocking inside the callback is intentional — it blocks the producer when the consumer falls behind, without dropping tokens. The docs do not mention this, but this is the only safe primitive here. Let me show you a pattern I use in every project. Compose does not batch state writes from collectAsState the same way it batches layout passes. Each emission can schedule an independent recomposition. | Device tier | Vsync budget | Max safe recompositions/sec | Safe token batch window | |---|---|---|---| | Flagship Snapdragon 8 Gen 3 | 8.3ms 120Hz | ~60 | 16ms | | Mid-range Snapdragon 6 Gen 1 | 16ms 60Hz | ~30 | 33ms | | Budget Helio G88 | 16ms 60Hz | ~15 | 66ms | At 25 tokens/sec on a budget device, one recomposition per token misses frames. The fix is chunked emission with a time window: fun Flow<String .chunkByTime windowMs: Long = 80L : Flow<String = flow { val accumulator = StringBuilder var lastEmit = System.currentTimeMillis collect { token - accumulator.append token val now = System.currentTimeMillis if now - lastEmit = windowMs { emit accumulator.toString accumulator.clear lastEmit = now } } if accumulator.isNotEmpty emit accumulator.toString } An 80ms window caps recompositions at ~12/sec — safely below the budget-device ceiling. @Composable fun StreamingResponse session: LlmInferenceSession, prompt: String { val text by remember prompt { session.tokenFlow prompt .chunkByTime windowMs = 80L .runningReduce { acc, chunk - acc + chunk } }.collectAsState initial = "" Text text = text, modifier = Modifier.fillMaxWidth } runningReduce accumulates the full response string without holding a mutable StringBuilder in composition scope. remember prompt restarts the flow cleanly on new prompts, avoiding stale emissions. Do not use MutableStateFlow to bridge the MediaPipe callback. Only a Channel with BufferOverflow.SUSPEND prevents dropped tokens from a native producer thread. Tune to your minimum supported device, not your desk Pixel. 80ms is a safe default. Tighten to 33ms only if your floor is mid-range 60Hz and you have profiler data to back it up. Before shipping, verify in Android Studio's Compose Layout Inspector: skipped frames should be zero during active streaming; recomposition count on Text should be ≤15/sec on your lowest-supported device. The full pattern is three composable pieces: a Channel -based callback bridge, a time-windowed chunking operator, and a stateless composable that folds chunks with runningReduce . Each piece solves one specific constraint — thread ownership, back-pressure, and frame budget — and they stay decoupled enough to test independently. Profile on your lowest-supported hardware before shipping. Flagship benchmarks will lie to you.