{"slug": "wiring-android-s-mediapipe-llm-inference-api-to-a-streaming-ui", "title": "Wiring Android's MediaPipe LLM Inference API to a Streaming UI", "summary": "A developer published a pattern for streaming on-device LLM token output from MediaPipe's LLM Inference API into a Jetpack Compose Text component without dropped tokens or jank. The approach bridges MediaPipe's non-suspending callback into a Kotlin Flow via a bounded Channel with capacity 64 and BufferOverflow.SUSPEND, then batches emissions in an 80ms time window to cap recompositions at roughly 12 per second, keeping budget devices such as the Helio G88 within their frame budget at 25 tokens/sec.", "body_md": "\n\n```\n---\ntitle: \"Streaming LLM Tokens in Android with Jetpack Compose: Back-Pressure and the Frame Budget\"\npublished: true\ndescription: \"Wire MediaPipe's LLM Inference API to Jetpack Compose using Kotlin Flow back-pressure, correct buffer sizing, and a 16ms recomposition budget that keeps mid-range devices jank-free at 20+ tokens/sec.\"\ntags: kotlin, android, mobile, architecture\ncanonical_url: https://mvpfactory.co/blog/streaming-llm-tokens-android-compose\n---\n```\n\nBy the end of this tutorial you will have a production-ready pattern for streaming on-device LLM token output into a Compose `Text` component — without dropped tokens, without jank on mid-range hardware, and without the recomposition storm that trips up most first implementations.\n\nHere is the gotcha that will save you hours: **MediaPipe does not give you a suspend function. It gives you a callback on a thread you do not own.** The naive bridge will work fine on your Pixel 9 and silently drop frames on your users' Helio G88.\n\nMediaPipe exposes token output like this:\n\n```\ninferenceSession.addQueryChunk(prompt)\ninferenceSession.generateAsync(\n    { partialResult, done ->\n        onToken(partialResult, done)\n    }\n)\n```\n\nTwo constraints that matter:\n\nIgnore these two constraints and you get frame drops that only appear under thermal throttling on mid-range hardware. That is exactly the device your users own.\n\nThe cleanest bridge is a `Channel` with a bounded buffer. Here is the minimal setup to get this working:\n\n```\nfun LlmInferenceSession.tokenFlow(prompt: String): Flow<String> =\n    channelFlow {\n        val buffer = Channel<String>(capacity = 64, onBufferOverflow = BufferOverflow.SUSPEND)\n\n        addQueryChunk(prompt)\n        generateAsync { partial, done ->\n            runBlocking { buffer.send(partial) }\n            if (done) buffer.close()\n        }\n\n        for (token in buffer) send(token)\n    }.flowOn(Dispatchers.Default)\n```\n\nA capacity of 64 covers roughly 2–3 seconds of burst at 25 tok/sec before `SUSPEND` kicks in. `runBlocking` inside the callback is intentional — it blocks the producer when the consumer falls behind, without dropping tokens. The docs do not mention this, but this is the only safe primitive here.\n\nLet me show you a pattern I use in every project. Compose does not batch state writes from `collectAsState` the same way it batches layout passes. Each emission can schedule an independent recomposition.\n\n| Device tier | Vsync budget | Max safe recompositions/sec | Safe token batch window | \n|---|---|---|---|\n| Flagship (Snapdragon 8 Gen 3) | 8.3ms (120Hz) | ~60 | 16ms | \n| Mid-range (Snapdragon 6 Gen 1) | 16ms (60Hz) | ~30 | 33ms | \n| Budget (Helio G88) | 16ms (60Hz) | ~15 | 66ms | \n\nAt 25 tokens/sec on a budget device, one recomposition per token misses frames. The fix is chunked emission with a time window:\n\n```\nfun Flow<String>.chunkByTime(windowMs: Long = 80L): Flow<String> =\n    flow {\n        val accumulator = StringBuilder()\n        var lastEmit = System.currentTimeMillis()\n\n        collect { token ->\n            accumulator.append(token)\n            val now = System.currentTimeMillis()\n            if (now - lastEmit >= windowMs) {\n                emit(accumulator.toString())\n                accumulator.clear()\n                lastEmit = now\n            }\n        }\n        if (accumulator.isNotEmpty()) emit(accumulator.toString())\n    }\n```\n\nAn 80ms window caps recompositions at ~12/sec — safely below the budget-device ceiling.\n\n```\n@Composable\nfun StreamingResponse(session: LlmInferenceSession, prompt: String) {\n    val text by remember(prompt) {\n        session.tokenFlow(prompt)\n            .chunkByTime(windowMs = 80L)\n            .runningReduce { acc, chunk -> acc + chunk }\n    }.collectAsState(initial = \"\")\n\n    Text(text = text, modifier = Modifier.fillMaxWidth())\n}\n```\n\n`runningReduce` accumulates the full response string without holding a mutable `StringBuilder` in composition scope. `remember(prompt)` restarts the flow cleanly on new prompts, avoiding stale emissions.\n\n**Do not use `MutableStateFlow` to bridge the MediaPipe callback.** Only a `Channel` with `BufferOverflow.SUSPEND` prevents dropped tokens from a native producer thread.\n\n**Tune to your minimum supported device, not your desk Pixel.** 80ms is a safe default. Tighten to 33ms only if your floor is mid-range 60Hz and you have profiler data to back it up.\n\n**Before shipping, verify in Android Studio's Compose Layout Inspector:** skipped frames should be zero during active streaming; recomposition count on `Text` should be ≤15/sec on your lowest-supported device.\n\nThe full pattern is three composable pieces: a `Channel`-based callback bridge, a time-windowed chunking operator, and a stateless composable that folds chunks with `runningReduce`. Each piece solves one specific constraint — thread ownership, back-pressure, and frame budget — and they stay decoupled enough to test independently.\n\nProfile on your lowest-supported hardware before shipping. Flagship benchmarks will lie to you.", "url": "https://wpnews.pro/news/wiring-android-s-mediapipe-llm-inference-api-to-a-streaming-ui", "canonical_source": "https://dev.to/software_mvp-factory/wiring-androids-mediapipe-llm-inference-api-to-a-streaming-ui-50eh", "published_at": "2026-10-28 15:07:18+00:00", "updated_at": "2026-10-28 15:11:34.378368+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["MediaPipe", "Jetpack Compose", "Kotlin", "Android", "Google"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/wiring-android-s-mediapipe-llm-inference-api-to-a-streaming-ui", "markdown": "https://wpnews.pro/news/wiring-android-s-mediapipe-llm-inference-api-to-a-streaming-ui.md", "text": "https://wpnews.pro/news/wiring-android-s-mediapipe-llm-inference-api-to-a-streaming-ui.txt", "jsonld": "https://wpnews.pro/news/wiring-android-s-mediapipe-llm-inference-api-to-a-streaming-ui.jsonld"}}