Wiring Android's MediaPipe LLM Inference API to a Streaming UI

---
title: "Streaming LLM Tokens in Android with Jetpack Compose: Back-Pressure and the Frame Budget"
published: true
description: "Wire MediaPipe's LLM Inference API to Jetpack Compose using Kotlin Flow back-pressure, correct buffer sizing, and a 16ms recomposition budget that keeps mid-range devices jank-free at 20+ tokens/sec."
tags: kotlin, android, mobile, architecture
canonical_url: https://mvpfactory.co/blog/streaming-llm-tokens-android-compose
---

By the end of this tutorial you will have a production-ready pattern for streaming on-device LLM token output into a Compose Text component — without dropped tokens, without jank on mid-range hardware, and without the recomposition storm that trips up most first implementations.

Here is the gotcha that will save you hours: MediaPipe does not give you a suspend function. It gives you a callback on a thread you do not own. The naive bridge will work fine on your Pixel 9 and silently drop frames on your users' Helio G88.

MediaPipe exposes token output like this:

inferenceSession.addQueryChunk(prompt)
inferenceSession.generateAsync(
    { partialResult, done ->
        onToken(partialResult, done)
    }
)

Two constraints that matter:

Ignore these two constraints and you get frame drops that only appear under thermal throttling on mid-range hardware. That is exactly the device your users own.

The cleanest bridge is a Channel with a bounded buffer. Here is the minimal setup to get this working:

fun LlmInferenceSession.tokenFlow(prompt: String): Flow<String> =
    channelFlow {
        val buffer = Channel<String>(capacity = 64, onBufferOverflow = BufferOverflow.SUSPEND)

        addQueryChunk(prompt)
        generateAsync { partial, done ->
            runBlocking { buffer.send(partial) }
            if (done) buffer.close()
        }

        for (token in buffer) send(token)
    }.flowOn(Dispatchers.Default)

A capacity of 64 covers roughly 2–3 seconds of burst at 25 tok/sec before SUSPEND kicks in. runBlocking inside the callback is intentional — it blocks the producer when the consumer falls behind, without dropping tokens. The docs do not mention this, but this is the only safe primitive here.

Let me show you a pattern I use in every project. Compose does not batch state writes from collectAsState the same way it batches layout passes. Each emission can schedule an independent recomposition.

At 25 tokens/sec on a budget device, one recomposition per token misses frames. The fix is chunked emission with a time window:

fun Flow<String>.chunkByTime(windowMs: Long = 80L): Flow<String> =
    flow {
        val accumulator = StringBuilder()
        var lastEmit = System.currentTimeMillis()

        collect { token ->
            accumulator.append(token)
            val now = System.currentTimeMillis()
            if (now - lastEmit >= windowMs) {
                emit(accumulator.toString())
                accumulator.clear()
                lastEmit = now
            }
        }
        if (accumulator.isNotEmpty()) emit(accumulator.toString())
    }

An 80ms window caps recompositions at ~12/sec — safely below the budget-device ceiling.

@Composable
fun StreamingResponse(session: LlmInferenceSession, prompt: String) {
    val text by remember(prompt) {
        session.tokenFlow(prompt)
            .chunkByTime(windowMs = 80L)
            .runningReduce { acc, chunk -> acc + chunk }
    }.collectAsState(initial = "")

    Text(text = text, modifier = Modifier.fillMaxWidth())
}

runningReduce accumulates the full response string without holding a mutable StringBuilder in composition scope. remember(prompt) restarts the flow cleanly on new prompts, avoiding stale emissions.

Do not use MutableStateFlow to bridge the MediaPipe callback. Only a Channel with BufferOverflow.SUSPEND prevents dropped tokens from a native producer thread.

Tune to your minimum supported device, not your desk Pixel. 80ms is a safe default. Tighten to 33ms only if your floor is mid-range 60Hz and you have profiler data to back it up.

Before shipping, verify in Android Studio's Compose Layout Inspector: skipped frames should be zero during active streaming; recomposition count on Text should be ≤15/sec on your lowest-supported device.

The full pattern is three composable pieces: a Channel-based callback bridge, a time-windowed chunking operator, and a stateless composable that folds chunks with runningReduce. Each piece solves one specific constraint — thread ownership, back-pressure, and frame budget — and they stay decoupled enough to test independently.

Profile on your lowest-supported hardware before shipping. Flagship benchmarks will lie to you.