5 Places a Real-Time Camera Filter Drops Frames

A real-time camera filter has about 33 milliseconds per frame at 30 FPS. When a filter that looked fine in a demo stutters on a real handset, the loss is almost always in one of five places: format conversion, frame backpressure, where inference actually executes, synchronous readback, or sustained thermal load. Here is how I find which one.

I have not benchmarked anyone's SDK here and there are no handset figures below. A number without a named device, an OS build and a thermal state cannot be reused by anyone reading it. What follows is the order I work through when a camera pipeline misses its budget, plus the arithmetic that tells you whether a given stage can possibly fit before you go looking for it in a trace.

What I did do is verify every mechanism against primary documentation and link it: the image analysis contract and frame format from Android's own docs, the delegate behaviour from the TensorFlow Lite repository and its issue tracker, the profiling tools from Google and Apple, and the readback stall from a handset vendor's own optimisation guidance. The arithmetic is reproducible on paper, two divisions and a multiplication, so if any of it is wrong you can check it in about a minute.

The five places below are ordered by how often they turn out to be the answer in my own work. Ranking them by theoretical cost would give a different order.

Two divisions set the ceiling for everything else.

At 30 FPS you have 1000 / 30 = 33.3 ms to receive a frame, convert it, run whatever model you are running, render, and composite. At 60 FPS that halves to 1000 / 60 = 16.7 ms. Each stage you add spends part of that, and the preview is only as smooth as its slowest frame.

Resolution is the other multiplier, and it is worth doing on paper first. A 1080p frame is 1920 x 1080 = 2,073,600 pixels. A 720p frame is 1280 x 720 = 921,600 pixels. That is a factor of 2.25, so any per-pixel stage costs 2.25 times more at 1080p than at 720p before you change a line of code.

I start here because the arithmetic frequently ends the investigation on its own. If a stage has to touch every pixel of a 1080p frame and you have four such stages, the budget was gone before the model loaded.

For a camera filter that a person is looking at while moving their own face, 30 is usually enough and 60 is usually not worth what it costs, because halving the budget to 16.7 ms tends to force a resolution cut that is more visible than the smoothness gain.

The exception is anything the user physically tracks with their hand or head, where the extra samples do register. I would decide this by shipping 30 and instrumenting it before assuming the answer, since the 95th percentile at 30 FPS is a better predictor of perceived quality than the target rate.

Camera hardware on Android hands you YUV_420_888

, a planar YCbCr format with 8 bits per sample and chroma subsampled 4:2:0. Most image processing code, and most tutorial code, wants RGBA instead. So a conversion gets inserted, often before anyone has decided whether it is needed.

The cost is a byte count you can compute without a profiler.

At 4:2:0, a pixel averages 1.5 bytes. A 1080p YUV frame is 2,073,600 x 1.5 = 3,110,400 bytes, roughly 3.11 MB. The same frame as RGBA_8888

at 4 bytes per pixel is 2,073,600 x 4 = 8,294,400 bytes, roughly 8.29 MB. Running that conversion at 30 FPS means writing 8.29 x 30 = 248.7 MB per second, versus 93.3 MB/s if you had left the frame in its native layout.

That traffic is not free on a mid-range memory bus, and it happens before your actual effect has done anything.

What I look at: whether the conversion is needed at all. A shader can sample the Y and UV planes directly and do the color transform on the GPU as part of a pass you are already running. CameraX will also hand ImageAnalysis

an RGBA output natively if you genuinely need RGBA, which at least moves the conversion out of your own loop.

The cause is usually more boring than a slow filter.

If you are using CameraX ImageAnalysis

, the analyzer contract puts the responsibility on you to close each image. Google's documentation is explicit that if images are not closed, they may block further images from being produced, which stalls the preview, or get dropped according to whichever backpressure strategy is configured. Earlier versions closed the image for you; current ones do not, so that multi-frame analysis stays possible. Close the ImageProxy

, not the Media.Image

it wraps, because closing the wrapped image directly breaks the image sharing mechanism inside CameraX.

So a freeze that arrives a second or two in, instead of immediately, usually means a leaked frame reference. Filter speed is a red herring in that case.

The second half of this is the strategy itself, and it is worth knowing what you already have. Non-blocking is the default: STRATEGY_KEEP_ONLY_LATEST

caches only the newest frame in a buffer one deep and overwrites it while you are still working on the previous one. That is almost always what a live filter wants, because a queued frame is a stale frame by the time it renders. So if you never set a strategy, you are already on the non-blocking one, which means a freeze is far more likely to be a frame you did not close than a strategy you did not pick.

The blocking variety has to be chosen deliberately. STRATEGY_BLOCK_PRODUCER

queues images and starts dropping only once the queue is full, and Android's documentation notes that the blocking occurs across the entire camera device scope: if several use cases are bound to that camera, all of them stall while CameraX works through the queue. That is the mechanism that turns a slow analyzer into a frozen preview instead of a dropped frame or two.

// androidx.camera:camera-core:1.3.4
val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .build()

analysis.setAnalyzer(executor) { image: ImageProxy ->
    try {
        frameClock.mark()      // see the timing helper below
        renderEffect(image)    // your pipeline
    } finally {
        image.close()          // ImageProxy.close(), required including on the failure path
    }
}

// When the screen goes away, including when the user backgrounds the app
// mid-session rather than pressing a stop button:
analysis.clearAnalyzer()
cameraProvider.unbind(analysis)

The finally

block matters more than it looks. One exception on one frame, with close()

sitting on the happy path only, produces the symptom above and nothing in the log that points at it.

A GPU delegate that reports a believable inference time is not proof that inference ran on the GPU. TensorFlow Lite's GPU delegate documentation covers which operations the backend supports, and unsupported ops may fall back to CPU depending on version and configuration. Reported behavior is not uniform: developers have filed cases where the expected fallback did not happen and produced a runtime error instead, and cases where it silently did.

Either way, a plausible millisecond figure comes back. It may be describing an execution path other than the one you intend to ship.

That picture comes from the documentation and the issue tracker, so verify it against the version you actually build with before you rely on it.

What I look at: the execution path in a trace, instead of trusting the timing. On Android that means Perfetto, which has shipped in Android system images since Android 9 and has been enabled by default on most devices since Android 11. On iOS the Metal debugger and Metal System Trace in Xcode show the parallel CPU and GPU timeline, which is where a stage that claimed the GPU but ran elsewhere becomes obvious.

I have not tried to name the exact log line for a fallback here. The strings vary across versions and I could not verify a current one while writing this, so I have left it out. The symptom is the reliable signal: GPU utilization that stays flat while inference time stays constant.

If any stage pulls pixels back from the GPU with a blocking read, the pipeline empties.

The mechanism is well understood and worth stating precisely, because it explains why the cost never shows up where you look for it. A blocking readback cannot return until every queued draw command ahead of it has completed, so the CPU waits for the GPU to drain. The call itself is cheap. The stall is what costs you, and it lands on whichever line touches the data.

Samsung's own OpenGL ES guidance, which is a handset maker writing about its own hardware, is one of the clearer public writeups of the pattern. The standard remedy is a pixel buffer object: bind the buffer, issue the read, and it returns immediately while the copy happens asynchronously, then use a fence to check for completion a frame or two later.

The question I ask first: is the readback needed on every frame at all? Histogram, autoexposure and face-region logic often tolerate a fraction of the preview rate, and dropping one of those to every fifth frame is usually a larger win than optimizing the read itself.

A pipeline that holds its budget for 30 seconds tells you very little, because a phone under sustained camera load heats up and the governor responds by lowering clocks. The camera sensor, the GPU and the neural accelerator all work at once, which is close to a worst case for sustained power draw.

What I look at: I run one scene for several minutes, log per-frame intervals throughout, and compare the last minute with the first. If the tail is materially worse than the head, the problem is thermal and no amount of per-stage optimization will move it. Screen brightness, whether the device is charging, and case material all change the result, which is why this one has to run on real hardware in a realistic state, never on a bench.

It is also the step teams skip most often, since it is slow and it needs a person holding a phone.

Here is the timing helper referenced in the snippet above. It keeps the full interval history, since averaging smooths away the exact spikes you are hunting.

class FrameClock(private val capacity: Int = 600) {
    private val intervalsMs = ArrayDeque<Long>(capacity)
    private var lastNs = 0L

    fun mark() {
        val nowNs = System.nanoTime()
        if (lastNs != 0L) {
            if (intervalsMs.size == capacity) intervalsMs.removeFirst()
            intervalsMs.addLast((nowNs - lastNs) / 1_000_000)
        }
        lastNs = nowNs
    }

    /** p is 0.0 to 1.0. Returns null until at least one interval is recorded. */
    fun percentileMs(p: Double): Long? {
        if (intervalsMs.isEmpty()) return null
        val sorted = intervalsMs.sorted()
        val index = Math.round(p * (sorted.size - 1)).toInt()
        return sorted[index]
    }
}

Read percentileMs(0.95)

. A pipeline averaging 28 ms with a 95th percentile of 70 ms is visibly janky, and the average on its own will never say so.

I keep this per device, per resolution. It is the artifact I hand to whoever asks why the filter is slow.

Those figures are my own starting budgets. I have yet to see a device leave all six of them intact, which is the point of filling in the measured column beside them. The gap between the two columns is the article you should be writing about your own app.

In most stuttering camera pipelines I have looked at, the effect itself was cheap. The time went to a frame copied into a format nothing needed, or queued when it should have been dropped, or read back synchronously, or executed somewhere nobody checked, and none of it surfaced in a 30 second run on a cool phone.

Do the two divisions and the byte count before you open a profiler. They cost a minute and they routinely tell you the answer, or at least tell you which of the five places to look in first.