Wiring Android's MediaPipe Graph API to TensorFlow Lite for Custom On-Device Pipelines: Beyond the Pre-Built Tasks

---
title: "MediaPipe Graph API + TFLite: Custom On-Device Pipelines for Android"
published: true
description: "Go beyond MediaPipe's Task API. Wire CalculatorGraphs, build JNI bridges in Kotlin, and hit sub-16ms streaming inference on mid-range Snapdragon hardware using the GPU zero-copy path."
tags: [android, kotlin, mobile, architecture]
canonical_url: https://blog.mvpfactory.co/mediapipe-graph-api-tflite-custom-on-device-pipelines
---

By the end of this tutorial you will have a working MediaPipe CalculatorGraph

wired to a custom TFLite model via a Kotlin/JNI bridge, with GPU delegate sharing and packet timestamp discipline that keeps frame latency under 16ms on mid-range Snapdragon hardware.

MediaPipe's Task API is a productivity shortcut, not a production ceiling. The moment you need a custom model, non-standard preprocessing, or a true zero-copy GPU path, you are in CalculatorGraph

C++ territory with a JNI bridge — and the documentation gets sparse fast. Let me show you the architecture end-to-end.

MediaPipe's runtime is a directed acyclic graph of Calculator

nodes connected by typed Packet

streams. Each calculator is a C++ class with three methods: GetContract

, Open

, and Process

. The graph config is a protobuf text file.

node {
  calculator: "TfLiteInferenceCalculator"
  input_stream: "TENSORS:preprocessed_tensors"
  output_stream: "TENSORS:output_tensors"
  options: {
    [mediapipe.TfLiteInferenceCalculatorOptions.ext] {
      model_path: "custom_model.tflite"
      delegate { gpu {} }
    }
  }
}

Every frame carries a microsecond timestamp that propagates through the entire pipeline. That one design decision handles branch synchronization — you can fuse optical flow with frame-level inference results without a separate locking mechanism.

Here is the minimal setup to get this working. Keep your Kotlin layer clean — it never sees C++ types directly. It hands off ByteBuffer

or Bitmap

objects and receives structured output through callbacks.

class MediaPipeGraphRunner(modelPath: String) {
    private external fun nativeInit(modelPath: String): Long
    private external fun nativeProcess(handle: Long, frameData: ByteBuffer, timestamp: Long)
    private external fun nativeRelease(handle: Long)

    private val nativeHandle: Long = nativeInit(modelPath)

    fun processFrame(frame: ByteBuffer, timestampUs: Long) =
        nativeProcess(nativeHandle, frame, timestampUs)
}

On the C++ side, CalculatorGraph::AddPacketToInputStream

is your entry point. Timestamp discipline is non-negotiable here — packets arriving out of order stall the graph silently.

The GPU delegate path shares the GL context with your camera preview pipeline. That is where the real performance win lives. These numbers were measured on a Pixel 6a (Snapdragon 778G, Android 14) running MediaPipe 0.10.14, processing 640×480 YUV frames at 30fps — median latency over 500 frames:

The zero-copy path requires your camera pipeline to produce SurfaceTexture

-backed Image

objects and pass the texture ID through the graph via GlTextureFrameCalculator

. The docs do not mention this clearly, but it requires your entire upstream pipeline to run on the same GL thread — design for this upfront, not as a retrofit.

The graph silently drops packets when a downstream calculator falls behind. Enable profiling to catch this early:

profiler_config {
  enable_profiler: true
  trace_enabled: true
}

Retrieve per-calculator latency at runtime via graph.profiler()->GetCalculatorProfiles(...)

. A sustained drop rate above 5% on a 30fps stream means a pipeline bottleneck, not a hardware limit. Verify the profiler method signature against the MediaPipe 0.10.x source before shipping — the API surface has shifted across minor versions.

Timestamp handling in preprocessing calculators. If your preprocessing takes variable time, preserve the original input timestamp explicitly: use cc->Outputs().Tag("OUT").AddPacket(packet.At(cc->InputTimestamp()))

— not the output time. Using output time breaks downstream synchronization quietly and is a nightmare to debug.

** SetServiceObject is not a profiling hook.** It is a typed dependency-injection mechanism for sharing parsed models or GL contexts across calculator nodes. Keep profiling in

CalculatorGraphConfig.profiler_config

and use SetServiceObject

for shared state that multiple nodes need concurrent read access to.The bottleneck is almost never inference. Benchmark at the calculator level, not end-to-end. Preprocessing and format conversion are where cycles disappear on mobile.

Design GPU texture sharing from day one. On mid-range Snapdragon hardware, this is the difference between 60fps and not reaching it. Retrofitting zero-copy GPU sharing after your camera pipeline is built costs significantly more than threading it through upfront.

The MediaPipe graph-level API gives you precise control over preprocessing, model delegation, and frame synchronization that the Task API simply cannot expose. With the JNI bridge keeping Kotlin clean and packet timestamps handling synchronization, you have a maintainable path to sub-16ms on-device streaming inference.

For deeper reference: MediaPipe Graph API docs and the TFLite GPU delegate guide.