The complete pipeline for shipping ML on Android — from converting a model to running inference at 35 ms on-device, with the code, the pitfalls, and the performance rules.
Three years ago, a health-tech client wanted a screening feature in their Android app: point the camera at a skin image, and get a risk score locally, with no network call. The reason was not convenience — it was privacy. Medical images leaving the device would have triggered compliance, consent, and a GDPR conversation the startup was not ready for. They needed the model to run on the phone.
The first attempt failed because the team treated it like a normal feature. They wrapped a TensorFlow SavedModel in a thin service, ran inference on the main thread, and shipped a model that had not been quantized. The result: app start times ballooned, inference froze the UI, the APK gained 45 MB, and the battery graph looked like a cliff. The feature was turned off two weeks after launch.
I have rebuilt that pipeline four times since, for a dozen clients across different industries. This article is the exact process I now use — the same sequence every time, with the code and the pitfalls that taught me the rules.
The first decision is not technical; it is architectural. You have three options:
My default for anything with private data, or anything that needs a response in under a second, is on-device. The health app went on-device and the risk score came back in under 100 ms, which changed the whole product feel.
You train in TensorFlow or PyTorch, but Android runs TFLite. The conversion step is where most people first stumble. From a trained model:
import tensorflow as tf
model = tf.saved_model.load("path/to/saved_model")
converter = tf.lite.TFLiteConverter.from_saved_model("path/to/saved_model")
tflite_model = converter.convert()
with open("model.tflite", "wb") as f:
f.write(tflite_model)
If you convert this way, you get a float32 model — and it will be big and slow on-device. The important part is quantization. Converting weights from float32 to int8 shrinks the model by 75 percent and can make inference several times faster on devices with the right hardware, at a small accuracy cost:
converter = tf.lite.TFLiteConverter.from_saved_model("path/to/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset # sample inputs
tflite_quant = converter.convert()
The representative_dataset is a small set of typical inputs — the quantizer uses them to calibrate the range of values. Without it, integer-only quantization will not work. This is the most common conversion failure I see: people skip the calibration set and then wonder why the model converts wrong or loses accuracy.
For PyTorch models, export to ONNX first, then convert via onnx2tf or the official converter. Keep the pipeline in CI so the model file is a build artifact, not a manual download — I have seen three versions of "model.tflite" in a repo because someone forgot to replace the file.
In the app module, add the TFLite runtime. Keep it small: the base interpreter is around 1.5 MB; the full tensorflow-lite-support adds utilities and ML Kit glue but also weight. Start minimal:
dependencies {
implementation("org.tensorflow:tensorflow-lite:2.16.1")
// optional, for GPU delegation:
implementation("org.tensorflow:tensorflow-lite-gpu:2.16.1")
}
Put the model in src/main/assets/:
app/src/main/assets/model.tflite
Do not paste the model into res/raw and do not load it from a network path at startup. Assets are packed into the APK and load via the interpreter natively.
This is the rule that saves your UI. the model and running inference both do I/O and compute — doing either on the main thread gives you an ANR. The correct pattern is a small wrapper class that loads once and keeps the interpreter alive for reuse:
class Classifier(context: Context) {
private val interpreter: Interpreter
private val inputBuffer: TensorBuffer
private val outputBuffer: TensorBuffer
init {
val options = Interpreter.Options()
options.numThreads = 4
val modelBytes = context.assets.open("model.tflite").use { it.readBytes() }
val buffer = ByteBuffer.allocateDirect(modelBytes.size)
buffer.put(modelBytes)
buffer.rewind()
interpreter = Interpreter(buffer, options)
val inputShape = intArrayOf(1, 224, 224, 3)
val outputShape = intArrayOf(1, NUM_CLASSES)
inputBuffer = TensorBuffer.createFixedSize(inputShape, DataType.FLOAT32)
outputBuffer = TensorBuffer.createFixedSize(outputShape, DataType.FLOAT32)
}
fun predict(pixels: FloatArray): FloatArray {
inputBuffer.loadArray(pixels)
interpreter.run(inputBuffer.buffer, outputBuffer.buffer)
return outputBuffer.floatArray
}
}
Two details worth calling out:
Interpreter per call is the number one performance bug in ML-on-Android code I review. Model can take hundreds of milliseconds; keep one interpreter alive.Dispatchers.Default or an executor. Never block the main thread with interpreter.run.
suspend fun predictAsync(pixels: FloatArray): FloatArray =
withContext(Dispatchers.Default) { classifier.predict(pixels) }
Models do not accept raw images or raw text. They accept tensors with a specific shape, range, and ordering. The two mistakes that dominate:
Normalization. Most image models expect input in the range [-1, 1] or [0, 1]. A raw Bitmap gives you 0-255 per channel. Forgetting to normalize produces a model that outputs garbage with total confidence — and it is maddening because it works in Python and breaks on Android.
private fun bitmapToFloatArray(bitmap: Bitmap): FloatArray {
val scaled = Bitmap.createScaledBitmap(bitmap, 224, 224, true)
val pixels = IntArray(224 * 224)
scaled.getPixels(pixels, 0, 224, 0, 0, 224, 224)
val input = FloatArray(224 * 224 * 3)
for (i in pixels.indices) {
val color = pixels[i]
val r = ((color shr 16) and 0xFF) / 255.0f
val g = ((color shr 8) and 0xFF) / 255.0f
val b = (color and 0xFF) / 255.0f
input[i * 3] = r * 2f - 1f
input[i * 3 + 1] = g * 2f - 1f
input[i * 3 + 2] = b * 2f - 1f
}
return input
}
Channel order. TFLite models are usually NHWC: batch, height, width, channels (RGB). Android's Bitmap gives you ARGB. If you build the float array in the wrong order, your classifier "works" in tests with synthetic data and fails on real photos. Write the preprocessing once, and test it against a known input from the training script — if the Python notebook and the app produce different numbers for the same pixel, one of them is wrong.
With the interpreter loaded and the input prepared, running it is one call. The output handling depends on the task:
argmax for the label, and read the confidence — and decide a confidence threshold below which the app should say "uncertain" instead of guessing.
val probs = classifier.predict(input)
val maxIdx = probs.indices.maxByOrNull { probs[it] } ?: -1
val confidence = probs[maxIdx]
if (confidence < 0.6f) {
// Show "unable to classify confidently", not a wrong answer.
} else {
val label = labels[maxIdx]
val pct = (confidence * 100).toInt()
}
The confidence threshold is not optional. A model that always answers, even when it has no idea, destroys user trust in exactly the health-app scenario my client built. "I do not know" is a feature.
If your task is a common one — text recognition, barcode scanning, face detection, pose estimation, language identification, translation — do not hand-roll a TFLite integration. Google's ML Kit wraps these models with a clean API and handles preprocessing, threading, and model bundling for you:
dependencies {
implementation("com.google.mlkit:text-recognition:16.0.1")
}
val recognizer = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
recognizer.process(image)
.addOnSuccessListener { visionText ->
val text = visionText.text
// text is plain String; feed it to your pipeline
}
.addOnFailureListener { e ->
// handle failure; never leave the user with a spinner
}
ML Kit's on-device models are ready-made, sized sensibly, and frequently updated. The rule I use: if ML Kit covers your task, use it. Hand-roll TFLite only for custom models or unusual tasks. The first time you integrate a custom model, you will understand why this rule exists.
On-device inference has three performance levers, and teams usually reach for the wrong one first.
GPU and NNAPI delegates. The default CPU interpreter is slow for big models. The GPU delegate is a drop-in accelerator on most devices, and NNAPI delegates to vendor hardware (Qualcomm Hexagon, etc.):
val options = Interpreter.Options()
options.addDelegate(GpuDelegate())
The honest caveat: delegates are hardware-dependent and occasionally produce subtly different outputs than the CPU path. The right pattern is fallback — try GPU, catch the failure, retry with CPU:
class FallbackInterpreter(model: MappedByteBuffer) {
val interpreter: Interpreter
init {
val gpu = Interpreter.Options().apply { addDelegate(GpuDelegate()) }
interpreter = try {
Interpreter(model, gpu)
} catch (e: RuntimeException) {
Interpreter(model, Interpreter.Options()) // CPU fallback
}
}
}
Threading. options.numThreads = 4 helps on multi-core phones but can be slower on single-core devices. Benchmark on both classes of device before shipping the setting as a constant.
Warm-up. The first inference call is always the slowest — model operators get JIT-tuned. Run a dummy inference once at startup (or lazily, on a background thread) and discard the result. My health app's first call was 250 ms; after warm-up, steady-state dropped to around 35 ms. That is the difference between a feature that feels instant and one that feels broken.
Model size discipline. Every MB of model is MB of APK, and 200 MB+ APKs get rejected or see install drops. My default budget is under 15 MB for the model in most apps. Quantize to int8, and if the model is still too big, question the architecture — you do not need a 200 MB transformer on-device for a screening tool.
A model is a dependency, and it needs dependency hygiene like any other.
model_v2.tflite, keep the version in your build metadata, and log which version produced which result.
The health app's on-device model now scores a skin image in roughly 35 ms, offline, on a mid-range phone, with the image never leaving the device. It was not a magic feature. It was conversion, quantization, one careful preprocessing function, a background thread, and a warm-up call. That is the whole discipline.
The lesson from the failed first attempt: ML on Android fails on the same boring reasons every other Android feature fails — main-thread work, unversioned assets, and a missing threshold. Do the boring steps correctly, and the model quietly works, which is exactly what a good feature should do.
*Gulshan Yad