Real-time computer vision is one of the most useful applications of machine learning on mobile devices. Android phones provide cameras, GPU acceleration, and enough processing power to run optimized detection models locally.
In this tutorial, we will build the architecture for a Kotlin application that captures camera frames, runs a YOLO-style object detector, and tracks detected objects across frames.
CameraX
|
ImageAnalysis
|
Frame Conversion
|
YOLO Detector
|
Non-Maximum Suppression
|
Object Tracker
|
UI Overlay
The important part is to keep camera capture, inference, tracking, and rendering separate.
Add the CameraX dependencies compatible with your project:
dependencies {
implementation("androidx.camera:camera-camera2:<version>")
implementation("androidx.camera:camera-lifecycle:<version>")
implementation("androidx.camera:camera-view:<version>")
}
Create an ImageAnalysis
use case:
val analysis = ImageAnalysis.Builder()
.setBackpressureStrategy(
ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST
)
.build()
KEEP_ONLY_LATEST
is important for real-time applications. If inference is slower than the camera, processing every old frame creates a growing queue and increases latency.
Implement an analyzer that receives camera frames:
class ObjectAnalyzer(
private val detector: YoloDetector
) : ImageAnalysis.Analyzer {
override fun analyze(image: ImageProxy) {
try {
detector.detect(image)
} finally {
image.close()
}
}
}
Always close ImageProxy
when processing is complete.
Your detector should convert camera input into the tensor format expected by your model.
Conceptually:
class YoloDetector {
fun detect(image: ImageProxy): List<Detection> {
val input = preprocess(image)
val output = runModel(input)
return postprocess(output)
}
}
A detection normally contains a bounding box, class ID, confidence score, and optionally additional metadata.
data class Detection(
val classId: Int,
val confidence: Float,
val box: RectF
)
Most object detection models expect a fixed input size.
For example:
Camera frame
↓
Rotate
↓
Crop / resize
↓
Normalize
↓
Tensor
Be careful with aspect ratio. Incorrect scaling can make objects appear distorted and reduce detection accuracy.
YOLO-style models can return many candidate boxes. Filter low-confidence detections:
val filtered = detections.filter {
it.confidence >= 0.5f
}
Then apply non-maximum suppression (NMS) to remove overlapping boxes representing the same object.
The exact threshold should be tuned using your target model and dataset.
Detection tells you what is visible in a frame. Tracking attempts to maintain an object's identity across multiple frames.
A simplified tracked object could be:
data class TrackedObject(
val id: Int,
var box: RectF,
var classId: Int,
var confidence: Float
)
A basic tracker can associate the current detection with an existing object using Intersection over Union (IoU).
fun iou(a: RectF, b: RectF): Float {
val left = maxOf(a.left, b.left)
val top = maxOf(a.top, b.top)
val right = minOf(a.right, b.right)
val bottom = minOf(a.bottom, b.bottom)
if (right <= left || bottom <= top) return 0f
val intersection =
(right - left) * (bottom - top)
val union =
a.width() * a.height() +
b.width() * b.height() -
intersection
return intersection / union
}
For more robust tracking, consider established algorithms such as SORT or ByteTrack.
Use a custom Android view to draw detections over the camera preview.
class DetectionOverlay : View(context) {
var detections: List<Detection> = emptyList()
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
for (detection in detections) {
canvas.drawRect(
detection.box,
paint
)
}
}
}
You will need coordinate transformation because model coordinates and preview coordinates may differ.
Real-time inference requires careful optimization.
Useful techniques include:
Do not optimize only for FPS. A smaller model with slightly lower accuracy may provide a much better mobile user experience.
Track at least:
Camera FPS
Inference latency
End-to-end latency
Memory usage
CPU/GPU utilization
For example:
val start = System.nanoTime()
val detections = detector.detect(image)
val elapsedMs =
(System.nanoTime() - start) / 1_000_000
Use real devices for benchmarking because emulator performance does not represent typical mobile hardware.
Camera frames can have different orientations depending on the device.
Pass rotation information into preprocessing:
val rotation = image.imageInfo.rotationDegrees
Failing to handle rotation correctly can produce incorrect bounding boxes or significantly reduce accuracy.
A production application should also handle:
Combining Kotlin, CameraX, YOLO, and object tracking creates a powerful on-device computer vision pipeline.
The key to a production-ready implementation is not simply running a model. You must also control frame backpressure, coordinate transformations, memory allocations, inference latency, and camera lifecycle.
This foundation can be extended into applications such as traffic monitoring, retail analytics, industrial inspection, sports analysis, and smart navigation.
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx