Real-Time Object Detection and Tracking with Kotlin and YOLO on Android

A developer detailed the architecture for building a real-time object detection and tracking app on Android using Kotlin and YOLO-style models. The approach separates camera capture, inference, tracking, and rendering, and includes techniques like non-maximum suppression and IoU-based tracking to maintain object identities across frames.

Real-time computer vision is one of the most useful applications of machine learning on mobile devices. Android phones provide cameras, GPU acceleration, and enough processing power to run optimized detection models locally. In this tutorial, we will build the architecture for a Kotlin application that captures camera frames, runs a YOLO-style object detector, and tracks detected objects across frames. CameraX | ImageAnalysis | Frame Conversion | YOLO Detector | Non-Maximum Suppression | Object Tracker | UI Overlay The important part is to keep camera capture, inference, tracking, and rendering separate. Add the CameraX dependencies compatible with your project: dependencies { implementation "androidx.camera:camera-camera2:<version " implementation "androidx.camera:camera-lifecycle:<version " implementation "androidx.camera:camera-view:<version " } Create an ImageAnalysis use case: val analysis = ImageAnalysis.Builder .setBackpressureStrategy ImageAnalysis.STRATEGY KEEP ONLY LATEST .build KEEP ONLY LATEST is important for real-time applications. If inference is slower than the camera, processing every old frame creates a growing queue and increases latency. Implement an analyzer that receives camera frames: class ObjectAnalyzer private val detector: YoloDetector : ImageAnalysis.Analyzer { override fun analyze image: ImageProxy { try { detector.detect image } finally { image.close } } } Always close ImageProxy when processing is complete. Your detector should convert camera input into the tensor format expected by your model. Conceptually: class YoloDetector { fun detect image: ImageProxy : List<Detection { val input = preprocess image val output = runModel input return postprocess output } } A detection normally contains a bounding box, class ID, confidence score, and optionally additional metadata. data class Detection val classId: Int, val confidence: Float, val box: RectF Most object detection models expect a fixed input size. For example: Camera frame ↓ Rotate ↓ Crop / resize ↓ Normalize ↓ Tensor Be careful with aspect ratio. Incorrect scaling can make objects appear distorted and reduce detection accuracy. YOLO-style models can return many candidate boxes. Filter low-confidence detections: val filtered = detections.filter { it.confidence = 0.5f } Then apply non-maximum suppression NMS to remove overlapping boxes representing the same object. The exact threshold should be tuned using your target model and dataset. Detection tells you what is visible in a frame. Tracking attempts to maintain an object's identity across multiple frames. A simplified tracked object could be: js data class TrackedObject val id: Int, var box: RectF, var classId: Int, var confidence: Float A basic tracker can associate the current detection with an existing object using Intersection over Union IoU . fun iou a: RectF, b: RectF : Float { val left = maxOf a.left, b.left val top = maxOf a.top, b.top val right = minOf a.right, b.right val bottom = minOf a.bottom, b.bottom if right <= left || bottom <= top return 0f val intersection = right - left bottom - top val union = a.width a.height + b.width b.height - intersection return intersection / union } For more robust tracking, consider established algorithms such as SORT or ByteTrack. Use a custom Android view to draw detections over the camera preview. js class DetectionOverlay : View context { var detections: List<Detection = emptyList override fun onDraw canvas: Canvas { super.onDraw canvas for detection in detections { canvas.drawRect detection.box, paint } } } You will need coordinate transformation because model coordinates and preview coordinates may differ. Real-time inference requires careful optimization. Useful techniques include: Do not optimize only for FPS. A smaller model with slightly lower accuracy may provide a much better mobile user experience. Track at least: Camera FPS Inference latency End-to-end latency Memory usage CPU/GPU utilization For example: val start = System.nanoTime val detections = detector.detect image val elapsedMs = System.nanoTime - start / 1 000 000 Use real devices for benchmarking because emulator performance does not represent typical mobile hardware. Camera frames can have different orientations depending on the device. Pass rotation information into preprocessing: val rotation = image.imageInfo.rotationDegrees Failing to handle rotation correctly can produce incorrect bounding boxes or significantly reduce accuracy. A production application should also handle: Combining Kotlin, CameraX, YOLO, and object tracking creates a powerful on-device computer vision pipeline. The key to a production-ready implementation is not simply running a model. You must also control frame backpressure, coordinate transformations, memory allocations, inference latency, and camera lifecycle. This foundation can be extended into applications such as traffic monitoring, retail analytics, industrial inspection, sports analysis, and smart navigation. SDK Flutter: https://github.com/v-modal/vmodal sdk flutter https://github.com/v-modal/vmodal sdk flutter SDK Android: https://github.com/v-modal/vmodal sdk android https://github.com/v-modal/vmodal sdk android Discord: https://discord.gg/K72z28KUx https://discord.gg/K72z28KUx