{"slug": "how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit", "title": "How I Built an In-Place Android Screen Translator with Jetpack Compose, ML Kit, and Gemini Vision", "summary": "A developer built ALST (AI Live Screen Translation), an open-source Android app that translates screen content in real time and overlays the translated text directly on the original coordinates. The app uses a dual-engine architecture with Google's Gemini 3.6 Flash for single-pass vision translation and ML Kit for offline on-device translation, and it implements zero-leak memory management for continuous screen capture on Android 14+.", "body_md": "Language barriers in mobile apps, games, and foreign media are a constant friction point. Traditional screen translation tools on Android usually force users to take manual screenshots, freeze the screen, jump between applications, or deal with rigid line-by-line OCR card overlays that block the view.\n\nTo solve this, I built ALST (AI Live Screen Translation) — an open-source, real-time Android screen translator that translates any screen content in-place and renders the translated text directly over the original screen coordinates without ever leaving your active app.\n\nIn this article, I'll walk through the system architecture, how I integrated Gemini 3.6 Flash for single-pass vision translation, and how to prevent memory leaks during continuous high-density screen capturing in Android 14+.\n\n🏗️ High-Level System Architecture\n\nALST is built following Clean Architecture and MVVM/MVI design patterns. The codebase is strictly partitioned into single-responsibility modules:\n\ncore/capture: MediaProjection, VirtualDisplay, & ImageReader pipeline\n\ncore/ocr: Google ML Kit Text Recognition v2 engine\n\ncore/translator: Dual translation engine (ML Kit On-Device + Gemini Flash)\n\ncore/overlay: WindowManager floating views & Compose Canvas rendering\n\nservice: ScreenTranslatorService (Foreground) & QSTranslateTileService\n\ndata: Jetpack DataStore preferences (BYOK API keys, language options)\n\nui: Material 3 Glassmorphic Dashboard & Overlay UI\n\n🧠 1. Single-Pass Vision AI vs. Traditional Chaining\n\nIn conventional translation tools, the pipeline is split into three heavy steps:\n\nCapture screen frame -> Run local OCR -> Extract text blocks & coordinates.\n\nSend extracted raw text to a Translation API -> Receive translated strings.\n\nDraw text cards over the screen.\n\nWhile this works, it often loses contextual meaning (e.g., in video games, manga, or slang-heavy social posts).\n\nWith ALST, when running in Cloud AI Mode, the app uses Google's Gemini 3.6 Flash Multimodal Vision API:\n\nRaw bitmap frame buffers are sent directly to the model.\n\nIn a single pass, Gemini extracts the text, understands local context/idioms, and returns both the translated text and exact bounding box coordinates (Rect).\n\nThis dramatically improves translation quality and contextual awareness while reducing pipeline complexity.\n\n📴 2. Dual-Engine Architecture (Cloud + 100% Offline)\n\nRecognizing that users aren't always connected to high-speed internet, ALST implements a flexible Dual-Engine System:\n\nCloud Engine (Gemini 3.6 Flash): Uses the com.google.ai.client.generativeai SDK with a Bring-Your-Own-Key (BYOK) model for deep contextual translation.\n\nOn-Device Engine (Google ML Kit): Combines ML Kit Text Recognition v2 with ML Kit On-Device Translation. It operates 100% offline with sub-50ms latency and zero server dependencies.\n\nUsers can toggle seamlessly between these two engines inside the app dashboard, persisted via Jetpack DataStore Preferences.\n\n🔋 3. Zero-Leak Memory Engineering in Android 14+\n\nCapturing raw 1440p / 4K screen frames produces bitmaps that consume over 15MB–20MB of RAM per frame. Doing this continuously using MediaProjection and ImageReader quickly leads to OutOfMemoryError (OOM) crashes if buffer recycling isn't managed strictly.\n\nHere is how ALST handles zero-leak memory management inside ScreenCaptureManager:\n\n```\nsuspend fun captureSingleFrame(): Bitmap? = withContext(Dispatchers.Default) {\n    val image = imageReader?.acquireLatestImage() ?: return@withContext null\n    try {\n        val planes = image.planes\n        val buffer = planes[0].buffer\n        val pixelStride = planes[0].pixelStride\n        val rowStride = planes[0].rowStride\n        val rowPadding = rowStride - pixelStride * screenWidth\n\n        val bitmap = Bitmap.createBitmap(\n            screenWidth + rowPadding / pixelStride,\n            screenHeight,\n            Bitmap.Config.ARGB_8888\n        )\n        bitmap.copyPixelsFromBuffer(buffer)\n        Bitmap.createBitmap(bitmap, 0, 0, screenWidth, screenHeight)\n    } finally {\n        // CRITICAL: Always close the image buffer immediately to return it to VirtualDisplay\n        image.close()\n    }\n}\n```\n\n🎯 4. In-Place Overlay & Spatial Coordinate Mapping\n\nTo render translated text boxes directly over original text without distorting the layout:\n\nALST creates a dynamic WindowManager view of type TYPE_APPLICATION_OVERLAY.\n\nCoordinates returned from OCR/Gemini are scaled against physical screen density (DisplayMetrics) and system navigation/notch insets.\n\nUsing Jetpack Compose Canvas, dark translucent glassmorphic cards with rounded corners are drawn exactly over original bounding boxes, placing high-contrast translated text right where your eyes expect it.\n\n📱 5. Deep System Integration\n\nALST integrates directly into Android system controls for maximum convenience:\n\nDraggable Floating Action Button (FAB): A frosted overlay button with magnetic screen-edge snapping.\n\nQuick Settings Tile (TileService): Allows users to trigger screen translation directly from the Android status bar pull-down menu.\n\nAndroid 14/15 Compliance: Runs via a Foreground Service registered with foregroundServiceType=\"mediaProjection\" and handles runtime permissions securely via a translucent trampoline activity.\n\n🔒 6. Privacy First\n\nALST operates strictly on a Bring Your Own Key (BYOK) model:\n\nNo intermediary proxy servers.\n\nNo telemetry or user tracking.\n\nAPI keys stay stored strictly inside local sandboxed DataStore storage.\n\nCaptured screen frames exist purely in volatile RAM during processing and are immediately garbage-collected.\n\n📦 Source Code & Links\n\nALST is 100% free and open-source under the **MIT License**.\n\nIf you find the architecture interesting or useful, feel free to drop a star ⭐️ on the GitHub repository or open an issue for feature requests and discussions!", "url": "https://wpnews.pro/news/how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit", "canonical_source": "https://dev.to/navidseyedain/how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit-and-gemini-vision-1667", "published_at": "2026-08-29 13:29:23+00:00", "updated_at": "2026-08-29 13:48:58.164118+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "large-language-models", "developer-tools"], "entities": ["ALST", "Gemini 3.6 Flash", "Google ML Kit", "Jetpack Compose", "Android"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit", "markdown": "https://wpnews.pro/news/how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit.md", "text": "https://wpnews.pro/news/how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit.txt", "jsonld": "https://wpnews.pro/news/how-i-built-an-in-place-android-screen-translator-with-jetpack-compose-ml-kit.jsonld"}}