How I Built an In-Place Android Screen Translator with Jetpack Compose, ML Kit, and Gemini Vision

A developer built ALST (AI Live Screen Translation), an open-source Android app that translates screen content in real time and overlays the translated text directly on the original coordinates. The app uses a dual-engine architecture with Google's Gemini 3.6 Flash for single-pass vision translation and ML Kit for offline on-device translation, and it implements zero-leak memory management for continuous screen capture on Android 14+.

Language barriers in mobile apps, games, and foreign media are a constant friction point. Traditional screen translation tools on Android usually force users to take manual screenshots, freeze the screen, jump between applications, or deal with rigid line-by-line OCR card overlays that block the view. To solve this, I built ALST AI Live Screen Translation — an open-source, real-time Android screen translator that translates any screen content in-place and renders the translated text directly over the original screen coordinates without ever leaving your active app. In this article, I'll walk through the system architecture, how I integrated Gemini 3.6 Flash for single-pass vision translation, and how to prevent memory leaks during continuous high-density screen capturing in Android 14+. 🏗️ High-Level System Architecture ALST is built following Clean Architecture and MVVM/MVI design patterns. The codebase is strictly partitioned into single-responsibility modules: core/capture: MediaProjection, VirtualDisplay, & ImageReader pipeline core/ocr: Google ML Kit Text Recognition v2 engine core/translator: Dual translation engine ML Kit On-Device + Gemini Flash core/overlay: WindowManager floating views & Compose Canvas rendering service: ScreenTranslatorService Foreground & QSTranslateTileService data: Jetpack DataStore preferences BYOK API keys, language options ui: Material 3 Glassmorphic Dashboard & Overlay UI 🧠 1. Single-Pass Vision AI vs. Traditional Chaining In conventional translation tools, the pipeline is split into three heavy steps: Capture screen frame - Run local OCR - Extract text blocks & coordinates. Send extracted raw text to a Translation API - Receive translated strings. Draw text cards over the screen. While this works, it often loses contextual meaning e.g., in video games, manga, or slang-heavy social posts . With ALST, when running in Cloud AI Mode, the app uses Google's Gemini 3.6 Flash Multimodal Vision API: Raw bitmap frame buffers are sent directly to the model. In a single pass, Gemini extracts the text, understands local context/idioms, and returns both the translated text and exact bounding box coordinates Rect . This dramatically improves translation quality and contextual awareness while reducing pipeline complexity. 📴 2. Dual-Engine Architecture Cloud + 100% Offline Recognizing that users aren't always connected to high-speed internet, ALST implements a flexible Dual-Engine System: Cloud Engine Gemini 3.6 Flash : Uses the com.google.ai.client.generativeai SDK with a Bring-Your-Own-Key BYOK model for deep contextual translation. On-Device Engine Google ML Kit : Combines ML Kit Text Recognition v2 with ML Kit On-Device Translation. It operates 100% offline with sub-50ms latency and zero server dependencies. Users can toggle seamlessly between these two engines inside the app dashboard, persisted via Jetpack DataStore Preferences. 🔋 3. Zero-Leak Memory Engineering in Android 14+ Capturing raw 1440p / 4K screen frames produces bitmaps that consume over 15MB–20MB of RAM per frame. Doing this continuously using MediaProjection and ImageReader quickly leads to OutOfMemoryError OOM crashes if buffer recycling isn't managed strictly. Here is how ALST handles zero-leak memory management inside ScreenCaptureManager: suspend fun captureSingleFrame : Bitmap? = withContext Dispatchers.Default { val image = imageReader?.acquireLatestImage ?: return@withContext null try { val planes = image.planes val buffer = planes 0 .buffer val pixelStride = planes 0 .pixelStride val rowStride = planes 0 .rowStride val rowPadding = rowStride - pixelStride screenWidth val bitmap = Bitmap.createBitmap screenWidth + rowPadding / pixelStride, screenHeight, Bitmap.Config.ARGB 8888 bitmap.copyPixelsFromBuffer buffer Bitmap.createBitmap bitmap, 0, 0, screenWidth, screenHeight } finally { // CRITICAL: Always close the image buffer immediately to return it to VirtualDisplay image.close } } 🎯 4. In-Place Overlay & Spatial Coordinate Mapping To render translated text boxes directly over original text without distorting the layout: ALST creates a dynamic WindowManager view of type TYPE APPLICATION OVERLAY. Coordinates returned from OCR/Gemini are scaled against physical screen density DisplayMetrics and system navigation/notch insets. Using Jetpack Compose Canvas, dark translucent glassmorphic cards with rounded corners are drawn exactly over original bounding boxes, placing high-contrast translated text right where your eyes expect it. 📱 5. Deep System Integration ALST integrates directly into Android system controls for maximum convenience: Draggable Floating Action Button FAB : A frosted overlay button with magnetic screen-edge snapping. Quick Settings Tile TileService : Allows users to trigger screen translation directly from the Android status bar pull-down menu. Android 14/15 Compliance: Runs via a Foreground Service registered with foregroundServiceType="mediaProjection" and handles runtime permissions securely via a translucent trampoline activity. 🔒 6. Privacy First ALST operates strictly on a Bring Your Own Key BYOK model: No intermediary proxy servers. No telemetry or user tracking. API keys stay stored strictly inside local sandboxed DataStore storage. Captured screen frames exist purely in volatile RAM during processing and are immediately garbage-collected. 📦 Source Code & Links ALST is 100% free and open-source under the MIT License . If you find the architecture interesting or useful, feel free to drop a star ⭐️ on the GitHub repository or open an issue for feature requests and discussions