How I Built an In-Place Android Screen Translator with Jetpack Compose, ML Kit, and Gemini Vision

Language barriers in mobile apps, games, and foreign media are a constant friction point. Traditional screen translation tools on Android usually force users to take manual screenshots, freeze the screen, jump between applications, or deal with rigid line-by-line OCR card overlays that block the view.

To solve this, I built ALST (AI Live Screen Translation) — an open-source, real-time Android screen translator that translates any screen content in-place and renders the translated text directly over the original screen coordinates without ever leaving your active app.

In this article, I'll walk through the system architecture, how I integrated Gemini 3.6 Flash for single-pass vision translation, and how to prevent memory leaks during continuous high-density screen capturing in Android 14+.

🏗️ High-Level System Architecture

ALST is built following Clean Architecture and MVVM/MVI design patterns. The codebase is strictly partitioned into single-responsibility modules:

core/capture: MediaProjection, VirtualDisplay, & ImageReader pipeline

core/ocr: Google ML Kit Text Recognition v2 engine

core/translator: Dual translation engine (ML Kit On-Device + Gemini Flash)

core/overlay: WindowManager floating views & Compose Canvas rendering

service: ScreenTranslatorService (Foreground) & QSTranslateTileService

data: Jetpack DataStore preferences (BYOK API keys, language options)

ui: Material 3 Glassmorphic Dashboard & Overlay UI

🧠 1. Single-Pass Vision AI vs. Traditional Chaining

In conventional translation tools, the pipeline is split into three heavy steps:

Capture screen frame -> Run local OCR -> Extract text blocks & coordinates.

Send extracted raw text to a Translation API -> Receive translated strings.

Draw text cards over the screen.

While this works, it often loses contextual meaning (e.g., in video games, manga, or slang-heavy social posts).

With ALST, when running in Cloud AI Mode, the app uses Google's Gemini 3.6 Flash Multimodal Vision API:

Raw bitmap frame buffers are sent directly to the model.

In a single pass, Gemini extracts the text, understands local context/idioms, and returns both the translated text and exact bounding box coordinates (Rect).

This dramatically improves translation quality and contextual awareness while reducing pipeline complexity.

📴 2. Dual-Engine Architecture (Cloud + 100% Offline)

Recognizing that users aren't always connected to high-speed internet, ALST implements a flexible Dual-Engine System:

Cloud Engine (Gemini 3.6 Flash): Uses the com.google.ai.client.generativeai SDK with a Bring-Your-Own-Key (BYOK) model for deep contextual translation.

On-Device Engine (Google ML Kit): Combines ML Kit Text Recognition v2 with ML Kit On-Device Translation. It operates 100% offline with sub-50ms latency and zero server dependencies.

Users can toggle seamlessly between these two engines inside the app dashboard, persisted via Jetpack DataStore Preferences.

🔋 3. Zero-Leak Memory Engineering in Android 14+

Capturing raw 1440p / 4K screen frames produces bitmaps that consume over 15MB–20MB of RAM per frame. Doing this continuously using MediaProjection and ImageReader quickly leads to OutOfMemoryError (OOM) crashes if buffer recycling isn't managed strictly.

Here is how ALST handles zero-leak memory management inside ScreenCaptureManager:

suspend fun captureSingleFrame(): Bitmap? = withContext(Dispatchers.Default) {
    val image = imageReader?.acquireLatestImage() ?: return@withContext null
    try {
        val planes = image.planes
        val buffer = planes[0].buffer
        val pixelStride = planes[0].pixelStride
        val rowStride = planes[0].rowStride
        val rowPadding = rowStride - pixelStride * screenWidth

        val bitmap = Bitmap.createBitmap(
            screenWidth + rowPadding / pixelStride,
            screenHeight,
            Bitmap.Config.ARGB_8888
        )
        bitmap.copyPixelsFromBuffer(buffer)
        Bitmap.createBitmap(bitmap, 0, 0, screenWidth, screenHeight)
    } finally {
        // CRITICAL: Always close the image buffer immediately to return it to VirtualDisplay
        image.close()
    }
}

🎯 4. In-Place Overlay & Spatial Coordinate Mapping

To render translated text boxes directly over original text without distorting the layout:

ALST creates a dynamic WindowManager view of type TYPE_APPLICATION_OVERLAY.

Coordinates returned from OCR/Gemini are scaled against physical screen density (DisplayMetrics) and system navigation/notch insets.

Using Jetpack Compose Canvas, dark translucent glassmorphic cards with rounded corners are drawn exactly over original bounding boxes, placing high-contrast translated text right where your eyes expect it.

📱 5. Deep System Integration

ALST integrates directly into Android system controls for maximum convenience:

Draggable Floating Action Button (FAB): A frosted overlay button with magnetic screen-edge snapping.

Quick Settings Tile (TileService): Allows users to trigger screen translation directly from the Android status bar pull-down menu.

Android 14/15 Compliance: Runs via a Foreground Service registered with foregroundServiceType="mediaProjection" and handles runtime permissions securely via a translucent trampoline activity.

🔒 6. Privacy First

ALST operates strictly on a Bring Your Own Key (BYOK) model:

No intermediary proxy servers.

No telemetry or user tracking.

API keys stay stored strictly inside local sandboxed DataStore storage.

Captured screen frames exist purely in volatile RAM during processing and are immediately garbage-collected.

📦 Source Code & Links

ALST is 100% free and open-source under the MIT License.

If you find the architecture interesting or useful, feel free to drop a star ⭐️ on the GitHub repository or open an issue for feature requests and discussions!