{"slug": "building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen", "title": "Building an AI Agent for Android, Part 24: AccessibilityService — Reading the Screen and Dispatching Actions", "summary": "A developer building StareBrain, an on-device AI agent for Android, documented how the agent uses Android's AccessibilityService to read screen content and dispatch UI actions on behalf of users. The service is declared with canRetrieveWindowContent, canPerformGestures and flagReportViewIds, and a ScreenReader class traverses the accessibility tree to capture screen snapshots while a singleton service instance lets the agent's TaskExecutor reach it with null-checked fallbacks. The developer notes the approach requires explicit user permission and is subject to Google Play policies.", "body_md": "*This is part of an ongoing series documenting the architecture of StareBrain, an on-device AI agent for Android. Previous parts covered the overlay UI, drag-to-reposition, auto-collapse, voice input, session handoff, natural language planning, plan validation, and on-device inference. This part covers AccessibilityService: how StareBrain reads screen content and dispatches UI actions on behalf of the user.*\n\nAn AI agent that can only open apps isn't very useful. To complete multi-step tasks — filling a form, navigating a settings menu, composing a message — the agent needs to read what's on screen and interact with it programmatically.\n\nAndroid's `AccessibilityService` is the sanctioned API for this. It was designed for assistive technology, but it's the right tool for any agent that needs to understand and interact with the UI of arbitrary apps. The tradeoff: it requires explicit user permission, and Google Play has specific policies around its use that we'll cover at the end.\n\nThe service is declared in `AndroidManifest.xml` with the required permission and metadata:\n\n```\n<service\n    android:name=\".accessibility.StareBrainAccessibilityService\"\n    android:permission=\"android.permission.BIND_ACCESSIBILITY_SERVICE\"\n    android:exported=\"true\">\n    <intent-filter>\n        <action android:name=\"android.accessibilityservice.AccessibilityService\" />\n    </intent-filter>\n    <meta-data\n        android:name=\"android.accessibilityservice\"\n        android:resource=\"@xml/accessibility_service_config\" />\n</service>\n```\n\nThe configuration file at `res/xml/accessibility_service_config.xml`:\n\n```\n<accessibility-service xmlns:android=\"http://schemas.android.com/apk/res/android\"\n    android:accessibilityEventTypes=\"typeWindowStateChanged|typeWindowContentChanged|typeViewClicked|typeViewFocused\"\n    android:accessibilityFeedbackType=\"feedbackGeneric\"\n    android:accessibilityFlags=\"flagReportViewIds|flagRequestEnhancedWebAccessibility\"\n    android:canPerformGestures=\"true\"\n    android:canRetrieveWindowContent=\"true\"\n    android:notificationTimeout=\"100\" />\n```\n\nKey flags:\n\n`canRetrieveWindowContent`: required for reading the UI tree` canPerformGestures`: required for touch simulation` flagReportViewIds`: gives us stable resource IDs to target actions\n\n```\nclass StareBrainAccessibilityService : AccessibilityService() {\n\n    companion object {\n        @Volatile\n        private var instance: StareBrainAccessibilityService? = null\n\n        fun getInstance(): StareBrainAccessibilityService? = instance\n        fun isConnected(): Boolean = instance != null\n    }\n\n    override fun onServiceConnected() {\n        instance = this\n        StareBrainApp.onAccessibilityConnected()\n    }\n\n    override fun onAccessibilityEvent(event: AccessibilityEvent) {\n        // Handled by ScreenReader — see below\n    }\n\n    override fun onInterrupt() {\n        // Service interrupted — reset any pending action state\n    }\n\n    override fun onDestroy() {\n        instance = null\n        super.onDestroy()\n    }\n}\n```\n\nThe singleton pattern here is intentional. `AccessibilityService` is managed by the OS — we can't instantiate it ourselves. Other components (the agent execution engine, `TaskExecutor`) reach it through `getInstance()`, and we guard every call with a null check and a fallback to `StepStatus.Failed`.\n\n`ScreenReader` wraps the accessibility tree traversal:\n\n```\nclass ScreenReader(private val service: StareBrainAccessibilityService) {\n\n    fun captureCurrentScreen(): ScreenSnapshot {\n        val rootNode = service.rootInActiveWindow\n            ?: return ScreenSnapshot.empty()\n\n        return ScreenSnapshot(\n            packageName = rootNode.packageName?.toString() ?: \"\",\n            nodes = collectNodes(rootNode),\n            capturedAt = System.currentTimeMillis()\n        )\n    }\n\n    private fun collectNodes(root: AccessibilityNodeInfo): List<UINode> {\n        val nodes = mutableListOf<UINode>()\n        traverseTree(root, nodes, depth = 0)\n        return nodes\n    }\n\n    private fun traverseTree(\n        node: AccessibilityNodeInfo,\n        output: MutableList<UINode>,\n        depth: Int\n    ) {\n        if (depth > MAX_TREE_DEPTH) return\n\n        val uiNode = UINode(\n            resourceId = node.viewIdResourceName ?: \"\",\n            text = node.text?.toString() ?: \"\",\n            contentDescription = node.contentDescription?.toString() ?: \"\",\n            className = node.className?.toString() ?: \"\",\n            isClickable = node.isClickable,\n            isEditable = node.isEditable,\n            isFocused = node.isFocused,\n            bounds = node.getBoundsInScreen()\n        )\n\n        if (uiNode.isInteresting()) {\n            output.add(uiNode)\n        }\n\n        for (i in 0 until node.childCount) {\n            node.getChild(i)?.let { child ->\n                traverseTree(child, output, depth + 1)\n                child.recycle()\n            }\n        }\n    }\n\n    companion object {\n        private const val MAX_TREE_DEPTH = 15\n    }\n}\n\nprivate fun UINode.isInteresting(): Boolean =\n    isClickable || isEditable || text.isNotEmpty() || contentDescription.isNotEmpty()\n```\n\nTwo details worth noting:\n\n**`child.recycle()`**: `AccessibilityNodeInfo` objects must be recycled manually on Android versions below API 33. Missing this causes memory leaks that surface as degraded performance mid-task.\n\n**`MAX_TREE_DEPTH`**: Some apps (especially WebViews) produce extremely deep trees. Without a depth cap, traversal can take hundreds of milliseconds and produce thousands of nodes the model doesn't need.\n\n`ActionDispatcher` translates `AgentStep` actions into `AccessibilityNodeInfo` interactions:\n\n```\nclass ActionDispatcher(private val service: StareBrainAccessibilityService) {\n\n    fun dispatch(action: StepAction): ActionResult {\n        return when (action) {\n            is StepAction.Click -> clickNode(action.targetId)\n            is StepAction.TypeText -> typeIntoNode(action.targetId, action.text)\n            is StepAction.Scroll -> scroll(action.direction)\n            is StepAction.PressBack -> pressBack()\n            is StepAction.Navigate -> navigate(action.packageName)\n        }\n    }\n\n    private fun clickNode(resourceId: String): ActionResult {\n        val node = findNodeById(resourceId)\n            ?: return ActionResult.Failure(\"Node not found: $resourceId\")\n\n        val success = node.performAction(AccessibilityNodeInfo.ACTION_CLICK)\n        node.recycle()\n\n        return if (success) ActionResult.Success else ActionResult.Failure(\"Click failed on $resourceId\")\n    }\n\n    private fun typeIntoNode(resourceId: String, text: String): ActionResult {\n        val node = findNodeById(resourceId)\n            ?: return ActionResult.Failure(\"Node not found: $resourceId\")\n\n        node.performAction(AccessibilityNodeInfo.ACTION_FOCUS)\n\n        val args = Bundle().apply {\n            putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, text)\n        }\n        val success = node.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)\n        node.recycle()\n\n        return if (success) ActionResult.Success else ActionResult.Failure(\"Type failed on $resourceId\")\n    }\n\n    private fun pressBack(): ActionResult {\n        service.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK)\n        return ActionResult.Success\n    }\n\n    private fun findNodeById(resourceId: String): AccessibilityNodeInfo? {\n        val root = service.rootInActiveWindow ?: return null\n        return root.findAccessibilityNodeInfosByViewId(resourceId).firstOrNull()\n    }\n}\n```\n\n`AccessibilityService` requires the user to navigate to **Settings → Accessibility → StareBrain** and toggle it on manually. Android intentionally makes this non-programmable — no API can enable the service on the user's behalf.\n\nWe handle this with `AccessibilityPermissionGuard`:\n\n```\nclass AccessibilityPermissionGuard(private val context: Context) {\n\n    fun isGranted(): Boolean =\n        StareBrainAccessibilityService.isConnected()\n\n    fun requestPermission(onGranted: () -> Unit) {\n        // Show explanation dialog, then deep-link to settings\n        val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS).apply {\n            flags = Intent.FLAG_ACTIVITY_NEW_TASK\n        }\n        context.startActivity(intent)\n        // onGranted is called when isConnected() becomes true — polled by TaskExecutor\n    }\n}\n```\n\nWe show a one-time explanation before redirecting: what the permission enables, what data it can access, and that it never leaves the device. Users who understand what they're granting convert at a much higher rate than users who get dropped into the settings screen without context.\n\nGoogle Play's policy on `AccessibilityService` is strict: the service must be used to assist users with disabilities, or for a core use case that cannot be achieved another way. Automation is explicitly listed as a prohibited use in some contexts.\n\nStareBrain's position: the service is used to execute tasks the user explicitly requested, with confirmation steps before destructive actions (covered in Part 22). We document this clearly in the Play Store listing and in the permission explanation. Whether this satisfies review is an ongoing conversation — we'll cover the submission process and reviewer feedback in a later part.\n\nPart 25 will cover the full execution pipeline end to end: how a user instruction travels from voice or text input through planning, validation, approval, and step-by-step execution using the accessibility layer, with error recovery at each stage.\n\nThe pieces are all in place. The next part connects them.\n\n*StareBrain is an on-device AI agent for Android. If you're building with AccessibilityService or have hit the Play Store review wall, I'm in the comments.*", "url": "https://wpnews.pro/news/building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen", "canonical_source": "https://dev.to/starebrain/building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen-and-h9h", "published_at": "2026-11-14 05:13:13+00:00", "updated_at": "2026-11-14 05:23:37.257339+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["StareBrain", "Android", "Google Play", "AccessibilityService"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen", "markdown": "https://wpnews.pro/news/building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen.md", "text": "https://wpnews.pro/news/building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen.txt", "jsonld": "https://wpnews.pro/news/building-an-ai-agent-for-android-part-24-accessibilityservice-reading-the-screen.jsonld"}}