Building an AI Agent for Android, Part 24: AccessibilityService — Reading the Screen and Dispatching Actions

A developer building StareBrain, an on-device AI agent for Android, documented how the agent uses Android's AccessibilityService to read screen content and dispatch UI actions on behalf of users. The service is declared with canRetrieveWindowContent, canPerformGestures and flagReportViewIds, and a ScreenReader class traverses the accessibility tree to capture screen snapshots while a singleton service instance lets the agent's TaskExecutor reach it with null-checked fallbacks. The developer notes the approach requires explicit user permission and is subject to Google Play policies.

This is part of an ongoing series documenting the architecture of StareBrain, an on-device AI agent for Android. Previous parts covered the overlay UI, drag-to-reposition, auto-collapse, voice input, session handoff, natural language planning, plan validation, and on-device inference. This part covers AccessibilityService: how StareBrain reads screen content and dispatches UI actions on behalf of the user. An AI agent that can only open apps isn't very useful. To complete multi-step tasks — filling a form, navigating a settings menu, composing a message — the agent needs to read what's on screen and interact with it programmatically. Android's AccessibilityService is the sanctioned API for this. It was designed for assistive technology, but it's the right tool for any agent that needs to understand and interact with the UI of arbitrary apps. The tradeoff: it requires explicit user permission, and Google Play has specific policies around its use that we'll cover at the end. The service is declared in AndroidManifest.xml with the required permission and metadata: <service android:name=".accessibility.StareBrainAccessibilityService" android:permission="android.permission.BIND ACCESSIBILITY SERVICE" android:exported="true" <intent-filter <action android:name="android.accessibilityservice.AccessibilityService" / </intent-filter <meta-data android:name="android.accessibilityservice" android:resource="@xml/accessibility service config" / </service The configuration file at res/xml/accessibility service config.xml : <accessibility-service xmlns:android="http://schemas.android.com/apk/res/android" android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged|typeViewClicked|typeViewFocused" android:accessibilityFeedbackType="feedbackGeneric" android:accessibilityFlags="flagReportViewIds|flagRequestEnhancedWebAccessibility" android:canPerformGestures="true" android:canRetrieveWindowContent="true" android:notificationTimeout="100" / Key flags: canRetrieveWindowContent : required for reading the UI tree canPerformGestures : required for touch simulation flagReportViewIds : gives us stable resource IDs to target actions class StareBrainAccessibilityService : AccessibilityService { companion object { @Volatile private var instance: StareBrainAccessibilityService? = null fun getInstance : StareBrainAccessibilityService? = instance fun isConnected : Boolean = instance = null } override fun onServiceConnected { instance = this StareBrainApp.onAccessibilityConnected } override fun onAccessibilityEvent event: AccessibilityEvent { // Handled by ScreenReader — see below } override fun onInterrupt { // Service interrupted — reset any pending action state } override fun onDestroy { instance = null super.onDestroy } } The singleton pattern here is intentional. AccessibilityService is managed by the OS — we can't instantiate it ourselves. Other components the agent execution engine, TaskExecutor reach it through getInstance , and we guard every call with a null check and a fallback to StepStatus.Failed . ScreenReader wraps the accessibility tree traversal: class ScreenReader private val service: StareBrainAccessibilityService { fun captureCurrentScreen : ScreenSnapshot { val rootNode = service.rootInActiveWindow ?: return ScreenSnapshot.empty return ScreenSnapshot packageName = rootNode.packageName?.toString ?: "", nodes = collectNodes rootNode , capturedAt = System.currentTimeMillis } private fun collectNodes root: AccessibilityNodeInfo : List<UINode { val nodes = mutableListOf<UINode traverseTree root, nodes, depth = 0 return nodes } private fun traverseTree node: AccessibilityNodeInfo, output: MutableList<UINode , depth: Int { if depth MAX TREE DEPTH return val uiNode = UINode resourceId = node.viewIdResourceName ?: "", text = node.text?.toString ?: "", contentDescription = node.contentDescription?.toString ?: "", className = node.className?.toString ?: "", isClickable = node.isClickable, isEditable = node.isEditable, isFocused = node.isFocused, bounds = node.getBoundsInScreen if uiNode.isInteresting { output.add uiNode } for i in 0 until node.childCount { node.getChild i ?.let { child - traverseTree child, output, depth + 1 child.recycle } } } companion object { private const val MAX TREE DEPTH = 15 } } private fun UINode.isInteresting : Boolean = isClickable || isEditable || text.isNotEmpty || contentDescription.isNotEmpty Two details worth noting: child.recycle : AccessibilityNodeInfo objects must be recycled manually on Android versions below API 33. Missing this causes memory leaks that surface as degraded performance mid-task. MAX TREE DEPTH : Some apps especially WebViews produce extremely deep trees. Without a depth cap, traversal can take hundreds of milliseconds and produce thousands of nodes the model doesn't need. ActionDispatcher translates AgentStep actions into AccessibilityNodeInfo interactions: class ActionDispatcher private val service: StareBrainAccessibilityService { fun dispatch action: StepAction : ActionResult { return when action { is StepAction.Click - clickNode action.targetId is StepAction.TypeText - typeIntoNode action.targetId, action.text is StepAction.Scroll - scroll action.direction is StepAction.PressBack - pressBack is StepAction.Navigate - navigate action.packageName } } private fun clickNode resourceId: String : ActionResult { val node = findNodeById resourceId ?: return ActionResult.Failure "Node not found: $resourceId" val success = node.performAction AccessibilityNodeInfo.ACTION CLICK node.recycle return if success ActionResult.Success else ActionResult.Failure "Click failed on $resourceId" } private fun typeIntoNode resourceId: String, text: String : ActionResult { val node = findNodeById resourceId ?: return ActionResult.Failure "Node not found: $resourceId" node.performAction AccessibilityNodeInfo.ACTION FOCUS val args = Bundle .apply { putString AccessibilityNodeInfo.ACTION ARGUMENT SET TEXT CHARSEQUENCE, text } val success = node.performAction AccessibilityNodeInfo.ACTION SET TEXT, args node.recycle return if success ActionResult.Success else ActionResult.Failure "Type failed on $resourceId" } private fun pressBack : ActionResult { service.performGlobalAction AccessibilityService.GLOBAL ACTION BACK return ActionResult.Success } private fun findNodeById resourceId: String : AccessibilityNodeInfo? { val root = service.rootInActiveWindow ?: return null return root.findAccessibilityNodeInfosByViewId resourceId .firstOrNull } } AccessibilityService requires the user to navigate to Settings → Accessibility → StareBrain and toggle it on manually. Android intentionally makes this non-programmable — no API can enable the service on the user's behalf. We handle this with AccessibilityPermissionGuard : class AccessibilityPermissionGuard private val context: Context { fun isGranted : Boolean = StareBrainAccessibilityService.isConnected fun requestPermission onGranted: - Unit { // Show explanation dialog, then deep-link to settings val intent = Intent Settings.ACTION ACCESSIBILITY SETTINGS .apply { flags = Intent.FLAG ACTIVITY NEW TASK } context.startActivity intent // onGranted is called when isConnected becomes true — polled by TaskExecutor } } We show a one-time explanation before redirecting: what the permission enables, what data it can access, and that it never leaves the device. Users who understand what they're granting convert at a much higher rate than users who get dropped into the settings screen without context. Google Play's policy on AccessibilityService is strict: the service must be used to assist users with disabilities, or for a core use case that cannot be achieved another way. Automation is explicitly listed as a prohibited use in some contexts. StareBrain's position: the service is used to execute tasks the user explicitly requested, with confirmation steps before destructive actions covered in Part 22 . We document this clearly in the Play Store listing and in the permission explanation. Whether this satisfies review is an ongoing conversation — we'll cover the submission process and reviewer feedback in a later part. Part 25 will cover the full execution pipeline end to end: how a user instruction travels from voice or text input through planning, validation, approval, and step-by-step execution using the accessibility layer, with error recovery at each stage. The pieces are all in place. The next part connects them. StareBrain is an on-device AI agent for Android. If you're building with AccessibilityService or have hit the Play Store review wall, I'm in the comments.