Building an AI Agent for Android, Part 24: AccessibilityService — Reading the Screen and Dispatching Actions

This is part of an ongoing series documenting the architecture of StareBrain, an on-device AI agent for Android. Previous parts covered the overlay UI, drag-to-reposition, auto-collapse, voice input, session handoff, natural language planning, plan validation, and on-device inference. This part covers AccessibilityService: how StareBrain reads screen content and dispatches UI actions on behalf of the user.

An AI agent that can only open apps isn't very useful. To complete multi-step tasks — filling a form, navigating a settings menu, composing a message — the agent needs to read what's on screen and interact with it programmatically.

Android's AccessibilityService is the sanctioned API for this. It was designed for assistive technology, but it's the right tool for any agent that needs to understand and interact with the UI of arbitrary apps. The tradeoff: it requires explicit user permission, and Google Play has specific policies around its use that we'll cover at the end.

The service is declared in AndroidManifest.xml with the required permission and metadata:

<service
    android:name=".accessibility.StareBrainAccessibilityService"
    android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
    android:exported="true">
    <intent-filter>
        <action android:name="android.accessibilityservice.AccessibilityService" />
    </intent-filter>
    <meta-data
        android:name="android.accessibilityservice"
        android:resource="@xml/accessibility_service_config" />
</service>

The configuration file at res/xml/accessibility_service_config.xml:

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged|typeViewClicked|typeViewFocused"
    android:accessibilityFeedbackType="feedbackGeneric"
    android:accessibilityFlags="flagReportViewIds|flagRequestEnhancedWebAccessibility"
    android:canPerformGestures="true"
    android:canRetrieveWindowContent="true"
    android:notificationTimeout="100" />

Key flags:

canRetrieveWindowContent: required for reading the UI tree canPerformGestures: required for touch simulation flagReportViewIds: gives us stable resource IDs to target actions

class StareBrainAccessibilityService : AccessibilityService() {

    companion object {
        @Volatile
        private var instance: StareBrainAccessibilityService? = null

        fun getInstance(): StareBrainAccessibilityService? = instance
        fun isConnected(): Boolean = instance != null
    }

    override fun onServiceConnected() {
        instance = this
        StareBrainApp.onAccessibilityConnected()
    }

    override fun onAccessibilityEvent(event: AccessibilityEvent) {
        // Handled by ScreenReader — see below
    }

    override fun onInterrupt() {
        // Service interrupted — reset any pending action state
    }

    override fun onDestroy() {
        instance = null
        super.onDestroy()
    }
}

The singleton pattern here is intentional. AccessibilityService is managed by the OS — we can't instantiate it ourselves. Other components (the agent execution engine, TaskExecutor) reach it through getInstance(), and we guard every call with a null check and a fallback to StepStatus.Failed.

ScreenReader wraps the accessibility tree traversal:

class ScreenReader(private val service: StareBrainAccessibilityService) {

    fun captureCurrentScreen(): ScreenSnapshot {
        val rootNode = service.rootInActiveWindow
            ?: return ScreenSnapshot.empty()

        return ScreenSnapshot(
            packageName = rootNode.packageName?.toString() ?: "",
            nodes = collectNodes(rootNode),
            capturedAt = System.currentTimeMillis()
        )
    }

    private fun collectNodes(root: AccessibilityNodeInfo): List<UINode> {
        val nodes = mutableListOf<UINode>()
        traverseTree(root, nodes, depth = 0)
        return nodes
    }

    private fun traverseTree(
        node: AccessibilityNodeInfo,
        output: MutableList<UINode>,
        depth: Int
    ) {
        if (depth > MAX_TREE_DEPTH) return

        val uiNode = UINode(
            resourceId = node.viewIdResourceName ?: "",
            text = node.text?.toString() ?: "",
            contentDescription = node.contentDescription?.toString() ?: "",
            className = node.className?.toString() ?: "",
            isClickable = node.isClickable,
            isEditable = node.isEditable,
            isFocused = node.isFocused,
            bounds = node.getBoundsInScreen()
        )

        if (uiNode.isInteresting()) {
            output.add(uiNode)
        }

        for (i in 0 until node.childCount) {
            node.getChild(i)?.let { child ->
                traverseTree(child, output, depth + 1)
                child.recycle()
            }
        }
    }

    companion object {
        private const val MAX_TREE_DEPTH = 15
    }
}

private fun UINode.isInteresting(): Boolean =
    isClickable || isEditable || text.isNotEmpty() || contentDescription.isNotEmpty()

Two details worth noting:

child.recycle(): AccessibilityNodeInfo objects must be recycled manually on Android versions below API 33. Missing this causes memory leaks that surface as degraded performance mid-task.

MAX_TREE_DEPTH: Some apps (especially WebViews) produce extremely deep trees. Without a depth cap, traversal can take hundreds of milliseconds and produce thousands of nodes the model doesn't need.

ActionDispatcher translates AgentStep actions into AccessibilityNodeInfo interactions:

class ActionDispatcher(private val service: StareBrainAccessibilityService) {

    fun dispatch(action: StepAction): ActionResult {
        return when (action) {
            is StepAction.Click -> clickNode(action.targetId)
            is StepAction.TypeText -> typeIntoNode(action.targetId, action.text)
            is StepAction.Scroll -> scroll(action.direction)
            is StepAction.PressBack -> pressBack()
            is StepAction.Navigate -> navigate(action.packageName)
        }
    }

    private fun clickNode(resourceId: String): ActionResult {
        val node = findNodeById(resourceId)
            ?: return ActionResult.Failure("Node not found: $resourceId")

        val success = node.performAction(AccessibilityNodeInfo.ACTION_CLICK)
        node.recycle()

        return if (success) ActionResult.Success else ActionResult.Failure("Click failed on $resourceId")
    }

    private fun typeIntoNode(resourceId: String, text: String): ActionResult {
        val node = findNodeById(resourceId)
            ?: return ActionResult.Failure("Node not found: $resourceId")

        node.performAction(AccessibilityNodeInfo.ACTION_FOCUS)

        val args = Bundle().apply {
            putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, text)
        }
        val success = node.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)
        node.recycle()

        return if (success) ActionResult.Success else ActionResult.Failure("Type failed on $resourceId")
    }

    private fun pressBack(): ActionResult {
        service.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK)
        return ActionResult.Success
    }

    private fun findNodeById(resourceId: String): AccessibilityNodeInfo? {
        val root = service.rootInActiveWindow ?: return null
        return root.findAccessibilityNodeInfosByViewId(resourceId).firstOrNull()
    }
}

AccessibilityService requires the user to navigate to Settings → Accessibility → StareBrain and toggle it on manually. Android intentionally makes this non-programmable — no API can enable the service on the user's behalf.

We handle this with AccessibilityPermissionGuard:

class AccessibilityPermissionGuard(private val context: Context) {

    fun isGranted(): Boolean =
        StareBrainAccessibilityService.isConnected()

    fun requestPermission(onGranted: () -> Unit) {
        // Show explanation dialog, then deep-link to settings
        val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS).apply {
            flags = Intent.FLAG_ACTIVITY_NEW_TASK
        }
        context.startActivity(intent)
        // onGranted is called when isConnected() becomes true — polled by TaskExecutor
    }
}

We show a one-time explanation before redirecting: what the permission enables, what data it can access, and that it never leaves the device. Users who understand what they're granting convert at a much higher rate than users who get dropped into the settings screen without context.

Google Play's policy on AccessibilityService is strict: the service must be used to assist users with disabilities, or for a core use case that cannot be achieved another way. Automation is explicitly listed as a prohibited use in some contexts.

StareBrain's position: the service is used to execute tasks the user explicitly requested, with confirmation steps before destructive actions (covered in Part 22). We document this clearly in the Play Store listing and in the permission explanation. Whether this satisfies review is an ongoing conversation — we'll cover the submission process and reviewer feedback in a later part.

Part 25 will cover the full execution pipeline end to end: how a user instruction travels from voice or text input through planning, validation, approval, and step-by-step execution using the accessibility layer, with error recovery at each stage.

The pieces are all in place. The next part connects them.

StareBrain is an on-device AI agent for Android. If you're building with AccessibilityService or have hit the Play Store review wall, I'm in the comments.