# Building an Android AI Agent, Part 11: Mid-Task Replanning Without Starting Over

> Source: <https://dev.to/starebrain/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over-4nnf>
> Published: 2026-11-01 04:15:07+00:00

In Part 10 we added an obstruction detector that handles spinners, dismissible overlays, and permission dialogs inline. When none of those apply and the expected screen simply isn't there, the executor returns a `FAILED` step with a reason.

That's where this part picks up. The task hasn't failed — two of four steps completed successfully. What failed is the plan for step three. The agent needs a new plan for the remaining steps, built on top of what already happened.

The naive recovery is to send the original instruction back to the LLM and regenerate the full step list. This has two problems.

First, it replays completed steps. If step one opened the right app and step two filled in a form, regenerating from scratch may produce a plan that tries to do both again — which either fails (the form is already filled) or causes a side effect (it submits twice).

Second, it throws away context. The LLM that generated the original plan didn't know what the screen looks like right now. The replanning call does — and that's the only reason to make it.

The right call: send the completed steps, the failed step, the failure reason, and the current screen snapshot. Ask for a revised plan for the remaining steps only.

```
data class ReplanRequest(
    val originalInstruction: String,
    val completedSteps: List<AgentStep>,
    val failedStep: AgentStep,
    val failureReason: String,
    val currentSnapshot: ScreenSnapshot,
    val remainingStepCount: Int
)

data class ReplanResponse(
    val revisedSteps: List<AgentStep>,
    val explanation: String
)
```

The `completedSteps` tell the LLM what has already happened and must not be repeated. The `failedStep` and `failureReason` tell it what went wrong. The `currentSnapshot` tells it what the screen actually looks like right now. The `remainingStepCount` gives it a soft target — how many steps were originally planned for the rest of the task, so it doesn't over-engineer the recovery.

The `ScreenSnapshot` needs to be readable by the LLM without overwhelming the context window. A flat summary works better than a full node dump.

```
fun ScreenSnapshot.toPromptSummary(): String {
    val sb = StringBuilder()
    sb.appendLine("Current screen: ${packageName ?: "unknown"}")
    if (visibleTexts.isNotEmpty()) {
        sb.appendLine("Visible text (first 10):")
        visibleTexts.take(10).forEach { sb.appendLine("  - $it") }
    }
    if (interactableIds.isNotEmpty()) {
        sb.appendLine("Interactable elements (first 10):")
        interactableIds.take(10).forEach { sb.appendLine("  - $it") }
    }
    return sb.toString()
}
```

Capping at 10 items each keeps the prompt tight. The LLM doesn't need the full accessibility tree — it needs enough to understand what's on screen and what's tappable.

```
suspend fun requestReplan(
    request: ReplanRequest,
    apiClient: StareBrainApiClient
): ReplanResponse {
    val prompt = buildString {
        appendLine("Original instruction: ${request.originalInstruction}")
        appendLine()
        appendLine("Completed steps:")
        request.completedSteps.forEachIndexed { i, step ->
            appendLine("  ${i + 1}. ${step.instruction} [DONE]")
        }
        appendLine()
        appendLine("Failed step: ${request.failedStep.instruction}")
        appendLine("Failure reason: ${request.failureReason}")
        appendLine()
        appendLine("Current screen state:")
        appendLine(request.currentSnapshot.toPromptSummary())
        appendLine()
        appendLine("Generate revised steps for the remaining task only.")
        appendLine("Do not repeat completed steps.")
        appendLine("Target approximately ${request.remainingStepCount} steps.")
    }
    return apiClient.replan(prompt)
}
fun AgentTask.withRevisedSteps(revised: List<AgentStep>): AgentTask {
    val kept = steps.take(currentStepIndex) // completed steps stay
    val merged = kept + revised
    return copy(
        steps = merged,
        status = TaskStatus.IN_PROGRESS
    )
}
```

The completed steps are preserved exactly. The revised steps replace everything from the failure point forward. The executor picks up from `currentStepIndex` as normal — it doesn't know a replan happened.

Not every failure deserves a replan. Two cases where replanning is the wrong call:

**Repeated failures on the same step.** If the revised plan hits the same failure on the same screen twice, the screen state is genuinely unexpected and the LLM isn't going to reason its way out of it. Surface the failure to the user with enough context to decide whether to continue or abandon.

**Confirmed steps that failed.** If the user confirmed an action and it still failed, that's a product bug, not a planning problem. Replanning would just generate another confirmation prompt for the same broken action.

```
fun shouldReplan(step: AgentStep, task: AgentTask): Boolean {
    if (step.result?.confirmedBy != null) return false // confirmed action failed — surface to user
    val previousAttempts = task.steps.count { 
        it.instruction == step.instruction && it.status == StepStatus.FAILED 
    }
    return previousAttempts < 2
}
```

Part 12: the AgentContext system — how StareBrain builds and maintains a model of what the user is doing across app switches, so the agent has context before the user says anything.