Building an Android AI Agent, Part 11: Mid-Task Replanning Without Starting Over

A developer building an Android AI agent added mid-task replanning that preserves completed steps instead of regenerating the full plan from scratch. The approach sends the LLM the completed steps, the failed step with its failure reason, a capped screen snapshot summary, and a target count for remaining steps, then merges the revised steps onto the existing task. The engineer argues that naive full regeneration risks replaying finished actions and causing duplicate side effects, while also discarding the live screen context that makes replanning useful.

In Part 10 we added an obstruction detector that handles spinners, dismissible overlays, and permission dialogs inline. When none of those apply and the expected screen simply isn't there, the executor returns a FAILED step with a reason. That's where this part picks up. The task hasn't failed — two of four steps completed successfully. What failed is the plan for step three. The agent needs a new plan for the remaining steps, built on top of what already happened. The naive recovery is to send the original instruction back to the LLM and regenerate the full step list. This has two problems. First, it replays completed steps. If step one opened the right app and step two filled in a form, regenerating from scratch may produce a plan that tries to do both again — which either fails the form is already filled or causes a side effect it submits twice . Second, it throws away context. The LLM that generated the original plan didn't know what the screen looks like right now. The replanning call does — and that's the only reason to make it. The right call: send the completed steps, the failed step, the failure reason, and the current screen snapshot. Ask for a revised plan for the remaining steps only. data class ReplanRequest val originalInstruction: String, val completedSteps: List<AgentStep , val failedStep: AgentStep, val failureReason: String, val currentSnapshot: ScreenSnapshot, val remainingStepCount: Int data class ReplanResponse val revisedSteps: List<AgentStep , val explanation: String The completedSteps tell the LLM what has already happened and must not be repeated. The failedStep and failureReason tell it what went wrong. The currentSnapshot tells it what the screen actually looks like right now. The remainingStepCount gives it a soft target — how many steps were originally planned for the rest of the task, so it doesn't over-engineer the recovery. The ScreenSnapshot needs to be readable by the LLM without overwhelming the context window. A flat summary works better than a full node dump. fun ScreenSnapshot.toPromptSummary : String { val sb = StringBuilder sb.appendLine "Current screen: ${packageName ?: "unknown"}" if visibleTexts.isNotEmpty { sb.appendLine "Visible text first 10 :" visibleTexts.take 10 .forEach { sb.appendLine " - $it" } } if interactableIds.isNotEmpty { sb.appendLine "Interactable elements first 10 :" interactableIds.take 10 .forEach { sb.appendLine " - $it" } } return sb.toString } Capping at 10 items each keeps the prompt tight. The LLM doesn't need the full accessibility tree — it needs enough to understand what's on screen and what's tappable. suspend fun requestReplan request: ReplanRequest, apiClient: StareBrainApiClient : ReplanResponse { val prompt = buildString { appendLine "Original instruction: ${request.originalInstruction}" appendLine appendLine "Completed steps:" request.completedSteps.forEachIndexed { i, step - appendLine " ${i + 1}. ${step.instruction} DONE " } appendLine appendLine "Failed step: ${request.failedStep.instruction}" appendLine "Failure reason: ${request.failureReason}" appendLine appendLine "Current screen state:" appendLine request.currentSnapshot.toPromptSummary appendLine appendLine "Generate revised steps for the remaining task only." appendLine "Do not repeat completed steps." appendLine "Target approximately ${request.remainingStepCount} steps." } return apiClient.replan prompt } fun AgentTask.withRevisedSteps revised: List<AgentStep : AgentTask { val kept = steps.take currentStepIndex // completed steps stay val merged = kept + revised return copy steps = merged, status = TaskStatus.IN PROGRESS } The completed steps are preserved exactly. The revised steps replace everything from the failure point forward. The executor picks up from currentStepIndex as normal — it doesn't know a replan happened. Not every failure deserves a replan. Two cases where replanning is the wrong call: Repeated failures on the same step. If the revised plan hits the same failure on the same screen twice, the screen state is genuinely unexpected and the LLM isn't going to reason its way out of it. Surface the failure to the user with enough context to decide whether to continue or abandon. Confirmed steps that failed. If the user confirmed an action and it still failed, that's a product bug, not a planning problem. Replanning would just generate another confirmation prompt for the same broken action. fun shouldReplan step: AgentStep, task: AgentTask : Boolean { if step.result?.confirmedBy = null return false // confirmed action failed — surface to user val previousAttempts = task.steps.count { it.instruction == step.instruction && it.status == StepStatus.FAILED } return previousAttempts < 2 } Part 12: the AgentContext system — how StareBrain builds and maintains a model of what the user is doing across app switches, so the agent has context before the user says anything.