{"slug": "building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over", "title": "Building an Android AI Agent, Part 11: Mid-Task Replanning Without Starting Over", "summary": "A developer building an Android AI agent added mid-task replanning that preserves completed steps instead of regenerating the full plan from scratch. The approach sends the LLM the completed steps, the failed step with its failure reason, a capped screen snapshot summary, and a target count for remaining steps, then merges the revised steps onto the existing task. The engineer argues that naive full regeneration risks replaying finished actions and causing duplicate side effects, while also discarding the live screen context that makes replanning useful.", "body_md": "In Part 10 we added an obstruction detector that handles spinners, dismissible overlays, and permission dialogs inline. When none of those apply and the expected screen simply isn't there, the executor returns a `FAILED` step with a reason.\n\nThat's where this part picks up. The task hasn't failed — two of four steps completed successfully. What failed is the plan for step three. The agent needs a new plan for the remaining steps, built on top of what already happened.\n\nThe naive recovery is to send the original instruction back to the LLM and regenerate the full step list. This has two problems.\n\nFirst, it replays completed steps. If step one opened the right app and step two filled in a form, regenerating from scratch may produce a plan that tries to do both again — which either fails (the form is already filled) or causes a side effect (it submits twice).\n\nSecond, it throws away context. The LLM that generated the original plan didn't know what the screen looks like right now. The replanning call does — and that's the only reason to make it.\n\nThe right call: send the completed steps, the failed step, the failure reason, and the current screen snapshot. Ask for a revised plan for the remaining steps only.\n\n```\ndata class ReplanRequest(\n    val originalInstruction: String,\n    val completedSteps: List<AgentStep>,\n    val failedStep: AgentStep,\n    val failureReason: String,\n    val currentSnapshot: ScreenSnapshot,\n    val remainingStepCount: Int\n)\n\ndata class ReplanResponse(\n    val revisedSteps: List<AgentStep>,\n    val explanation: String\n)\n```\n\nThe `completedSteps` tell the LLM what has already happened and must not be repeated. The `failedStep` and `failureReason` tell it what went wrong. The `currentSnapshot` tells it what the screen actually looks like right now. The `remainingStepCount` gives it a soft target — how many steps were originally planned for the rest of the task, so it doesn't over-engineer the recovery.\n\nThe `ScreenSnapshot` needs to be readable by the LLM without overwhelming the context window. A flat summary works better than a full node dump.\n\n```\nfun ScreenSnapshot.toPromptSummary(): String {\n    val sb = StringBuilder()\n    sb.appendLine(\"Current screen: ${packageName ?: \"unknown\"}\")\n    if (visibleTexts.isNotEmpty()) {\n        sb.appendLine(\"Visible text (first 10):\")\n        visibleTexts.take(10).forEach { sb.appendLine(\"  - $it\") }\n    }\n    if (interactableIds.isNotEmpty()) {\n        sb.appendLine(\"Interactable elements (first 10):\")\n        interactableIds.take(10).forEach { sb.appendLine(\"  - $it\") }\n    }\n    return sb.toString()\n}\n```\n\nCapping at 10 items each keeps the prompt tight. The LLM doesn't need the full accessibility tree — it needs enough to understand what's on screen and what's tappable.\n\n```\nsuspend fun requestReplan(\n    request: ReplanRequest,\n    apiClient: StareBrainApiClient\n): ReplanResponse {\n    val prompt = buildString {\n        appendLine(\"Original instruction: ${request.originalInstruction}\")\n        appendLine()\n        appendLine(\"Completed steps:\")\n        request.completedSteps.forEachIndexed { i, step ->\n            appendLine(\"  ${i + 1}. ${step.instruction} [DONE]\")\n        }\n        appendLine()\n        appendLine(\"Failed step: ${request.failedStep.instruction}\")\n        appendLine(\"Failure reason: ${request.failureReason}\")\n        appendLine()\n        appendLine(\"Current screen state:\")\n        appendLine(request.currentSnapshot.toPromptSummary())\n        appendLine()\n        appendLine(\"Generate revised steps for the remaining task only.\")\n        appendLine(\"Do not repeat completed steps.\")\n        appendLine(\"Target approximately ${request.remainingStepCount} steps.\")\n    }\n    return apiClient.replan(prompt)\n}\nfun AgentTask.withRevisedSteps(revised: List<AgentStep>): AgentTask {\n    val kept = steps.take(currentStepIndex) // completed steps stay\n    val merged = kept + revised\n    return copy(\n        steps = merged,\n        status = TaskStatus.IN_PROGRESS\n    )\n}\n```\n\nThe completed steps are preserved exactly. The revised steps replace everything from the failure point forward. The executor picks up from `currentStepIndex` as normal — it doesn't know a replan happened.\n\nNot every failure deserves a replan. Two cases where replanning is the wrong call:\n\n**Repeated failures on the same step.** If the revised plan hits the same failure on the same screen twice, the screen state is genuinely unexpected and the LLM isn't going to reason its way out of it. Surface the failure to the user with enough context to decide whether to continue or abandon.\n\n**Confirmed steps that failed.** If the user confirmed an action and it still failed, that's a product bug, not a planning problem. Replanning would just generate another confirmation prompt for the same broken action.\n\n```\nfun shouldReplan(step: AgentStep, task: AgentTask): Boolean {\n    if (step.result?.confirmedBy != null) return false // confirmed action failed — surface to user\n    val previousAttempts = task.steps.count { \n        it.instruction == step.instruction && it.status == StepStatus.FAILED \n    }\n    return previousAttempts < 2\n}\n```\n\nPart 12: the AgentContext system — how StareBrain builds and maintains a model of what the user is doing across app switches, so the agent has context before the user says anything.", "url": "https://wpnews.pro/news/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over", "canonical_source": "https://dev.to/starebrain/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over-4nnf", "published_at": "2026-11-01 04:15:07+00:00", "updated_at": "2026-11-01 04:21:57.475493+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence", "large-language-models"], "entities": ["Android", "StareBrainApiClient"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over", "markdown": "https://wpnews.pro/news/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over.md", "text": "https://wpnews.pro/news/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over.txt", "jsonld": "https://wpnews.pro/news/building-an-android-ai-agent-part-11-mid-task-replanning-without-starting-over.jsonld"}}