{"slug": "the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible", "title": "The LLM can declare that a task \"appears to be finished. It will not be possible to check it off.", "summary": "A developer building an AI Copilot for student projects described redesigning task-completion logic so that a language model can never mark work as done on its own. The system now requires an explicit student confirmation flag before any task status can be set to COMPLETED, and the copilot's mark_task_completed tool no longer writes to the database, instead only prompting the model to suggest completion. The developer cited Anthropic research on sycophancy and a PNAS field experiment showing unguarded GPT-4 tutoring access harmed student performance, framing the design as an application of OWASP's Excessive Agency guidance that the model suggests while the system decides.", "body_md": "We develop an [AI Copilot](https://elivio.experimentlabs.in/copilot) whose functionality unfolds across projects with students, including a step-by-step project plan, an interactive voice session so that it can view what appears on their shared screen and a task list that culminates in a shared public Proof of Work page. For such an item \"done\" is the whole purpose. If the task is done when it is not, the progress report is incorrect, the proof-of-work page is incorrect and the student discovers that the statement \"I'm done\" is apt as it being done.\n\nLanguage models are extremely excited to let you know you're finished. It isn't merely a guess. The five top AI assistants tested by Anthropic \"always appear sycophantic\" to favour \"responses that align with user beliefs over truthful responses\" ( [Sharma et al., 2023](https://arxiv.org/abs/2310.13548)). I'm done – is a user belief.\n\nIn school the price is greater than its appearance. In a field experiment involving almost a thousand students from high schools in maths, the students who received a tutor in the form of the GPTEE were able to get their practice grades 48% higher. The students who got the access when it was removed had a score 17% worse than those that never had the access. Largely that harm was taken away by a second tutor supplied with prompts to protect learning. The authors' word for how students used the unguarded one: a \"crutch\" ([Bastani et al., PNAS 2025](https://doi.org/10.1073/pnas.2422633122)).\n\nThis post's topics are how we distinguish 'this looks finished' from 'this is finished' - no one can actually see what is finished, someone else looks at it, and the server re-runs only when the student pushes the button. It includes the bugs that we encountered in the judge (most of the lessons were here), research that aligns with each decision.\n\nWe can't hear you, but it refuses anyway, according to rule 1: one writer that is.\n\nAll the status changes of a task occur in one function. Only allows a call to pass a status of `COMPLETED` if the caller sets `confirmed: true`:\n\n```\nasync updateActionStatus(\n  userId: string,\n  actionId: string,\n  status: ActionStatus,\n  options: { confirmed?: boolean } = {}\n) {\n  if (status === ActionStatus.COMPLETED && !options.confirmed) {\n    throw new AppError(\n      'Task completion must be explicitly confirmed by the student',\n      HTTP_STATUS_CODES.CONFLICT\n    );\n  }\n\n  // transaction: ownership check, write the status,\n  // then complete the week and plan if every task in them is done\n}\n```\n\nThere are only two call sites that are confirmed as `true`: one in the complete flow in a session and the student's own one in the status control in the plan page. There is nothing in the AI arena that does. Only the function is considered as a door as it is unlikely that anyone would ever write that second one directly.\n\nIt is a textbook's example of OWASP's Top 10 for LLM applications. In the LLM06:2025 Excessive Agency, it is defined as an app that is \"excessive autonomy\" which means it does not verify and approve high-impact actions independently. Its mitigation is similar to our design doc: \"Don't let an LLM make decisions about “all” vs “not all” when it comes to authorizing an action, implement authorization in downstream systems instead. The shape of a task completion isn't one of the security incidents, and the same model prevails: the model suggests, the system decides.\n\nRule 2: The “complete” tool will have no output.\n\nThe copilot provides a tool called `mark_task_completed` that the copilot can use. The first iteration of this was as the model guessed from the conversation that the task was complete, it would print out `COMPLETED` and alert the student that they could proceed to the next task. The only sanction was the court of judgment of the model.\n\nThe tool is not used to access the database today. It will give an instruction to the model:\n\n``` js\nexport const markTaskCompletedTool = tool(\n  async ({ reason }) => {\n    return `COMPLETION_NEEDS_CONFIRM: Do not mark this task complete yourself and do not claim it is done. It looks finished (${reason}). Tell the student their work looks complete and ask them to run the completion check and tap \"Mark complete\" to confirm — only their tap marks it done.`;\n  },\n  {\n    name: 'mark_task_completed',\n    description: `Signal that the student's current task LOOKS finished so they can be prompted to confirm. This does NOT complete the task — only the student's one-tap confirm does. ...`,\n    // schema: studentId, actionId, reason\n  }\n);\n```\n\nWhat's the point of keeping the tool? The model will attempt to make attempts to act when the student says \"I'm done\". This is more effective than hoping the prompt holds – a tool that routes that impulse to the next right direction of the check/that button click.\n\nIts sibling, mark_task_in_progress writes. It's easy and inexpensive to get started on a task wrong. Finishing one isn't.\n\nThe bug is the exact same tool, which is described twice.\n\nWe have a text chat based on LangChain tools.We have a text chat with LangChain tools. Raw function declarations are required for our voice mode ( Gemini Live ). Thus, there were two files that contained the description of the tool, which is what we were looking for. We changed one but not the other and the voice read out the previous text which indicates that the tool marks the task COMPLETED. So it did what it said it would: it informed pupils that their task was ripe for harvesting. No writing was on them.\n\nIt was a wrong database and anyway the student was misled. In fact, anthropic's advice on agents is to \"pay equal attention to tool definitions and specification as you do your prompts\" ([Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)) and \"even small tweaks to tool descriptions can make dramatic improvements\" ([Writing effective tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents)). That's two ways round. A little bit of a stale word description killed more than it helped.\n\nIt is quite simple to implement the fix, and it does: A unit test reads in both files and aborts if the two descriptions no longer say the same key phrases. The said message of assertion is \"the two copies must not drift. According to the test file: \"Only the wording is regressing and this file is the reason for that.\n\nThe judge: a separate call which only reads.\n\nWhy not simply have the copilot self check? As models are not good at it. A research article on self-correction which reads: \"Self-correction is not effective for \"LLMs\" when they perform without external intervention, and sometimes even when they attempt to do so. ([Huang et al., ICLR 2024](https://arxiv.org/abs/2310.01798))\". The request by the co-pilot to the student that just agreed \"are you sure?\" is the kind of self-correction that was just asked.\n\nThus, a separate model call is made when the student executes the completion check. Has only one task, no tools, and never marks anything. If that sounds familiar, this is Anthropic's evaluator-optimizer workflow, in which the “one LLM call generates a response, while another provides evaluation and feedback in a loop,” and which “works best when we have clear evaluation criteria.” It also includes some modifications to the [Dual LLM pattern](https://simonwillison.net/2023/Apr/25/dual-llm-pattern/), which has been proposed by Simon Willison, where the model that controls untrusted content does not have access to tools. We get back a JSON, validate it and then code gets to adjudicate the next action.\n\nIt is not too late to start doing inexpensive things that will keep the grader away from the graded agent. Even though they \"show evidence that they were not acting according to user intentions,\" METR has recorded frontier versions “modifying the tests or scoring code” to achieve higher scores ([METR, 2025](https://metr.org/blog/2025-06-05-reward-hacking-recent/)). We don't have any tool that involves the check and/or task status.\n\nThe judge receives 3 inputs:\n\nTask spec: task title, description and the week in which the task is to be performed.\n\nThe last 60 turns of the transcript for the session. We capped it since if it was too long, it would blow the input limit and the student would not be able to check it at all. It also alleviates a common pitfall in models: input information towards the start or end of the context is better for modeling purposes. ([Liu et al., 2023](https://arxiv.org/abs/2307.03172)) The judge's prompt is placed first, followed by the rules and the transcript in between.\n\nReturns a list of steps, not a yes/no:\n\n```\ninterface VerifyResult {\n  requirements: { title: string; met: boolean; note: string }[];\n  allMet: boolean;\n  blockers: string[];\n  summary: string;\n}\n```\n\nEach requirement the student encounters, he or she writes a note in front of it indicating what remains. With training on the MATH dataset ([Lightman et al., 2023](https://arxiv.org/abs/2305.20050)), \"Let's Verify Step by Step\" has been found to \"significantly outperform\" outcome supervision with feedback on only the final outcome. That is also an environment with the same intuition: A checklist specifies where it did not meet, and \"not done yet\" is not a checklist item.\n\nDon't evaluate the role, evaluate the work!\n\nWe ran our first version through the judge's hands and it was the same context which the copilot is used to. That context concludes with the copilot's own instructions such as one informing the user about when to offer an auto-suggestion for the completion check. The judge decided that is a deliverable and \"run the completion check\" is a blocker. It would never be fulfilled by the student, as they were putting it into action. When it comes to regression test, our regression test just sums up it up in one line: \"All requirement green, allMet false, dead button.\n\nIt was obvious that a fix would be to construct the judge's input from the task only:\n\n``` js\n// Judge against the TASK, never the copilot persona.\nconst taskSpec = [\n  `Task: ${context.action.title}`,\n  context.action.description,\n  `Week ${context.week.weekNumber} — ${context.week.title}: ${context.week.description}`,\n].join('\\n\\n');\n```\n\nThe Chain-of-Verification paper carries an interesting point: answering the questions of verification should be done \"independently so that the answers are not influenced by the other answers\" ( [Dhuliawala et al., 2023](https://arxiv.org/abs/2309.11495)). If a judge is in the agent's context, then the judge is not independent.\n\nFour quickies we picked up the hard way\n\nThe typical list of judges of the LLMs seems to include position bias, verbosity bias, and self-enhancing bias to list a few ([Zheng et al., 2023](https://arxiv.org/abs/2306.05685))), as well as a \"sensitivity to prompt complexity and length\" and a \"tendency toward leniency\" ([Thakur et al., 2024](https://arxiv.org/abs/2406.12624)). We went astray in both cases: over-restrictive in the first two rules below, and too lax in the remaining two.\n\nNote: The copilot's stretch questions are not requirements. The copilot will dig for depth: \"what was the depth of it?,\" \"what else did you notice?\". After reading the transcript, the judge included those things in the task. If a student was successful at the task, she was told that she did not provide the “other” examples that had not been requested. The prompting has changed: Requirements are now only given from the task description and what the session agreed to create at the beginning of the session.\n\nIncluding the task description in the transcript prioritizes the exact value over the task description. The second regression case: The task must have exactly two roles to choose from, \"Owner\" and \"Contributor\". Student made \"Prompt\", copilot said “OK”. A judge, who reads that transcript, is tempted to agree. Now, for named values, a match on the work is a miss if the match is against the task text, verbatim. As the voice copilot views a student's screen, the same rule is applied.\n\nThe judge's output also goes straight to the student, so it writes in second person (\"You haven't shared your observations yet\"), not \"The student has not...\".\n\nIf the tap is re-run then the check is repeated.\n\nThe result of the check is displayed as a card and the card has a \"Mark complete\" button. That's clearly the approach- to take the card at its face value; if it read “allMet”, do the task. We don't. When the button is tapped, the server will run the check again:\n\n```\n// Authoritative gate: the server, not the client's earlier card, is the judge.\nconst check = await runCompletionCheck(userId, session, screenshotBase64);\nif (!check.allMet) {\n  return { completed: false, check }; // HTTP 200 with a fresh checklist\n}\n\nawait planService.updateActionStatus(\n  userId,\n  session.actionId,\n  ActionStatus.COMPLETED,\n  { confirmed: true } // the student tapped\n);\n```\n\nTwo details matter:\n\nWhen the task is finished the remainder (closing the session, summarising it in memory for the following task; updating the progress report) will be best-effort. That is a task which is already complete, it isn't supposed to create an error for the student.\n\nThe Berkeley study found ” task verification” is one of the three entire classes of failure in multi-agent systems – which includes “premature termination” and “incorrect verification”). It claims that relying solely on the final (low-level) check is not enough ([Cemri et al., 2025](https://arxiv.org/abs/2503.13657)). We arrived at three layers for that reason: 1st: the model sends a signal 2nd: the student runs the check 3rd: the server does a check at the time of the commit.\n\nThe judge is the bad guy!The judge is the bad guy!\n\nOccasionally the student is correct and the teacher is incorrect and the student is \" stuck \" taking the check again. The judge research is simple to come to an end. Even in a simple setup, the best judges were not equal to the human judges, \"suggesting caution [may be warranted] when using judges in more complex setups\" (Thakur et al.). A voice session with a screen shot is a quite complex setup.\n\nThe only stupid stuck judge we see is when the check fails twice in a row, while the same check blocker is present on every check-iteration.The case we don't pretend it's the student's fault is if a check fails twice; on the first iteration and again on the second, and the same blocker is present in each check.\n\n```\nif (\n  prior && !prior.allMet && !result.allMet &&\n  JSON.stringify(prior.blockers) === JSON.stringify(result.blockers)\n) {\n  result.summary =\n    \"This check keeps returning the same result — that's likely on our side, not yours. \" +\n    result.summary;\n}\n```\n\nIt's a very basic approach, but makes \"the AI says I'm wrong forever\" reportable.\n\nWhat this doesn't solve\n\nIt's not not nothing that completes without AI, it's The AI never decides. The plan page continues to have a manual \"Mark Complete\" control, and remains as the student's confirmation. We think that was intentional! The guard is there for the model to be able to mark work without it being considered the student's responsibility, but not to remove it from the student's responsibility! However, that does not mean that all paths are checked for the flow of the session, it only means that the check protects the flow of the session.\n\nThe completion judge is not yet based on the notion of being human. It has 2 \"live\" regression cases and the wording tests. In our plan-generation judge, however, doesn't as of yet, but it should given all the above.\n\nEach completion takes TWO judge calls - the check and then re-check on the tap. We felt it was worth for it to be correct.\n\nThe screen shots are stored in the Process memory. Run while service is executing, as one instance. When we scale out without sending sessions to the same instance, frames get spread out and the checks go back to checking the transcript only. Miss, that's a miss, that's a wrong answer but a lot of answers that are wrong.\n\n`{ to the last }` in the judge's JSON to get the answer. Has survived, but the first thing I'd change to a structured-output mode.\nTakeaways\n\nWhat do you do about \"done\" with your agents? I'm particularly interested to see if anyone has a better solution than a second Model call to look at one person's screen to access work than I have, and that has a life independent of the Model.\n\nI work at [Elivio](https://elivio.experimentlabs.in/copilot) where this copilot has been used by the college and study abroad students. This is where you'll find the completion check if you wish to see it in action.\n\nReferences", "url": "https://wpnews.pro/news/the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible", "canonical_source": "https://dev.to/experimentlabs/the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible-to-check-it-off-1ff8", "published_at": "2026-09-17 12:14:28+00:00", "updated_at": "2026-09-17 12:23:03.922974+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-products", "large-language-models", "ai-ethics"], "entities": ["Anthropic", "OWASP", "PNAS", "GPT-4", "Elivio"], "alternates": {"html": "https://wpnews.pro/news/the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible", "markdown": "https://wpnews.pro/news/the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible.md", "text": "https://wpnews.pro/news/the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible.txt", "jsonld": "https://wpnews.pro/news/the-llm-can-declare-that-a-task-appears-to-be-finished-it-will-not-be-possible.jsonld"}}