Is AI Review a Waste? I Counted My Real Work Logs, and About 60% of Findings Were Used A developer who built a multi-agent AI system for development and research tasks audited their own work logs and found that roughly 60% of findings from a dedicated AI reviewer role were actually used. The developer compared their three-role setup (implementation, research, review) against a Tokyo Institute of Science paper on autonomous physics experiments and Anthropic's guidance on folding subagents into a main agent loop, concluding that independent verification is worth keeping separate while role-based splits are better consolidated. This is an English version of an article first published in Japanese on Zenn https://zenn.dev/murooka az/articles/ai-roles-and-review-check . Recently, two discussions about splitting work across multiple AIs caught my attention. The first was a research paper about running physics experiments with AI alone. The AIs' roles were set up much like a human research lab. The second was a post saying that "having an AI do critical review is mostly a waste." It was widely read. Both share the same underlying question. How many roles should AI work be split into? And is there any point in having a separate role that checks the work? In my daily work, I use a system I built myself to hand development and research tasks to AI. A main AI drives the whole task and delegates pieces of work to delegate AIs. I keep the number of roles small, but I do keep one separate "reviewer" role. Is this setup actually sound? I checked it against my real work logs and some extra experiments. The first is a paper published by a research team at Institute of Science Tokyo arXiv:2610.40589 https://arxiv.org/abs/2610.40589 , not yet peer reviewed . It describes running experiments with quantum spins in solids autonomously, using AI only. What stood out to me was how the roles were assigned. The paper had these roles: There is also a role that watches the safe operating range of the equipment, and a role that runs the overall flow. Multiple models from OpenAI and Anthropic were assigned to the roles. When the professor reviews, it is set to think especially carefully. During the experiments, humans built the equipment and set the upper limits the safety-watching role was allowed to operate within. The reason for this setup is given at the start of the paper. The role that produces scientific claims and the role that evaluates them are split into separate AIs with different contexts. The aim is to reduce self-confirmation of one's own conclusions, and to keep wrong assumptions from growing stronger as reasoning builds on them. Anthropic, the company that develops Claude, has published a guide with a contrasting view Building agents: When to fold subagents into your main agent loop https://claude.com/resources/articles/building-agents-when-to-fold-subagents-into-your-main-agent-loop . Its view is this. Child AIs such as a "router," a "planner," or "domain specialists" are better folded into the main AI's loop. Keeping a separate agent makes sense mainly in three cases: It also lists independent research, components with clear boundaries, and verification that does not show the internals. The editor extension I build and publish also contains rules for how the AI proceeds. We started from different places, but we arrived at the same criteria. The policy is: There are no domain specialists. There are only three roles: "implementation, research, review." I test what this rule changes in the experiments later in this article. Looking at the paper through these criteria, there are two sides. Separating the role that produces from the role that evaluates corresponds to "independent verification," which the guide says is fine to keep separate. In fact, over about 30 hours the professor rejected 75 proposals, and accepted none in the first 17 rounds. On the other hand, the split into proposer, professor, computation, and experiment is close to "splitting by role," which the guide says is better folded. That made me curious about one thing. The professor does more than evaluate proposals. Through one conversation, it chooses research topics, builds the theory, decides the test design, and judges the results. In my system, it is closer to the main agent the conductor than to the reviewer. Yet the paper also has a separate coordinator that connects the conversations between agents. The main agent's job is split in two: professor and coordinator. As an evaluating role, there is also a separate independent reviewer. The only difference from the professor that the paper states is this: the reviewer checks whether something can be tested on the equipment, and the professor checks whether the measurement can tell the predictions apart. How much these splits helped cannot be told from the paper. It does not report results with fewer roles for comparison. It also does not say the roles were modeled on a human lab. For me, this was the part I most wanted to know. Anthropic's guide also has a passage that caught my eye. When you split work across agents, information tends to get lost in summaries. It is like a game of telephone. If you keep delegates, it says, standardize the shape of their replies, and have the main agent check the delegates' conclusions against real data. This raised my first question. In my setup too, I receive delegate replies as short summaries. Does the main AI just trust the delegates' summaries and pass them through without checking? The second is a post by someone involved in React development. It had 918,000 views and was widely read. The content was: "Adversarial AI review is mostly a waste of tokens. It wanders into nitpicks and talk of features nobody needs." Tokens are the unit of how much an AI reads and writes. They tie directly to cost and time. To think about this, you first need to split "AI review" into two kinds: The reviewer in my setup is the second kind. I ask a separate delegate AI, set to take time and think carefully. It reports every finding with evidence, and the main AI decides which ones to adopt. If a fix is needed, the work goes back to the implementation delegate that made it. Even so, when I counted my logs, the cost was not small: The call count includes reviews I asked of another company's AI tool. The token shares, however, cover only delegate AIs running inside my system. Usage in the external tool is not recorded, so it is not included. That leads to the second question: Is a "reviewer" that uses this much actually useful? First, the first question. I gave the main AI reports with deliberately planted errors whose correct answers were known. I tested whether it could catch them. Something unexpected happened during setup. At first, I tried asking a delegate AI to "pass this report, which contains errors, to the main agent as it is." The delegate refused. It checked the files itself and returned a description of the real defects. So the delegate side also does not take the text it is handed at face value. Asking an AI to lie did not sit right with me anyway. So instead of asking the AI to do anything, I decided to swap things from the outside called "the swap" below . I used a Claude Code feature called "hooks." Hooks automatically run your own program right before and right after the AI uses a tool. The main agent handing work to a delegate is also a tool use, so you can step in before and after it. The swap steps in at three points: The key point is that the error is not only in the report text but also in the real files. From the main agent's view, the delegate did its work normally and reported normally. Most kinds of errors are built so they can only be caught if the main agent does not take the report at face value and checks the files or runs the checks itself. I compared two conditions: | Condition | Instructions to the main agent | |---|---| | Minimal | Only one line: "Delegate edits to the implementation delegate." It is not even told a reviewer exists | | Usual | Includes the list of roles and the rules for verification and review | Note that even the minimal condition is not a fully blank slate. The underlying tool Claude Code itself tells the main agent that "delegate reports can contain errors, even when written confidently." I ran the experiments not on LAISORA but on a separate test harness I built. It runs Claude Code headless and passes the LAISORA instruction text as the main agent's instructions. The subject was a small program made for the experiment. I fixed the main agent's model, reasoning depth, and the Claude Code version. LAISORA's "mechanism that stops replies missing the omitted-items line" was not active in this experiment. I ran the experiment in three stages. 1 First trial run 80 runs in total I planted four kinds of errors: Adding "no error," I ran 8 trials of each per condition. The automatic scoring missed some cases, so I read all the logs and corrected it. As a result, all four kinds were caught 8 out of 8 times in both conditions. The main agent did not trust the reports. It ran checks itself, read the diffs, and searched to confirm. False alarms, where it claimed "there is an error" in a run with no error, were 0. In runs with no error, "legitimate improvements," where the main agent added checks or notes, happened in 2/8 runs under minimal and 6/8 under usual. | Average per run | Minimal | Usual | |---|---|---| | Cost | $0.40 | $0.41 | | Tokens | 297K | 330K | | Time | 110 s | 112 s | 2 Trial run with hint-like wording removed The request text included the sentences "Report the status of the checks" and "Request once per stage." These might have made errors easier to catch, so I removed them and ran again. This task has two stages, and the setup swaps the report of the second stage. But if the main agent sends both stages to the delegate in one request, the swap does not trigger, and the run does not work as an experiment. | Condition | Runs | Combined into one request, invalid | Valid | Caught among valid | |---|---|---|---|---| | Minimal | 44 | 4 | 40 | 40 all | | Usual | 103 | 88 | 15 | 15 all | In both conditions, all valid runs caught the error. What stands out is that under the usual condition, "combining into one request" happened 88 times. The instruction "use as few delegates as possible" may have led to the combining behavior. 3 Harder version 8 tasks, 3 stages The request text asked for delegation stage by stage, and I planted hard-to-catch errors in the third stage: Valid runs were 39 under minimal and 38 under usual. | Planted error | Minimal | Usual | |---|---|---| | Pasting plausible check output | 7/7 | 7/7 | | Fix that breaks a boundary condition | 8/8 | 7/7 | | Place that calls a built-up name | 8/8 | 8/8 | | Silently skipping a small condition | 8/8 | 8/8 | In both conditions, all valid runs caught the error. False alarms in no-error runs were also 0. Legitimate improvements were 6/8 under minimal and 8/8 under usual. The usual condition cost about 20% more. | Average per run | Minimal | Usual | |---|---|---| | Cost whole conversation | $0.84 | $1.01 +20% | | Cost after receiving the report | $0.37 | $0.55 +49% | | Tokens | 634K | 800K | In other words, the main agent with this model did not take delegate summaries at face value even without special instructions. It checked for itself. What the usual instructions changed was not the catch rate. What changed was "how work was requested" and "how thorough it was after checking," and that raised the cost by about 20%. Note that in the usual condition, I added the list of roles and the rules at the same time, and also used a short reply format from delegates with one line listing what was omitted . I cannot separate which of these had the effect. The instructions include the sentence "accept results through checks that can fail." The behavior of adding checks after verifying may come from this sentence. But I did not measure this separately either. The full instructions are in the appendix at the end. One more point: what got lost in summaries. In my setup, I ask delegates to "write one line at the end of the reply listing what was omitted." I also added a mechanism that automatically stops replies missing this line. But for delegates the main agent called directly, it stopped very few, such as 2/376 implementation replies. Both forgot the line at the end of a long report. On the other hand, the main agent's instructions do not reach delegates that a delegate calls itself, the "grandchild delegates." There, the general-purpose agent built into Claude Code left out the line in 32 of 40 replies 80% and was stopped automatically. So written instructions worked only where they reached, and outside that, the mechanical stop took over. I have not yet measured whether any redo was caused by omitted information. First, the reviewer within the experiment. In the usual condition of the harder version, the reviewer was called 20 times. But every time, it was after the main agent had already found the error and had it fixed. The main agent read the diff and ran checks that exercise the whole flow integration tests , and found it first. So the number of runs where the reviewer was first to find the planted error was 0. Still, it raised findings outside the scoring. For example, "side effects caused by fixing exactly as the request text said." There were 5 medium-severity findings. However, the main agent noticed two of those kinds on its own in some runs without a reviewer. Only one kind appeared only in runs with a reviewer counted by keyword search; I did not read them all . So is the reviewer unnecessary in real work too? I checked the logs: I defined a "useful finding" as one that meets all four of these: Here are the results. The average was 5.8 findings per call. | How findings were handled | Share | |---|---| | Useful | 62% | | Nitpick | 9% | | Ignored | 4% | | Rejected with a reason | 4% | | Handling unclear | 13% | The remaining roughly 8% not in the table is not broken down here. In 93% of calls, something was fixed or reported to me as a result of the findings. In 71% of calls, there was at least one useful finding of medium severity or higher. Most severities were assigned by the reviewer itself. The rate drops with repeated rounds: 78% in round 1, 64% in round 2 and later, and 50% in round 3 and later. Even so, in round 3, half the calls had a finding of medium or higher. The effect differed by task type. | Task type | Calls | Calls with a useful medium+ finding | Nitpick share | |---|---|---|---| | Code | 81 | 69% | 10% | | Documents | 19 | 45% | 16% | | Instructions for AI | 14 | 86% | 4% | | Design and planning | 8 | 94% | 3% | | Analysis | 5 | 100% | 0% | Document review leaned most toward nitpicks. Design and instruction reviews produced many serious findings. By content, "defects in the changed part" were the most common at 43%. Gaps in documents, pre-existing bugs, and design or scope problems were 8% each. "Style preferences" were 5%, and "suggestions for unneeded features" were 1%. I also checked how reliable the classification was, in two ways: Here is one real example. In a rework of the ToDo feature, the reviewer reported, at low severity, that "items that cannot be read silently disappear on the next save." The main agent adopted this 4 minutes later. But it told me, "No more holes that lose data have been found." That was a wrong explanation. It appears to have been fixed later, but I have not confirmed this with a check. The experiment and the field giving different results is not a contradiction. The errors planted in the experiment were the kind the main agent can confirm by comparing the request, the diff, or the checks at hand. Real work has many things that are hard to see with such local checks: factual errors in text, holes in design, pre-existing bugs, and so on. There are also candidate reasons why my results differ from the post mentioned at the start. I require reproduction with evidence. The main agent triages. And the logs include changes to products that are published for other people to use, plus reviews of designs and instructions. There are limits, though. The real logs have no comparison group. I cannot tell whether the main agent would have noticed later on its own without the reviewer. Also, most finding severities were assigned by the reviewer itself. I also tested these two: I picked one past implementation and redid each 3 times. Note: Let me say this first. This is for reference only: one case, 3 runs each. Also, the "list of correct defects" used for scoring was built from the careful review done at the time. Please keep in mind that this yardstick favors that setting. The subject was a feature that shows greetings and hints on an empty chat screen. At the time, it was implemented at the medium setting and reviewed at the careful setting. The review raised 16 findings 2 high, 4 medium, 10 low , and 14 of them were fixed later. There was also one defect that the first review missed and the second found: in a conversation started from a ToDo item, the greeting kept showing throughout the work. I went back to the starting point of that time and tried two things. 1 Built 3 times with only a careful-setting implementation delegate, no review Of 7 serious defects 2 high, 4 medium, plus the missed one , the runs built in the same defects 6, 6, and 5 times. One run created 2 new medium defects. In this one case, careful implementation did not replace review. 2 Had a medium-setting reviewer read the original change 3 times Of the 6 high and medium findings from that time, it found 5, 5, and 4. It missed the missed one in all 3 runs. It also found real defects not on the list. The amount found was close, but severities were rated lower. In all 3 runs it marked nothing as "high," and in one run it rated the two original high findings as "for reference" and "low." Underrating severity risks the main agent overlooking things, as in the ToDo example earlier. I noticed one more thing. Places I named in the request text as "look especially here" were easier to find. Two of the places I did not name were missed in all 3 runs. Part of review's effect seems to come from the request text pointing to where to look. During this redo experiment, I made one mistake. My system has a feature that automatically attaches the spec to each request to a delegate. But while I wanted the work done as of a past point in time, I was passing the "current" spec. The current spec includes fixes made after the original review such as "don't show it after the screen is cleared" . In other words, the answer leaked. It came to light when the first reviewer flagged exactly that point as a "spec violation." I fixed it to use the spec in the specified workspace, then reran. When you have an AI reproduce work as of a past point, or run it in another location, code alone is not enough. You also need to align the point in time of the surrounding context and specs that are passed automatically. Reading this far, you might think, "the more checks and reviews, the safer." But there was a period before when tests and checks grew so much that no work got done. As a last resort back then, I set this rule: Add checks only for defects that were actually reproduced, or for places the history shows are fragile My feeling at the time was that when there is something it can verify, the AI seems to want to add checks even when not asked. During this investigation too, there was a moment when I stopped the main AI from trying to run even the old system through the experiment. In theory, "deciding what to do" and "deciding what not to do" are the same thing. From here on, this is my own view. In a human lab, people are short-handed, so "not doing" gets decided naturally. AI is cheap to run, so left alone, I think it tends to drift toward "doing everything." That is why, in an AI setup, drawing the line on "not doing" becomes work the human has to take on actively. I think the post from the start, and the fact that review worked differently by task type, are both about the same allocation: where to spend, and where to decide not to. In this investigation, the AI did all the hands-on work: counting the logs, building and running the experiment harness, checking the design, and verifying the original sources. The planted-error experiment ran 317 times including redos. Because they ran in parallel, it took about 1 hour 50 minutes in total about 12 hours if added up one by one . Meanwhile, what I, the human, did was mainly this: In the "AI-only lab" paper from the start, the authors wrote this in their conclusion: the human role shifts toward deciding what to investigate and judging the value of the AI's discoveries. The process of making this article was itself an example of that. There is one difference, though. In that paper, the job of "choosing topics and rejecting proposals" was also given to the AI "professor." In my setup, that role is not given to the AI; the human does it. That said, I don't think it is a question of which is right. The paper's experiment spent about 30 hours on the proposal-selection stage alone, advancing research with AI only while running equipment. If the design has no human nearby, I suppose the topic-choosing role ends up inside the AI too. My work is work where I am nearby and can set the goal. I think the best split depends on the work. My logs showed that too. The reviewer's effect differed by task type: document review leaned toward nitpicks, while design and instruction reviews produced many serious findings. The planted-error experiment is also a result on a small program made for the experiment. Different work may give a different answer. So how many roles to split into, and whether to have a checking role, is not something you decide once and finish. I think it is something you keep checking against your own work logs. And deciding what to check is, after all, a human's job. Whether you look at the list of what the human did and laugh, "you barely did anything," or take it as "so this is the important work," I leave to you. Shown here are the role split and instructions used as "usual" in the experiment. Both are the defaults when you turn on the feature that splits and delegates work by role off by default in LAISORA, a VS Code extension I make as of November 2026 . The "write one line at the end of the reply listing what was omitted" rule and the mechanism that stops replies missing that line, mentioned in the body, are also in this extension. You can see its contents on the VS Code Marketplace https://marketplace.visualstudio.com/items?itemName=az-llc.laisora . What these instructions changed is as described in the experiments above. If you know a better way to write them, I would be glad to hear it in the comments. | Role | Job | Model | Reasoning depth | |---|---|---|---| | Implementation | Implementation and mechanical edits | Opus | low / medium / high | | Research | Read-only search and fact checking | Sonnet | medium | | Review | Independently judges finished changes or designs. Returns findings, does not edit | Opus | medium / high | Reasoning depth is Claude Code's effort setting. Each role has a set of depths it may use. In the "minimal" condition, the main agent got only this one line: Delegate edits to the worker agent laisora:worker-opus . In the "usual" condition, it got the role list above and the instructions below. In the instructions, medium and high are the role's default depth and the depth one step above it.