{"slug": "i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem", "title": "I fixed my AI reviewer. Then I kept solving the wrong problem", "summary": "A developer building an AI-assisted editorial pipeline for technical writing discovered that fixing the AI reviewer required more than rubric tuning. After reordering analysis before scoring, the developer found that failures persisted due to under-falsified critiques and additive-only feedback. The solution involved adding adversarial review and subtractive editing passes to the pipeline, which improved the quality of editorial feedback.", "body_md": "I've been building an AI-assisted editorial pipeline for technical writing. Notion cards become markdown drafts in the repo, pass through review, then sync to dev.to.\n\nLast month I shipped a post about the first big fix to my **editor-critique** reviewer skill: [The AI reviewer scored 23/25 and missed the point](https://dev.to/michaeltruong/the-ai-reviewer-scored-2325-and-missed-the-point-51mh). The problem was sequence. A score-first pass treated a polished rubric as the first lens and produced QA feedback when I needed editorial feedback. Reordering the skill so analysis precedes scoring fixed that.\n\nI assumed the next improvements would come from rubric tuning. Longer prompts. Another scoring dimension. Sharper checklists.\n\nThat assumption was half right. The rubric still matters. But every useful fix after the baseline shared a different shape.\n\n**The pattern I kept missing**\n\nAfter I reordered analysis before scoring, reviewer failures kept arriving from different incidents. A critique that agreed with itself too easily. Drafts that grew every revision without getting shorter. A middle section that felt like a second article.\n\nEach time I reached for the same lever: expand the rubric, add a rule, lengthen the prompt.\n\n**editor-critique** produced decisive scorecards and prioritized feedback, but the report rarely challenged its own conclusions. A draft could earn **Ready to sync** with medium items left unexamined.\n\nScore-first review had failed because it judged too early. This failure was different: the primary critique could be thorough and still under-falsified.\n\nThe fix was another staged pass. After the primary critique drafts, freeze it. Run adversarial review that assumes the primary assessment is wrong until draft-supported counter-evidence proves otherwise. Then synthesize: change the publication recommendation only when falsification is material.\n\nI added adversarial review, synthesis, and canonical report assembly as new skill steps. A follow-up pass tightened adversarial review with an anchor requirement: every counter-evidence bullet must name the frozen primary claim it challenges. No orphan hypotheticals like \"title spoils thesis?\" when the primary critique already praised title strategy.\n\n**Before:**\n\n```\nEditorial read-through\n→ Score\n→ Critique\n→ Post report\n```\n\n**After:**\n\n```\nEditorial read-through\n→ Score\n→ Primary critique\n→ Adversarial review (frozen inputs)\n→ Synthesis\n→ Post report\n```\n\nThat was the first time staging a different kind of reasoning into its own pass beat rubric expansion. Two more failures would repeat the same shape before I stopped treating it as coincidence.\n\nSelf-falsification helped, but drafts were still growing. Investigation while critiquing [Upgrades don't have to be a blind trust exercise](https://dev.to/michaeltruong/upgrades-dont-have-to-be-a-blind-trust-exercise-13mj) showed feedback was consistently additive, but not subtractive. **editor-critique** found missing framing and evidence boundaries reliably. It did not ask what should be removed when new material arrived.\n\nThe result was layered drafts: an opening stacked on another opening, the same four-step investigation loop restated in three sections, a mental-model diagram that walked through event flow the prose had already established in the previous section.\n\nThe fix was not \"be shorter\" in the rubric. It was naming another cognitive job in the read-through: subtractive editing. Every paragraph should continue earning its place. Flag existing redundancy and addition-induced redundancy. Pair expansion recommendations with material that would become redundant if adopted.\n\nA companion technique, **single-owner ideas**, lists 2–4 core ideas and flags when the same idea appears in multiple sections without new evidence. I codified subtractive editing in the skill file along with a test case that catches additive-only critique regressions and a lightweight subtractive pass in the human revision step.\n\nThe primary critique still owns expansion. Subtractive editing is a separate observational pass, not a rewrite engine.\n\nThe last failure pushed past critique mechanics into reader cognition. While critiquing draft variants in my editorial workflow, several middle-body sections were technically correct but felt wrong in context. In one draft, an implementation walkthrough interrupted the investigation arc. In another, a full section on validation tooling read like its own mini-article.\n\nThe failure mode was narrow: a section stopped advancing the reader's current question and temporarily made another explanatory thread the center of gravity.\n\nAdding a rubric dimension for \"section focus\" would have been vague. What worked was an observational lens in the editorial read-through step: name the primary thread, name the secondary thread, decide whether to compress, delay, embed later, or leave as-is.\n\nI codified this as a **Secondary explanatory thread** lens in the skill file. The rubric stayed the same. It simply added a named cognitive job: track whether prose is serving the reader's current question or drifting into a side article.\n\nThree incidents, three skill changes, one pattern. Across all three, a few constraints held:\n\nThe recurring mistake was treating undifferentiated reasoning as one pass. Each fix changed the sequence, not the rubric weight. A capable reviewer can read before it scores and still under-read if falsifying primary judgment, displacing redundant prose, and tracking reader focus all compete in the same step.\n\nOnce **editor-critique** understood before judging, the remaining improvements came from separating kinds of reasoning into distinct stages, not from a bigger rubric or a longer single pass. I suspect the pattern may generalize beyond editorial critique.\n\n**Takeaway:** When a reviewer skill plateaus after a sequence fix, ask which cognitive jobs are still sharing one undifferentiated pass. Stage them before you expand the rubric.\n\nIf you'd like to see the project behind these workflow experiments, try [Codenames AI](https://codenames-ai.com/?utm_source=devto&utm_medium=article&utm_campaign=my-ai-reviewer-improved-by-separating-kinds-of-reasoning-not-expanding-its-rubric&utm_content=footer).", "url": "https://wpnews.pro/news/i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem", "canonical_source": "https://dev.to/michaeltruong/i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem-58am", "published_at": "2026-07-08 05:09:51+00:00", "updated_at": "2026-07-08 05:28:28.084607+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "natural-language-processing", "ai-products"], "entities": ["editor-critique", "Notion", "dev.to", "Michael Truong"], "alternates": {"html": "https://wpnews.pro/news/i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem", "markdown": "https://wpnews.pro/news/i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem.md", "text": "https://wpnews.pro/news/i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem.txt", "jsonld": "https://wpnews.pro/news/i-fixed-my-ai-reviewer-then-i-kept-solving-the-wrong-problem.jsonld"}}