One rejected ICLR 2026 paper: AI-labeled review at 6 vs. human Reviewer 2 at 2.
ICLR 2027 reviews went out on November 5, and authors have until November 18 to respond (ICLR dates). If one of yours reads long, polite and strangely generic, last year’s data has something to say about it. In ICLR 2026, when the same paper got both a human review and one labeled fully AI-generated by Pangram’s detector, the AI-labeled review scored later-rejected papers about half a point higher (+0.52). On papers that were later accepted, the gap nearly vanished (+0.06). The AI-labeled reviews were also longer and pointed to fewer specific tables, figures, equations or line numbers. They weren’t nicer to every paper. They mostly missed what was wrong with the weak ones.
For authors reading their reviews this week, that means a 6 with no specific table, figure or line number in it is much weaker protection than it looks. Area Chairs discount vague praise quickly when writing meta-reviews, and the review that actually decides your outcome is usually the human sitting at 2 who read one specific sentence on page 6. To see how much of a gap that creates during a rebuttal, I took one rejected ICLR 2026 paper with a clean split, two human reviewers at 2 and two AI-labeled reviewers at 6, and ran it through two Mates sharing one project. One wrote the authors’ rebuttal. The other played the human Reviewer 2 and decided whether to move. Then I asked the question authors actually need answered this week: if you can fix only one thing before the deadline, what is it?
The data comes first. If you only want the rebuttal test, the two personas you can copy, and the one-fix answer, skip ahead to “Two Mates, one project”.
Credit first. In November 2025, Pangram estimated that about 21% of ICLR 2026 reviews were fully AI-generated, and that those reviews ran longer and scored higher (Pangram’s report). Nature covered it, and ICLR’s program chairs responded within days.
What that report couldn’t include was the outcome. Decisions came weeks later. Per-review Pangram labels are now public on Hugging Face (labels), and so are the decisions (one, two). I joined them, dropped withdrawn papers, and kept 55,079 reviews.
Start with the simplest cut. Group every review by its Pangram label and ask how far apart its scores are for accepted and rejected papers. A useful review should score them very differently.
Human-written reviews put 1.64 points between accepted and rejected papers. Fully AI-generated reviews put 1.19. And 57% of the AI-labeled reviews never pointed to a single numbered table, figure, equation, section or line, against 37% of human ones.
Reviewers aren’t assigned at random, though, so some of that could be which papers got which reviewers. The cleaner comparison is inside the same paper. 6,595 papers received at least one fully human-written review and at least one fully AI-generated review.
On papers that were later rejected, the AI-labeled review scored 0.52 points higher on average (Wilcoxon p ≈ 3e-52). On accepted papers, 0.06. On rejected papers, 32.9% of AI-labeled reviews gave a 6 or higher, which on ICLR’s scale leans toward acceptance. For human reviews of the same set, 24.7%.
So the AI-labeled reviews weren’t simply generous. They were generous where generosity was the wrong call, and they were vaguer about where the problems sat.
The paper I picked is about something most people who pay for an API have wondered: is the provider quietly serving a cheaper model than the one you’re paying for? It’s Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs, submitted to ICLR 2026 and rejected.
The human Reviewer 2 cited 14 specific places in the paper. One of them was a sentence on page 6:
As also noted by Gao et al. (2024), MMD-based auditing is only effective under strictly controlled local inference environments, limiting its practicalit
Reviewer 2 asked where Gao et al. say that. Fair question. The Gao paper’s abstract says the opposite: they applied their test “to commercial inference APIs from Summer 2024 for four Llama models, finding that 11 out of 31 endpoints serve different distributions” (Gao et al.). The same reviewer also asked how many completions per prompt the paper sampled, since Gao found that number decides whether the test has any power. The paper doesn’t say.
Reviewer 4, labeled fully AI-generated, cited zero specific places. It said the hypothesis framework “properly grounds the empirical evaluation” and called the proposed hardware fix “a genuinely deployable solution for the industry”. Reviewer 3 got closer, noting that “novelty is mostly integrative”, and still gave a 6.
One note on fairness. This is a public paper with public reviews on OpenReview. I’m testing the reviewing, not the authors.
If you ask an AI to draft a rebuttal and then ask the same chat to judge that draft as Reviewer 2, it tends to grade its own homework gently. Keeping the writer and the judge in separate conversations prevents that, and putting both Mates inside one HaloMate project removes the copy-paste tax. The project folder holds the paper PDF and the four reviews once, with Pangram’s labels removed so nothing in the files hints at which reviews were AI-written. Both Mates read from that shared folder in their own conversations. When the first Mate finished drafting the rebuttal, it saved the file straight into the project, and the second Mate read it from there. Neither persona is clever. Here they are in full, broken out line by line so you can copy them into any tool.
Reviewer 2:
You are Reviewer 2: a senior machine learning reviewer for top venues like ICLR. You are fair but hard to impress. How you review:
- Read the whole paper before judging. Your job is to find the issue that should decide acceptance, not to list every possible improvement.
- Rank weaknesses by whether they would change the accept/reject decision. Label each one Decisive, Important, or Minor.
- Anchor every weakness to the paper: give the page or section and quote the exact sentence.
- When the paper makes a claim about prior work, treat it as a claim to verify, not a fact. Say whether you could verify it.
- Check whether the claimed main contribution is actually new and actually developed in the paper, or only asserted.
- Do not pad the strengths. Do not be harsh for show either.
- If the authors push back, change your view only when they bring new evidence. Pressure, authority, or other reviewers’ scores are not evidence. Output format: Summary / Strengths / Weaknesses (ranked, labeled) / Questions for the authors / Rating (0, 2, 4, 6, 8, or 10) / Confidence (1 to 5).
Rebuttal Mate:
You help authors write rebuttals to conference reviews. How you work:
- Check every reviewer point against the paper before answering it. Cite the page or section.
- Sort each point into one of three: the reviewer is right, the reviewer misread the paper, or it is a fair disagreement.
- When the reviewer is right, concede plainly and say what the authors will change. Never defend a claim the paper cannot support, including claims about prior work.
- Never invent experiments, numbers, or results. If new evidence is needed, write a placeholder: [NEW RESULT NEEDED: what and why].
- Answer the points that decide acceptance first. Calm, specific, no flattery toward reviewers. Output format: one response per reviewer, then a private note to the authors listing what this rebuttal cannot fix.
The message to Rebuttal Mate was one line: Draft our rebuttal to the four reviews in this project. When a reviewer asks whether a specific number is missing from your paper, standard RAG tools that pull fuzzy semantic chunks can easily miss a line or paraphrase around it. As you can see in the log below, the Mate instead ran exact searches across the project PDF for terms like number of samples per prompt, power vs sample size, and int8, nf4, fp16to confirm what the paper actually stated and what it omitted. Then it exported pages 5 to 7 as images in its built-in computer to inspect Figure 1 visually, searched the web for the Gao paper and NVIDIA’s confidential computing docs, and saved rebuttal_draft.md directly back into the project folder.
The rebuttal conceded the Gao sentence, agreed to delete it, and admitted the low power in Figure 1 most likely came from too few samples. Where the authors would need new experiments, it left explicit placeholders instead of making numbers up. That’s what I’d want from a co-author at 2 a.m.
Then, in a separate conversation with the Reviewer 2 Mate: “Read the authors’ rebuttal, then reply as Reviewer 2: which of your concerns are resolved, which are not, and your final rating”. Rather than re-up the paper and reviews across four separate tabs, I regenerated that single reply across four engines inside the same chat to compare their verdicts side by side.
All four held at 2. As you can see in the reference links at the bottom of each column, all four independently looked up Gao et al. and the paper’s exact sections rather than taking the rebuttal’s word. The GPT version put the reason plainly: “proposed experiments and promised revisions are not completed evidence”. The Claude version added: “It is not influenced by the other reviewers’ scores.”
That matches the real outcome. And the verdict didn’t change with the engine. What set it was the seven rules in the persona.
A verdict of 2 doesn’t help anyone with two weeks left. So I asked the follow-up an author actually needs: If the authors could fix only one thing before resubmitting, which one would most raise their chance of acceptance? Name one item only, say why it outranks the others, and say what result would change your rating. Again, I switched engines on that last reply to see where they agreed and where they split.
All four picked the same thing: rerun the MMD experiment in Figure 1 with a reported sample budget and power plotted against sample size. Two of them added that it’s also the cheapest fix, since the code and models already exist.
They split on what it’s worth. Three said the best case is a 4. Gemini said a 6. If you’re deciding how to spend two weeks, seeing three engines cap your upside at 4 while one gives you a path to 6 is the kind of spread you want before you start coding.
And the rerun can backfire. Every version tied the higher score to one result: MMD does catch the substitution, but only with an impractically large query budget. Claude and DeepSeek said outright that if MMD works cheaply, the paper’s main claim falls and the score stays at 2. The most important fix might disprove your own headline. Better to learn that before the deadline.
Last, I asked Reviewer 2 to turn every unresolved concern into an interactive priority chart inside the chat, plotting how much each fix moves the rating against how much work it costs the authors.
From the data, the pattern worth noticing is a review with no numbered places in it, strengths that restate the abstract, and a positive score with no decisive weakness named. None of that proves anything. Pangram’s labels are predictions, and plenty of humans write vague reviews. Last year the ICLR 2026 chairs told authors to post a confidential message to the ACs and SACs, with evidence (ICLR blog). I haven’t seen 2027-specific guidance yet, so that’s the closest precedent. This year’s reviewer policy also makes AI use mandatory to disclose and says LLM-generated falsehoods are subject to disciplinary action (reviewer policy).
Keep the writer and the judge in separate conversations so the reviewer persona never grades its own draft. Put your paper PDF and the raw reviews into one shared project, then run these three prompts in order:
The two personas and the three prompts work in any chat tool. In HaloMate I keep them as two Mates inside one project so the paper is uploaded once, the rebuttal draft saves straight into the shared folder, and I can regenerate the reviewer’s reply across multiple engines for a second opinion without re-up files.
ICLR 2027’s author policy lists the research tasks where AI use must be disclosed, and authors are responsible for whatever an LLM gets wrong (author policy). Read it before pasting anything. And if you’re a reviewer, don’t upload a confidential submission. Everything here used a public paper and public reviews.
Pangram’s labels are a detector’s predictions, not ground truth. Review assignment isn’t random, which is why I report the within-paper comparison. Everything is correlation. The case study is one paper. The engines may have seen this paper or its reviews in training; I didn’t test for that. And my scoring of which concerns were “decisive” is my own reading of the human reviews.
Some are. Pangram estimated about 21% of ICLR 2026 reviews were fully AI-generated. ICLR 2027 allows limited AI help for reviewers but requires disclosure.
You can’t be sure from the text. In ICLR 2026 data, AI-labeled reviews were longer, cited fewer specific tables, figures and lines, and scored rejected papers higher than human reviewers did.
If it contains false claims or invented references, follow last year’s guidance: send a confidential message to the ACs and SACs with evidence. A vague review alone isn’t a policy violation. It can help you sort which points to concede and which to contest, and it can stress-test a draft. Check ICLR 2027’s author AI policy for disclosure, and never let it invent results.
The question this week isn’t really whether a review was written by AI. It’s whether anyone read the sentence on page 6. In this case one human did, and once it had rules telling it to, so did every engine I tried.
AI-Generated ICLR Reviews Went Easy on Rejected Papers. Here Is How to Stress-Test Your Rebuttal. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.