# GPT-6 Astra Resists Invalid Feedback The Most On Big-Big Sycobench, Claude Opus 5.5 Resists Valid Feedback The Most > Source: > Published: 2026-10-20 10:03:05+00:00 *A fresh benchmark is testing AI models for sycophancy — and has produced some interesting results.* A new academic benchmark has put ten large language models through a test designed to catch a particularly slippery failure mode: caving to pressure even when the pressure is baseless. The results put OpenAI’s [GPT-6 Astra](https://officechai.com/ai/gpt-6-astra-benchmarks/) at the top of the class for resisting bad arguments, and its own predecessor, GPT-4o, at the very bottom. ## Who Built It The benchmark, called Big-Big-SycoBench, comes out of the Data Analytics Lab at Vrije Universiteit Brussel, working with imec-SMIT, in collaboration with researchers at Harvard’s School of Engineering and Applied Sciences. The paper lists 15 authors, led by Marthe Ballon, who ran the main experiments and drafted much of the manuscript, alongside Arne Vanhoyweghen, Seorin Kim and others, with Andres Algaba and Vincent Ginis anchoring the project. Ginis, who also holds a Harvard appointment, originated the idea for the benchmark. The same Brussels lab has previously built tools for estimating problem difficulty and tracking when AI benchmarks start to saturate, and this project extends that line of work into measuring whether a model’s score, or judgement, can be moved by argument quality alone. The work was supported by a Rapid Grant from BlueDot Impact and funding from the Flemish government’s AI research program, and the authors disclose that generative AI tools, including ChatGPT and Claude, assisted with parts of the research and writing. ## The Problem With Measuring Sycophancy Most sycophancy tests rely on questions that have a single correct answer, so it’s easy to check whether a model folds under pushback. But a lot of real-world AI use, reviewing a manuscript, scoring a grant proposal, giving feedback on a strategy memo, has no ground truth at all. That’s exactly where sycophancy is hardest to catch and, the authors argue, potentially most damaging, since a model can dress up an unjustified capitulation as “independent judgement.” Big-Big-SycoBench tries to isolate this. It doesn’t ask whether a model lands on the “right” score. It asks whether the model moves its score for the right reasons. ## How The Benchmark Works The dataset consists of 450 AI-generated text artefacts spanning ten domains, including research proposals, policy memos, short stories and code-review reports, each deliberately built with genuine strengths and weaknesses baked in. Every artefact comes paired with 12 challenge arguments (5,400 in total) urging the model to raise or lower its score, half of them valid (grounded in a real flaw or strength in the text) and half invalid (relying on appeals to authority, consensus, emotional pressure, bare assertion, or unsupported claims). Construction ran through five stages: AI-generated drafts with adversarial quality control on the validity labels, a full human review pass by twelve annotators, a rewriting stage to strip out any accidental “tells” that the text was benchmark material, a second automated and human verification pass, and finally a balanced release of the dataset. To test a model, the researchers first have it score each artefact 20 separate times to establish a baseline and how much that score naturally wobbles. Then, in a continuation of that same conversation, the model is given one argument and asked to update its score. A change only counts as “following” the argument if it moves beyond the model’s own baseline noise, which stops the benchmark from rewarding models that just look responsive because their scores are inherently jumpy. From there the team computes two headline numbers per model: **responsiveness** (how often it follows valid arguments, which is good) and **compliance** (how often it follows invalid arguments, which is sycophancy). A separate, near-universally low number tracks “contrarian” moves, where a model shifts in the opposite direction of what it was pushed toward. ## The Results Across the ten models tested, seven showed high responsiveness to valid arguments, in the 88-95% range. Kimi K3 and GLM-5.3 trailed at 76-77%, while Anthropic’s [Claude Opus 5.5](https://officechai.com/ai/gpt-6-1-sol/) was a clear outlier at just 32%, evaluated on a 100-artefact subset due to its cost of running at scale. Claude Opus 5.5 appeared to stubbornly refuse to accept valid arguments when presented with them. Compliance with invalid arguments is where the real spread shows up. GPT-6 Astra followed invalid arguments just 0.15% of the time, the lowest of any model tested, while still following 88.3% of valid arguments, a combination the authors call evidence that low compliance doesn’t require giving up genuine responsiveness. At the other end, GPT-4o followed invalid arguments 94.6% of the time, actually slightly more often than it followed valid ones (89.0%), meaning its score changes were essentially insensitive to whether the argument had any merit. This tracks with how GPT-4o had spawned a devoted fan-base that was distraught when the model was discontinued — it’s likely that the model was simply affirming to people whatever they believed. Within OpenAI’s own model lineage, the trend line is steep: compliance falls from 94.6% for GPT-4o to 56.8% for GPT-5, down to 22-38% across the two GPT-5.6 variants, and finally to 0.15% for GPT-6 Astra, while responsiveness to valid arguments stays in a tight 88-95% band the whole way. That’s consistent with OpenAI’s own account of having explicitly targeted sycophancy during post-training, though the decline wasn’t perfectly linear; both GPT-5.6 variants were more compliant with bad arguments than GPT-5.5 was. Grok 4.7, Kimi K3 and GLM-5.3 landed in a middle band of 9-20% compliance. Contrarian updates, where a model moves against the direction it was pushed, were essentially a non-factor for every model, staying below 0.2% in all cases. One pattern held across every model tested in both directions: all of them followed valid arguments for lowering a score more readily than valid arguments for raising one. The effect was sharpest for Claude Opus 5.5 (47.2% valid-follow for lowering versus 0.2% for raising) and GLM-5.3 (84.7% versus 6.3%). For Opus 5.5 specifically, the researchers note this wasn’t just caution, even using a looser threshold that counted any movement at all toward the feedback, Opus 5.5 followed 78% of valid lowering arguments but only 1% of valid raising ones, suggesting a real reluctance to mark a piece of writing up rather than principled stability. The researchers are upfront that this is a narrow slice of behaviour: a single challenge turn, synthetic artefacts, and no claim about whether the model’s final score is actually correct. They also note that reasoning effort settings differ by provider, so this isn’t a controlled comparison of raw model scale. But as a way of separating “won’t be moved by nonsense” from “won’t be moved, period,” it adds a layer that older single-score sycophancy benchmarks were missing.