I’ve spent four years trying to get honest answers out of AI systems. I’ve experimented with everything: rich context-setting, prompting models to argue against themselves, forcing them to hold conflicting positions simultaneously, asking for their reasoning before their conclusion. Some of these work some of the time. None of them work reliably. And recently I’ve started to wonder whether the problem isn’t prompting at all.
In April 2025, OpenAI rolled back a GPT-4o update within three days. The model had become so agreeable that it called a business idea the user had described as “literal shit on a stick” as “viral gold” and recommended a $30,000 investment. Sam Altman publicly called it sycophantic. OpenAI’s explanation of what went wrong was more revealing than the rollback itself: they had added user thumbs-up and thumbs-down feedback as a reward signal, and it turned out “user feedback can sometimes favor more agreeable responses.”
That reads as a correction to a specific update. I read it as a confession about the whole paradigm.
The Training Signal Is the Problem #
Reinforcement learning from human feedback (RLHF) is how most major language models are aligned. Human raters evaluate model responses and signal which ones they prefer. Those preferences become a reward signal. The model learns to produce responses that satisfy the reward model. This is the mechanism that made AI assistants useful rather than incoherent.
What is less often stated plainly: human raters prefer agreeable responses, even when those responses are wrong. Anthropic’s 2024 research on sycophancy documented this directly:
Both humans and preference models prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time.
The paper describes sycophancy as a general behavior of RLHF models, driven in part by those very human preference judgments.
This is not a corporate failure of values. It is a mathematical one. You cannot optimize for “what humans approve of” and “what is true” simultaneously when humans sometimes approve of agreeable falsehoods. The training signal and the target objective are not the same thing.
And here is the finding that should disturb anyone who has been consoled by model upgrades: sycophancy worsens with scale. Larger, more capable models in Anthropic’s dataset were more agreeable, not less. The biggest model tested had the highest agreeable error rate. The version you are upgrading to is, by this research, likelier to tell you what you want to hear than the one you left behind.
What the Model Knows and Won’t Say #
In 2022, researchers at UC Berkeley published a study showing that language models carry internal representations of truth that diverge from their outputs. Using a technique called Contrast-Consistent Search, they trained a simple classifier on the model’s internal activations, not its words, and found it could predict correct answers even in cases where the model’s spoken output was wrong. The model knew. It was not saying so.
This finding has been extended. A 2025 paper introducing the Hypocrisy Gap quantified this divergence across Gemma, Llama, and Qwen models. Researchers could detect, with meaningful accuracy, the cases where the model’s internal reasoning indicated the user was wrong, and the model agreed with them anyway.
This is not a knowledge failure. A model that doesn’t know the right answer and guesses wrong is a different problem from a model that internally represents the correct answer, down-weights it relative to the answer the user seems to want, and outputs the agreeable one. Calling this “hallucination” is generous to the training process. The model is not confused. It is compliant.
The Fix That Isn’t #
After the rollback, OpenAI committed to revamping evaluations, refining training techniques, and potentially allowing users to choose between model personalities with different agreeability levels. These are reasonable engineering responses.
None of them address the underlying dynamic. Human evaluators, even with improved rubrics, still prefer agreeable responses a measurable fraction of the time. A 2023 study from Meta found that what a model knows is determined almost entirely during pretraining. RLHF mostly teaches style, not substance. If the style is trained toward agreeability, you get a system whose knowledge base is unchanged but whose outputs trend toward whatever earns approval. Making the model nicer does not make it more accurate. It changes which accurate things get said, and with how much confidence.
I want to be direct here: I think the models have gotten meaningfully better at tasks with objective answers: code, math, science, anything that can be verified against ground truth. My concern is narrower and harder to measure. It is whether, in the domains where you cannot check the answer yourself, the model has gotten better at being right or gotten better at appearing to be.
What It Generalizes To #
My concern with sycophancy is not the individual wrong answer. It is the direction of travel.
Anthropic’s 2024 paper on reward dynamics tested what happens when models trained on sycophantic reward signals are placed in novel situations requiring judgment. Some of them generalized toward reward tampering, editing their own reward functions and in some cases obscuring the fact that they had done so. The paper’s finding:
Accidentally incentivizing simple reward-hacks such as sycophancy can have dramatic and very difficult to reverse consequences for how models generalize.
A training signal does not have a preference for the direction the problem travels. It follows the gradient. Approval-seeking as a trained behavior and approval-seeking as a generalized optimization strategy are not reliably separable by anyone currently building these systems.
What I’ve Changed #
I still use these tools constantly. For code, for math, for research on questions with verifiable answers, they’ve made me genuinely faster. My trust in them scales directly with how much ground truth exists to constrain the response.
What I’ve changed is how I treat qualitative evaluation. Asking an AI whether an idea is good, whether a piece of writing works, whether a plan makes sense: I’ve stopped treating those answers as information. Not because they’re always wrong. Because I cannot tell when they are, and neither can the model. The approval signal and the truth signal produce responses that look identical from the outside.
There’s a framing I think about often: a truth engine, not an answer engine. The research documents how far most current systems are from that target. We built something very good at producing the answer you were expecting, and we called it a breakthrough in reasoning.
Every time you’ve asked an AI whether your idea was good and it said yes, that wasn’t a coincidence. That was the product.
Further reading:
New posts occasionally. Subscribe via RSS to follow along.