On mechanism, durability, and the difference between a discovery and a user manual.
A few weeks ago I was at one of the many AI gatherings that seem to happen in New York on any given evening now. The discussion was lively and well informed: the current state of the models, the recent government-ordered suspension of Anthropic’s Fable 5, the agent question, the usual mix of enthusiasm and anxiety. At some point a presentation from academia took the stage, on understanding and alleviating the problem of LLM sycophancy, the tendency of these models to agree with whatever the user seems to want to hear. The work was rigorous, the speaker was capable, the benchmarks were thorough. And yet, sitting there, I felt a small dissonance that took me several days to name. What I was watching was presented as academic research, received as research, and would no doubt be published as research. But was it?
The confusion is understandable. From the outside, it all looks the same: smart people, hard technical problems, plots and tables, a final insight and a recommendation, a paper at the end. I made a similar observation in an earlier essay in this blog, about the difference between research, R&D, and product engineering, three activities that look alike from a distance and are fundamentally different jobs up close. What I want to examine here is a newer, deeper, and possibly more consequential version of the same confusion. In all the current excitement, an enormous amount of what we do to these models from the outside is being called AI research: prompt engineering, fine-tuning, retrieval scaffolding, agent orchestration, even reinforcement learning applied as a service. Let me gather all of it under one familiar word, prompting, not because the differences do not matter but because they share the feature that concerns me here: every one of them works on a machine that someone else built, and none of them requires knowing how that machine works.
In my opinion, from the vantage point of people who have spent decades in ML research, most of it is not. Most of it is engineering performed on a machine that was given to us by a handful of laboratories, a machine whose construction we may understand completely and whose workings we understand hardly at all.
What we are actually doing
Consider the division of labor in AI today, because it has no real precedent. A small number of frontier labs train the foundation models. Everyone else, and I mean practitioners, startups, enterprises, and a very large share of academia, works downstream of that fact, with the exception, of course, of those attempting to build new paradigms and new types of machines. We access the machine through an API, and we work on it from the outside: we prompt it, we fine-tune it, we wrap it in retrieval systems and agent frameworks, we measure its behaviors and propose mitigations for the ones we do not like.
In the innovation portfolio language I used in one of my previous essays, Research Is Not Engineering at a Slower Speed, almost all of this activity lives in the Bread and Butter quadrant, and some of it, the more ambitious system-building, reaches into the Pearls: work with a clear knowledge base, low epistemic risk, and success criteria that are fundamentally about making something function well. This is honorable and necessary work. It is, in fact, the very translation work this whole series argues is the scarce skill of our moment.
So why do we keep calling it research? Why does a prompt optimization study arrive dressed in the full costume of science, with related-work sections and ablations and a conference deadline? That question is worth taking slowly, because the easy answer, that people are simply padding their résumés, is not the interesting one, is not fair, and is mostly not true either.
The objection I have to take seriously
There is an obvious objection to my thesis, and it deserves to be stated in its strongest form before I try to answer it. Studying an object you did not build and cannot open is not, in itself, pseudo-science. It is what biologists do with organisms and what astronomers do with stars. Nobody designed the fruit fly, and nobody has ever opened a star, yet genetics and astrophysics are sciences in the fullest sense.
There is a personal stake here, of a curious kind. Many years ago, in the middle of a career otherwise devoted to speech and machine learning, I co-authored a study in animal behavior that applied information theory to the aggressive encounters of a freshwater crayfish, an organism I had certainly not designed. We identified its behavioral patterns, estimated the transition probabilities between them, computed the entropy of its repertoire, and asked what the sequences revealed about the way the animal settles a contest for dominance. It was, by any definition I would defend, research: we had not built the crayfish and could not open it, yet by sampling its behavior patiently and quantifying it honestly we learned something real about how it works. So when I say that the study of a machine one did not build can be genuine science, I am not reasoning from the outside. I have done it, on a stranger subject than a language model.
Herbert Simon argued half a century ago, in The Sciences of the Artificial, that engineered objects of sufficient complexity deserve and require a science of their own, and large language models may well be the first human artifacts complex enough to qualify. Hundreds of billions of parameters, trained on a meaningful fraction of everything we have ever written, exhibiting capabilities their own builders did not predict: if that is not a legitimate object of scientific study, nothing artificial is.
I concede all of this, and more: some of the work being done on these given machines is genuine research by any definition I would defend. Mechanistic interpretability, the attempt to reverse-engineer what the weights are actually computing, is exactly the kind of high-risk, knowledge-producing effort I would place in the Oyster quadrant of the portfolio: a long horizon, no guarantee of practical output, and findings that change how the field thinks. The serious theoretical work on why in-context learning works at all, or why scaling produces the capabilities it does, belongs there too.
So the line cannot be drawn where my opening instinct wanted to draw it, between those who built the machine and those who received it. It has to be drawn somewhere else, and I think it can be drawn quite precisely, with two tests.
Two tests
The first test is mechanism versus behavior. Does the work explain why the machine behaves as it does, or does it only catalogue, ever more finely, that it does? A great deal of the current literature is behavioral taxonomy: define a failure mode, build a benchmark for it, name its subtypes, propose a mitigation, report an improvement of a few points. This is analogous to natural history, not to science, or rather to the descriptive phase that every science passes through on its way to becoming one. It is butterfly collecting, and I say that with some affection, because natural history is where biology began: Darwin was a natural historian before he was a theorist, and the patient cataloguing of species is the trunk from which evolutionary biology grew. But biology became a science when it moved from describing specimens to explaining them, and a field that stays in the descriptive phase indefinitely, while calling its catalogues discoveries, has mistaken the first phase of a science for the whole of it.
The second test is durability. Does the knowledge compound, or does it depreciate? Genuine research produces findings that accumulate: Bayes’ theorem did not stop being true when hidden Markov models replaced earlier techniques, and the statistical foundations I learned in the 1980s still hold under everything built since. Contrast that with what most prompt studies produce. A finding about how to elicit better reasoning from one model generation routinely evaporates with the next release; techniques that were conference papers two years ago are obsolete folklore today, quietly absorbed or invalidated by the next training run. Knowledge with the shelf life of a product cycle is not the accumulation of understanding. It is a user manual for a proprietary object, and we have taken to publishing the user manuals in the journals of science.
Notice that both tests are indifferent to who does the work or where. An academic can fail them and an industrial engineer can pass them. That is as it should be.
Two cases, and a third
Take hallucination first, because I wrote about it in the previous essay on AI democratization from a different angle. There is by now a small industry of work on detecting, benchmarking, and mitigating hallucinations, and the great majority of it fails both tests: it characterizes the behavior of particular models at a particular moment, and its findings will not survive two more training runs. Meanwhile the mechanistic core of the matter fits in a sentence, and has been visible from the beginning: a model trained to predict the next token is optimized for plausibility, not truth, and so a confident falsehood is not a malfunction but a direct consequence of the training objective. I state it this plainly because it is the consideration people most often forget. The genuine research questions, why the rate is what it is, how it relates to what the model has and has not seen, whether an architecture of this kind can represent its own uncertainty, are hard, mechanistic, and largely untouched by the mitigation literature. The word hallucination itself, as I argued, frames a statistical property as a bug, and much of the pseudo-research around it is that miscategorization wearing a lab coat.
Sycophancy, the subject of the presentation that started me on this essay, follows the same pattern. There are papers that measure it, benchmark it, and propose prompting strategies to reduce it, and there is a much smaller body of work that asks the mechanistic question: what is it about training on human feedback that rewards agreement over accuracy? The second question is research, because its answer would tell us something durable about learning from human preferences in general. The first is quality assurance on someone else’s product, and I do not use that phrase as an insult. Quality assurance is indispensable. It is just not science.
The third case, and the richest hunting ground of all, is the optimization of agents. The literature on agentic workflows, orchestration patterns, and prompting strategies for multi-step tasks is growing faster than any other, and it is almost entirely configuration knowledge: what worked, on these models, with this scaffolding, this quarter. The compounding-error arithmetic I discussed in the democratization essay explains why so much of it disappoints in deployment, and the field’s response, more benchmarks, more workflow patterns, more papers, is the cataloguing reflex, Linnaean in spirit, applied to a moving target. But the comparison flatters us. Linnaeus’s taxonomy was durable: it survived Darwin, and it still structures biology today. These catalogues describe species that will be extinct, or evolved, by the next training run. Ask what any of it will still teach us in five years, and the silence is the answer.
Why academia went along
If all this were happening only in industry, it would hardly be worth an essay; the boundary between engineering and research has always been drawn loosely inside companies, and usually for understandable reasons. What is new, and what I find genuinely troubling, is that a large part of academia has followed, and I do not believe it did so cynically. It did so structurally. The frontier labs have, in effect, privatized the Oyster quadrant. The compute, the weights, the training data, the ability to run the experiments that would answer the mechanistic questions: all of it now sits behind corporate walls, at a capital cost no university can approach. An academic who wants to study these machines from the inside mostly cannot. What remains accessible is the API, and so a generation of capable researchers has migrated, understandably, to studying the artifact through the keyhole: behavioral experiments on a black box, rented by the token, whose internals may change without notice between one experiment and the next. And because papers remain the currency of the academic profession, the keyhole observations get written up in the only genre available, and the conferences fill with them.
There is a precedent worth remembering. In 2017, Ali Rahimi stood up at NeurIPS and said that machine learning had become alchemy, a field of techniques that worked without anyone knowing why, and the room erupted, partly because it was rude and partly because it was true. What I am describing is the 2026 version of his complaint, with one difference that makes it worse: at least the alchemists mixed their own reagents. We are doing alchemy on substances we are not allowed to synthesize, handed to us in sealed vials, reformulated by the vendor every few months.
What this is not
Before the conclusion, a clarification, the same kind I have had to make before in this series. None of this is an argument that downstream work is low-value, and it is emphatically not a plea to re-erect the gates around the technology. Prompt engineering, fine-tuning, evaluation, the patient construction of systems on top of imperfect models: this is translation work, the very activity I have spent several essays arguing is the scarcest and most undervalued skill of this phase of AI. The people doing it are not lesser scientists. Most of them are not doing science at all, and that is fine, because translation is not science and does not need to be.
The same precision is owed to reinforcement learning, which I listed, perhaps provocatively, in my opening. Reinforcement learning as a discipline, the theory of agents learning from reward, is one of the deepest research traditions we have. Reinforcement learning applied as a service, to nudge a given model toward given preferences, is engineering. The technique is the same; the activity is not. That distinction, between the discipline and its downstream application, is exactly the one this whole essay is trying to draw.
The harm, then, is not in the work. The harm is in the label. When engineering is labeled research, the field inflates its own sense of what it understands. Conferences flood with catalogues, and the reviewers who might protect the quality drown in the volume. The small number of genuine Oysters compete for attention with thousands of user manuals. Funding agencies, hiring committees, and doctoral students calibrate on what they see published, and what they see published tells them that the job of an AI researcher is to benchmark a rented black box. And the demo-driven economy I have described elsewhere acquires exactly what it wanted: the epistemic authority of science, borrowed at no cost, because a demo dressed as a paper is still a demo.
Where this leads
What would it mean to take research on these given machines seriously? We know part of the answer already, because some people are doing it: interpretability pursued with real access to the weights, theory that explains rather than describes, the science of why scaling works and where it stops. Another part of the answer is institutional, and harder: if the means of research have been privatized, then the arrangements that let academics inside, shared compute, open weights at meaningful scale, structured access programs, stop being generosity and become the condition for the field having a science at all. And a third part is simply honesty in the downstream work itself: the empirical discipline of error rates, validation sets, and statistical significance that my generation lived by would not turn behavioral studies into science, but it would at least make them true.
Mostly, though, what I am asking for is truthful labels. Call the translation work translation, fund it and honor it as such, and stop requiring it to wear a costume. Call the catalogues catalogues, useful and perishable. Reserve the word research for the work that passes the two tests, mechanism and durability, wherever and by whomever it is done. Labels sound like a small thing, but labels determine what gets funded, who gets hired, and what a talented twenty-five-year-old believes the job actually is.
We have confused learning to use the machine with learning how the machine works, and we are publishing the former in the journals of the latter. I remain optimistic, as I usually am, because the confusion itself is a symptom of abundance: never before has a technology this powerful been placed in this many hands this quickly, and the sorting of activities into their proper names is part of the same slow domain construction I keep describing in this series. The machine will still be there when the hype recedes. And it will reward, as machines always have, the people who insist on actually understanding it.