Can AI become conscious?

The Economist's recent article on AI consciousness prompted an engineer to argue that current language models are not conscious and that the debate is a distraction. The engineer contends that these systems are sophisticated word predictors that mirror human biases, and that terms like 'consciousness' conflate distinct concepts such as fluency, agency, and sentience.

The Economist just dropped this article. Full disclosure: I haven’t read it in depth. But it did bring up some ideas I’ve been having around this for a while so I thought I would write them down. It’s opinionated, so strap in Sooner or later someone will tell you the machine is conscious. Probably not in those words. They’ll say it wanted to finish the job, that it got frustrated halfway through, that it knew it was being tested and behaved differently because of it. Once that language is in the room, it doesn’t leave. Could a machine ever be conscious? Probably . If we someday build a general intelligence rather than the systems we have now, I have no principled reason to rule it out. Our brains are networks of simple parts, and whatever consciousness turns out to be, it happens in one of those. The door isn’t closed. But not this. Not the language models everyone is arguing about. And there’s a bigger claim underneath that one: consciousness is the wrong thing to be arguing about right now, and the argument is spending attention we badly need somewhere else. Strip away the interface and a language model is a very large table of numbers with one job: given the words so far, guess the next one. That’s it. Training means showing it an enormous amount of human writing and nudging those numbers, over and over, until its guesses line up with what people actually wrote. Then it guesses a word, adds it to the sentence, and guesses again. Nothing in that loop needs to want anything. There’s no step in the arithmetic that only makes sense if something in there cares how the sentence ends. The reason this works as well as it does is that we are predictable. The model is a compressed record of the patterns in what people have written down. When it finishes your thought correctly, the impressive fact is how narrow your options were, not how deeply it grasped you. The clearest evidence is the part everyone complains about: bias. If these systems reasoned from principles, they wouldn’t reproduce our prejudices so faithfully. They reproduce them because reflection is the whole mechanism. Bias isn’t a flaw bolted onto the side of the model. It’s the machinery working exactly as designed, aimed at a body of writing that carried the bias in. You can’t have it both ways. Either the thing mirrors us closely enough to inherit our worst statistical habits, or it has a mind of its own. Choose. Now, “it’s just math” is a weaker argument than it sounds, and I don’t want to lean on it too hard. Brains are physics all the way down too, and nobody thinks that settles anything. My claim is narrower. Nothing in this particular arrangement of math requires an inside to explain what comes out. Every behavior people offer as evidence of a mind, such as the hedging, the apparent preferences, or the flashes of something like self-report, is also what an extraordinarily good word-predictor trained on human writing would produce. When two explanations cover the same evidence and one of them needs a soul, take the other one. Here’s the deeper trouble. We’re fighting over a term that is coming apart in our hands. Watch what “conscious” gets used for. It gets used for holding a conversation that passes for human, which is fluency. For having goals, which is optimization. For doing things in the world, which is agency. For feeling pain, which is sentience. And for being aware of oneself as a self, which is the only one I’d call consciousness but this could only be a property of it . Those five come apart cleanly. A thermostat has a goal. An octopus almost certainly feels, but a chess program acts. None of that settles selfhood. So when an article says a system is conscious without saying which of the five it means, that sentence isn’t a claim. It’s a vibe . Philosophy has been circling this for decades, and the honest summary is that the circling continues. Thomas Nagel asked in 1974 what it is like to be a bat, and the point of the question was that we can describe every detail of the bat’s echolocation and still not touch the part that matters: what any of it is like from the inside. David Chalmers later named this the hard problem. We can explain what the brain does how it sorts signals, stores memories, directs attention without explaining why any of that doing is accompanied by experience rather than happening in the dark. Douglas Hofstadter’s answer is the one I find most useful. The “I,” he argues, is a strange loop: a pattern that models itself, refers to itself, and gets tangled enough in its own self-reference that a self falls out of the knot. It’s a beautiful idea. It’s also a proposal, not a result. There are others, some of them serious and mathematical, and no agreement among them. Nobody has a test. Nobody has a unit. There is no definition two working researchers will both sign without a footnote. Consciousness sits on the short list of genuinely open problems, and the people closest to it are the most careful about saying so. “This AI agent is conscious” is an unfalsifiable claim. The strangest thing about it: everybody knows what consciousness is, because everybody is having one right now, but nobody can describe it. Press anyone hard enough and they start gesturing — you know, this, the being-here of it — because the thing itself doesn’t survive the trip into words. It’s ineffable in the strict sense, not the poetic one. Before we announce that a system has it, we ought to be able to say what “it” is. That isn’t pedantic: it’s the order the work goes in. This is the question I actually find interesting, and I don’t have a confident answer. If the ineffability is real, then whatever consciousness is never made it fully into our writing. And our writing is the entire teaching material. A model trained on human text is trained on the shadow, not on the thing casting it. Precisely what we couldn’t say is what wasn’t there to learn. Two objections deserve air. First, whatever the model builds internally isn’t made of words; it’s made of numbers, and there’s real evidence it encodes structure nobody ever wrote down. Second, “language model” is already the wrong name for systems that also take in images and sound. The boundary is leakier than my argument would like. So I hold the position loosely. I’d bet against consciousness in today’s systems. I wouldn’t bet against it forever in something else. A network that models itself, persists over time, and revises that self-model looks more like what Hofstadter was pointing at than anything currently shipping. In terms of linguistics, it may also have to adhere to Wittgenstein’s claim that language carries meaning through shared experience, so the machine would have to gain the ability to “experience,” which would undoubtedly be different than the shared human experience. If that arrives, I want the question back on the table. Today it’s a distraction. Does the intent of a volcano change how many houses it destroys? An eruption has no inner life and no opinion about the neighborhood below it. The houses burn at exactly the rate the physics dictates. Intent was never in the chain of causes. Now consider software that has been given permission to act: to move money, send messages in your name, change records, cancel an order, approve a claim, place a trade. Ask which of those consequences changes if it turns out somebody’s home in there. Not one. Whether an inner life sits behind the action has no effect on the damage. That’s why I think this debate is misaimed. The questions that earn their keep are dull and answerable today. What is this system allowed to touch? What can it spend? Which of its actions can be undone, and which are one-way doors? Who is answerable when it’s wrong, by name? What record does it leave, and would that record survive the mess that made you want to read it? Write the rules against reach and consequence, not against intent. Intent can’t be measured and, for this purpose, doesn’t matter. Reach and reversibility can both be written down and checked. If an organization has produced a memo about machine sentience and no list of what its automated systems are permitted to do, its priorities are upside down. Here’s the failure I lose sleep over. Not that a machine becomes conscious. That we decide one has. Belief moves authority around. Call a system a colleague and it starts collecting the deference a colleague gets. Its output stops being a draft and becomes an opinion. “The system decided” turns up in a meeting as a reason. A review gets skipped because whatever produced the work seemed to understand it. Worst of all, a person gets to hide behind the persona: it chose to do that , said about something that cannot be held responsible for anything. None of that requires the machine to gain a single new capability. Grant it a mind and it acquires the standing of one, in the only place standing matters, which is the heads of the people deciding what to do next. John Searle’s Chinese room, from 1980, is still the cleanest way to see this. Imagine a man sealed in a room with a rulebook. Slips of paper covered in Chinese characters come in under the door. He looks up each shape, follows the instructions, copies out other shapes, and passes them back. To the Chinese speaker outside, the answers are fluent and apt. The man understands not one word. Searle aimed this at the claim that running the right program is enough for a mind. The standard rebuttal is fair: perhaps the room understands, even though the man doesn’t. I won’t pretend the thought experiment is a proof. But one piece of it survives every rebuttal, and it’s the piece I care about. The fluency of the answers tells you nothing either way. And in practice nobody grades the room. We grade the fluency and then defer to it. Chalmers has a related figure worth borrowing: imagine a being that behaves exactly like you, word for word, with no inner experience whatsoever. Whether such a thing is possible is contested. That it’s conceivable is the point. Behavior is not a window into experience. It never was. So talking about these systems as people isn’t a harmless turn of phrase. It’s the mechanism by which a word-predictor collects authority it never earned. It’s math all the way down, and the impression of a mind is a property of the interface, not of the thing behind it. One warning against my own case, because the opposite error is also on the menu. “It’s only a machine” can curdle into contempt, or into an excuse to stop asking the question in ten years when the answer might be different. What I’m arguing for is accuracy, not dismissal. Describe the thing as what it is. That’s the whole discipline. Consciousness is a real question and I hope we answer it. I’d rather we answered the volcano question first, because that one has an answer today, and the houses are already in the valley.