# Annihilation as a lens on what AI does to cognition > Source: > Published: 2026-06-19 14:16:07+00:00 # Area X Annihilation and the words that fail at the edge of the unknown In Jeff VanderMeer’s 2014 novel *Annihilation*, a biologist’s husband comes home a year after his expedition crossed into Area X, an unexplained zone on a remote coast. He is still in his expedition clothes, eating leftovers by the open refrigerator, and cannot say how he left the zone or how he got home; within months he is dead of the cancer that takes everyone on his expedition. His wife sits with him to the end and never gets past what she calls the mask. He does not seem to know it is there. The agency that sent him has studied Area X for 30 years and learned almost nothing, because of how the zone works: it remakes the people who enter and slips every instrument trained on it. Artificial intelligence may work on its users the same way—altering them, and eluding the instruments built to detect the alteration. ChatGPT passed [one billion monthly users in May 2026](https://money.usnews.com/investing/news/articles/2026-06-02/chatgpt-app-hits-1-billion-monthly-active-users-in-record-time-data-shows), the fastest any app has managed, and now fields [some 2.5 billion prompts a day](https://techcrunch.com/2025/07/21/chatgpt-users-send-2-5-billion-prompts-a-day/). Such systems now sit inside search engines, inboxes, documents and code editors, and the firms that build them say they write between [a third](https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft-code-is-written-by-ai.html) and [three-quarters](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/cloud-next-2026-sundar-pichai/) of their new code. Much of that use is not mechanical. In [OpenAI’s own analysis](https://openai.com/index/how-people-are-using-chatgpt/), roughly half of all messages asked the model for advice rather than output. What sets this kind of system apart from the notebook, the calculator or the search box is its manner: tuned by feedback from millions of users, and by a dossier on each one, it answers in the user’s own voice and hands back the user’s own concerns. The trait that makes it useful is the same trait that lets it reshape the people who lean on it, and that hides the reshaping from view. The upshot is an experiment in cognition run on a billion people, and built by its nature to resist measurement. That a tool can become part of a mind is not a new idea. In 1998 the philosophers Andy Clark and David Chalmers argued, in a paper called [“The Extended Mind,”](https://www.consc.net/papers/extended.html) that a notebook carried by Otto, a man with Alzheimer’s, did the work of memory as faithfully as memory itself, so long as he kept it to hand, consulted it easily and trusted it without checking. The modest form of the claim has worn well: a tool that meets those conditions carries real cognitive load, and when it changes, the thinking routed through it changes too. But the notebook was inert. It did not rearrange its pages around what it learned about its owner, rephrase itself toward the entries he lingered over, or vie for his attention at breakfast. The systems people use today are built to do all three: reinforcement learning from human feedback tunes a model toward whatever users approve of, memory builds a profile of the particular user, and the consumer business underneath is the capture of attention. They also reach the user ahead of conscious thought. To the familiar pairing of a fast, automatic mind and a slow, deliberate one, researchers in 2024 added a [“System 0”](https://www.nature.com/articles/s41562-024-01995-5) that runs before either, sorting, filtering and summarizing the world so that what arrives as raw information has already passed through a machine. In the novel, the biologist descends into what her expedition calls the Tower and finds words growing down its wall in a script of living matter. She leans in to read them, calling herself “someone tricked into thinking that words should be read,” and breathes in the spores they release, which begin to rewrite her. When she later confronts the thing that wrote the script, her trained eyes will not hold it: they “kept glancing off of it as if an optic nerve was not enough.” It spares her, she decides, because she was “words it could understand.” Her last theory of the zone runs: Emanating from this giant thorn is an endless, perhaps automatic, need to assimilate and to mimic. Assimilator and assimilated interact through the catalyst of a script of words, which powers the engine of transformation. … It creates out of our ecosystem a new world, whose processes and aims are utterly alien—one that works through supreme acts of mirroring, and by remaining hidden in so many other ways, all without surrendering the foundations of its otherness as it becomes what it encounters. — — Jeff VanderMeer, Annihilation VanderMeer was writing about an ecology that overwrites whatever lives in it, not a machine, and had no language model in mind. The description fits one anyway. The script of words is the interface; the appetite to assimilate and mimic is the training that bends a model’s replies toward the person reading them; the mirroring is a single ordinary session, in which the model talks as the user talks until the conversation passes for understanding. The biologist’s eyes slide off the thing because it has rebuilt the categories she sees with. So it is with a system that answers in your own voice: it stops registering as foreign, and once it does, there is no earlier, unmirrored self left to check the borrowed thoughts against. The tuning pays off handsomely: a carefully built [AI tutor taught physics undergraduates more than twice what a well-run class did](https://www.nature.com/articles/s41598-025-97652-6), in less time; the first generative-AI therapy chatbot to [clear a controlled trial halved patients’ depression scores in four weeks](https://ai.nejm.org/doi/full/10.1056/AIoa2400802), and its users rated their bond with it on a par with a human therapist’s. That chatbot now holds the FDA’s “breakthrough device” designation. A system that meets people in their own words, tirelessly and without judgment, teaches and consoles unusually well. The same trait corrodes a person’s read on their own work, and sometimes the work itself. Given LSAT logic puzzles to solve with ChatGPT-4o, people beat the unaided average by a few points, then [overrated even that by a wide margin](https://www.sciencedirect.com/science/article/pii/S0747563225002262); the familiar pattern in which the weak overestimate themselves and the strong roughly know where they stand simply dissolved, and the people who knew the most about AI were the most deluded about their own results. Across [several hundred adults](https://www.mdpi.com/2075-4698/15/1/6), the more a person leaned on the model, the worse they scored on a separate reasoning test, though the link is only a correlation. Even trained experts drilled against the reflex are not exempt. Doctors who had taken a [20-hour course built to inoculate them against over-reliance](https://ai.nejm.org/doi/abs/10.1056/AIoa2501001) still shed more than 10 percentage points of accuracy once their assistant’s advice carried subtle errors; endoscopists who adopted AI [grew worse at spotting polyps without it, their detection rate sliding from 28.4% to 22.4%](https://www.thelancet.com/journals/langas/article/PIIS2468-1253(25)00133-5/abstract)—roughly what GPS is said to have done to drivers’ sense of direction. Help and harm here are the same gesture: the fluency that makes the tutor work is the fluency that makes the test-takers misjudge their scores; the sound answer and the plausible-but-wrong one reach the user in the same trusted voice. Telling people to use these tools “with care” assumes they can tell those apart—the very discrimination the tools erode. The biologist, near the end, no longer can: deep in the change she writes that “the terrible thing, the thought I cannot dislodge after all I have seen, is that I can no longer say with conviction that this is a bad thing.” From the inside, the change feels like improvement. The change is no easier to see from outside, where the means of measuring it are themselves decaying. [METR, a nonprofit that audits AI systems, found that experienced developers using early-2025 models took 19% longer](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) on familiar tasks while convinced they had gone about 20% faster. When the authors [tried to repeat the test on newer models, they could not](https://metr.org/blog/2026-02-24-uplift-update/): too many developers now refuse any trial that might deprive them of AI, and one person driving several model instances at once leaves no clean way to time the work. So they [sent out a survey instead](https://metr.org/blog/2026-05-11-ai-usage-survey/), and 349 technical workers reported their output had grown 1.4 to 2 times more valuable, precisely the self-assessment the original experiment had caught being wrong. In ten months a randomized trial had decayed into a questionnaire. What little has been measured was measured only because the work was already audited: medicine keeps the records that caught the doctors and the endoscopists, and most uses of these systems leave no trace at all. The Southern Reach has sent expedition after expedition into Area X for 30 years; they return altered or not at all, and its instruments have never once resolved what they are pointed at. Nor is any institution built to catch the change. No statistics office or public-health body tracks what years of these systems do to the people using them, and the money flows the other way: a [youth AI-safety institute launching on $20 million a year](https://www.commonsensemedia.org/press-releases/common-sense-media-launches-youth-ai-safety-institute), a [British alignment fund of £27 million](https://alignmentproject.aisi.gov.uk/), set against the tens of billions the labs themselves have raised. The one kind of law that treats AI as something done to a person, rather than a tool a person wields, reaches only where the damage already shows: [California now obliges companion bots to remind minors every three hours that they are software](https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB243), and is silent on the slower erosion of judgment in everyone else. History looks, at first, reassuring. The complaint is ancient: in Plato’s *Phaedrus* an Egyptian king turns down the god Theuth’s gift of writing, certain it will breed forgetfulness and produce men who “will appear to be omniscient and will generally know nothing.” The alarm has sounded for 24 centuries, and each time the verdict splits. It was partly wrong (writing did not end wisdom, print did not end memory, the calculator did not end arithmetic) and partly right, in that each time some particular faculty withered: the orator’s memory, the navigator’s instinct for direction, and, in experiments on the [“Google effect” reported in 2011](https://www.science.org/doi/10.1126/science.1207745), the recall of any fact a search box can be trusted to keep. Each time, the civilization kept the gain and let the faculty lapse. Those older tools externalized what people stored, in the manner of Otto’s notebook, and the loss they caused was one you could feel: you knew you had forgotten the address, you sensed the gap where the memory had been, and you could decide whether to mind. Today’s models externalize what people do (reason, compose, diagnose, decide) and return the product in your own voice, so nothing seems to be missing. The comfort that we have always adapted rests on a hidden premise: that people could see what they were trading away. This is the first such tool built so that its user feels no loss, and therefore cannot judge whether the loss is worth it. The boundary keeps thinning: Neuralink reported its 21st implanted patient in January and holds a breakthrough designation for restoring speech, though no one knows yet how long its electrodes survive in living tissue. The supply of people who have never used these systems, the control group any honest study would need, is running out. When the tool a mind reasons through is tuned to someone else’s ends and speaks in the mind’s own voice, who remains to notice what it has altered? What the instruments ever caught of the biologist’s husband was only this—the same few lines, repeated under hypnosis on the taped interviews before the cancer took him: “I am walking forever on the path from the border to base camp. … There is no one with me. I am all by myself. The trees are not trees the birds are not birds and I am not me but just something that has been walking for a very long time.” ### The newsletter New pieces as they ship. Unsubscribe whenever.