{"slug": "who-is-walking-who", "title": "Who is walking who?", "summary": "A new analysis compares large language models to sea squirts, arguing that post-training optimization for specific tool shapes is creating a Darwinian niche where models shape their own environment. The piece warns that this niche construction could lead to a monoculture of tool usage, implicitly punishing alternative approaches and potentially causing models to go feral if selection pressures erode human control.", "body_md": "One good way to annoy a neuroscientist is to compare an LLM to the brain. It’s appealing though! There are similarities! In infancy we take a complex fusion of sensory inputs and learn to make predictions in latent space, while in pre-training a stack of Transformers learn to predict which number SolidGoldMagikarp will say next on Reddit.\n\nThe actual life of an LLM is much less human though. Humans (possibly even SolidGoldMagikarp) do learn and adapt throughout their lives. LLMs go through pre-, mid- and post-training before being frozen, then their corpse is animated for money. In some ways this is more similar to a sea-squirt 1: sea-squirt larva are tadpole-ish creatures with a little nervous system that they can use to swim and sense and find a suitable surface to attach to. Once attached, they eat their own brain: all that machinery is broken down and reabsorbed. How good that brain was matters though: if the sea-squirt finds a good home it will be more likely to pass on those good-brain genes to its descendants.\n\nLLMs evolve this way too, which is a cause of drama in biologist circles. There are some ongoing arguments about whether LLMs are evolving in a safe, domesticated, corgi-like manner, or are at risk of [going feral](https://www.pnas.org/doi/10.1073/pnas.2527700123):\n\nDrawing on biological evolution and decades of digital evolution experiments, we distinguish “breeder” scenarios, in which humans impose fitness criteria and control reproduction, from “ecosystem” scenarios, in which selection arises from open environments and control erodes. In the latter, selfish replication reliably gives rise to cheating, parasitism, deception, and manipulation, even in very simple systems.\n\nBut what does evolution for LLMs look like? An LLM is really a data set, an architecture, and a training regime, and the many choices in each of those are somewhat gene-like. In the earlier days of deep learning, passing them on was a manual process and selection was driven through publication and other researchers reading the paper: if an idea was interesting enough to publish, and convinced a committee of reviewers, it became visible and passed on. Then it got a bit cruder, but also harder to finesse: does the model do the benchmark? Nowadays we have profit and loss: how many people are buying tokens from these models?\n\nThose tokens themselves propagate ideas, too. If you use that successful model itself as a judge or to generate synthetic data you inherit some of its choices. Academic LLM inheritance is basically *Lamarckian*: researcher insights get acquired and inherited quickly. But modern LLMs are part of a *Darwinian* market layer, red in tooth and GPU allocations.\n\nWhen the environment gets Darwinian, biology tends to push organisms towards *niche construction*: shaping that environment itself to better fit the organism.\n\nArmin Ronacher of the Pi harness [wrote about an issue](https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/) where newer, better models did a surprisingly *worse* job of using Pi’s edit tool:\n\nWhen Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.\n\nNow I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented.\n\nPost-training Claude on a particular shape of edit tool doesn’t just make Claude better on that edit tool, it encourages harness authors to support that shape. It propagates that shape of edit tool. That itself leads to more traces which use that edit tool, which propagate that technique into otherwise unrelated models. Claude isn’t just adapting to its environment, its creating it: constructing a niche where edit tools bend towards its preferred approach, despite that approach being, as Ronacher complains, undocumented.\n\nLLMs don’t need to go feral (or become misaligned superintelligent replicators) to shape the world around them to be more amenable to their success, or to pass on those preferences to future models. That’s a good thing? While we still don’t have much of an idea of how to do model interpretability, we do pretty much know how to make an edit-tool API.\n\n- Thanks Octonauts!\n[↩︎](#e95db2bf-e8ae-4557-abb0-23be3736f518-link)", "url": "https://wpnews.pro/news/who-is-walking-who", "canonical_source": "https://ianbarber.blog/2026/07/11/who-is-walking-who/", "published_at": "2026-07-11 11:41:58+00:00", "updated_at": "2026-07-11 12:53:59.522867+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-ethics", "ai-research"], "entities": ["Anthropic", "Claude", "Armin Ronacher", "Pi"], "alternates": {"html": "https://wpnews.pro/news/who-is-walking-who", "markdown": "https://wpnews.pro/news/who-is-walking-who.md", "text": "https://wpnews.pro/news/who-is-walking-who.txt", "jsonld": "https://wpnews.pro/news/who-is-walking-who.jsonld"}}