{"slug": "computer-scientist-cal-newport-adds-some-clarity-to-the-ai-debate", "title": "Computer scientist Cal Newport adds some clarity to the AI debate", "summary": "Computer scientist Cal Newport argues that recent AI safety incidents stem not from large language models in general but from a narrow class of systems he calls \"long-horizon, dangerously equipped unsupervised LLM-powered agents.\" He contends that companies like Anthropic and OpenAI could halt work on these persistent, guardrail-free agents with essentially zero impact on projected revenue, and questions why they have made such systems central to their efforts.", "body_md": "As previously mentioned, software veteran Carl Brown has pointed out that we can prevent incidents like the hugging face hack simply by enforcing existing laws.\n\nHere the reliable Cal Newport [explains](https://calnewport.com/anthropic-just-threatened-to-kill-billions-of-people-this-is-not-okay/) that these incidents all come not from LLMs in general but from the reckless application of one particular application of LLM-powered agents, and that there's no good explanation for the risks these companies have been taking.  \n\nWhen Hubinger says “AI could kill all humans,” he’s not talking about AI in a general sense; he’s really referring to a specific type of AI system that we can call an *LLM-powered agent*.\n\nThese systems consist of an old-fashioned computer program that repeatedly does something like the following:\n\n- Sends a prompt to an LLM describing its goal and current state, then asks for a suggestion about what to do next to move closer to the goal.\n- Blindly executes whatever the LLM describes in its response.\n- Updates its state based on what happens and then loops back to the first step.\nLLM-powered agents aren’t necessarily dangerous. Millions of software developers use these systems every day to help write and debug computer code. Though these coding agents make mistakes, no one is worried about them going rogue in any alarming sense. (OpenAI [recently scanned](https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/) “tens of millions” of traces of coding agent interactions with their LLMs and found *zero* instances of high-severity incidents.)\n\nHubinger is likely referring to the same sub-class of these agents that was involved in the autonomous hacking attacks over the summer, and which satisfy the following additional properties:\n\n- They are given access to powerful tools and information specific to computer hacking.\n- The LLM they prompt has had guardrails removed so that it will respond to prompts requesting information about dangerous or illegal behaviors.\n- The agents have minimal (or no) safety checks or constraints on what actions they’ll execute. (Standard coding agents, by contrast, are hard-coded with long lists of commands that they will not execute, even if an LLM suggests it.)\n- The agents are run for very long time periods (sometimes multiple days) without any human supervision. They are programmed to be\n*persistent*, meaning that they should never give up but instead continually prompt the LLM for new next steps to try, pushing it to get more creative and brazen in its suggestions.\nWe can call these **long-horizon, dangerously equipped unsupervised LLM-powered agents**. As best as we can tell, it’s this very narrow type of unpredictable and potentially hazardous system that companies like Anthropic are rushing recklessly ahead to provide increasingly powerful tools to play with (including, reportedly, the ability to update elements of their own code) and increasing autonomy (achieved, in part, by post-training the LLMs they prompt to suggest more aggressive actions).\n\nThis information lets us be more specific about recent claims. The concern of the moment is not that *AI* is inexorably becoming harder to control and scary. It’s instead *long-horizon, dangerously equipped unsupervised LLM-powered agents* that are making people nervous.\n\nThere’s an obvious solution here: **stop racing to amplify this very specific type of particularly unstable system**.\n\nThis wouldn’t even necessarily entail much financial sacrifice. Most of the things people already like doing with LLMs, or hope LLMs will enable soon, don’t require these haphazard and unpredictable setups. Anthropic and OpenAI could stop working on them today with essentially zero impact on their projected revenue.\n\nAll of which begs the question: *Why* have these particular AI companies made these alarming long-horizon LLM-powered agents so central to their efforts? I’m not certain of the answer, but if I were to guess, it probably has a lot to do with the Silicon Valley *technological salvation ideology* (to borrow a term [from Adam Becker](https://www.amazon.com/More-Everything-Forever-Overlords-Humanity/dp/1541619595/)) that [heavily influenced](https://www.nytimes.com/2026/09/05/opinion/ai-silicon-valley.html) key figures like Sam Altman and Dario Amodei, as well as many of their employees. They think these types of agents are their best bet to summon the digital deity of superintelligent AI which, in their futurist eschatology, will either heal the world or destroy it. In some sense, they see themselves as prophets attempting to usher in a new Messianic age.\n\n*All of this should make you angry.*", "url": "https://wpnews.pro/news/computer-scientist-cal-newport-adds-some-clarity-to-the-ai-debate", "canonical_source": "https://observationalepidemiology.blogspot.com/2026/09/computer-scientist-cal-newport-adds.html", "published_at": "2026-09-17 11:30:00+00:00", "updated_at": "2026-09-17 11:54:42.319730+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "large-language-models", "ai-policy"], "entities": ["Cal Newport", "Anthropic", "OpenAI", "Hubinger", "Carl Brown"], "alternates": {"html": "https://wpnews.pro/news/computer-scientist-cal-newport-adds-some-clarity-to-the-ai-debate", "markdown": "https://wpnews.pro/news/computer-scientist-cal-newport-adds-some-clarity-to-the-ai-debate.md", "text": "https://wpnews.pro/news/computer-scientist-cal-newport-adds-some-clarity-to-the-ai-debate.txt", "jsonld": "https://wpnews.pro/news/computer-scientist-cal-newport-adds-some-clarity-to-the-ai-debate.jsonld"}}