{"slug": "llm-powered-self-sabotage", "title": "LLM-Powered Self-Sabotage", "summary": "Organizations are undermining themselves by treating ungrounded, LLM-generated numbers and synthetic research as facts, a phenomenon dubbed 'LLM-powered self-sabotage.' Executives demand AI adoption due to competitive pressure, but the resulting confabulated data—such as hallucinated market sizes and simulated user feedback—erodes decision-making quality. This automation misuse amplifies motivated reasoning and creates an illusion of knowledge while actual understanding declines.", "body_md": "# LLM-Powered Self-Sabotage\n\nWe do not need [LLM-powered industrial sabotage](/llm-powered-industrial-sabotage/).\nWith humans in the loop, we can self-sabotage much more efficiently.\n\nThe World Economic Forum lists LLM-driven misinformation among [the most significant near-term global risks](https://www.weforum.org/reports/global-risks-report-2024/), because generative systems drastically reduce the cost of producing fake but credible content.\nSimilarly, ENISA warns that generative AI increases [the efficiency and scale of manipulation campaigns](https://www.enisa.europa.eu/publications/enisa-threat-landscape-2023) in politics.\n\nBut we do not need adversaries weaponizing LLMs against us.\nMany organizations are already undermining themselves by accepting ungrounded, LLM-generated numbers and synthetic research as if they were facts.\nExecutives demand AI adoption [because competitors are doing it](/executive-deterministic-parrots/) and boards expect it.\n[Speed is paramount](/slow-the-fuck-down/), yet correctness is earily absent from each mandate.\n\n##\nConfabulated certainty\n\nA startup pivots after discovering that the total addressable market in a related area is much larger than it previously imagined.\nBehind the number there is no survey, no industry dataset, no methodology section, and no sensitivity analysis.\nThere is a prompt, though, and the model’s prediction of what a plausible market-sizing paragraph might look like.\nThere may be a link or two, but they lead to non-existent reports that few bother to uncover by clicking.\nThat is hardly surprising, because LLMs complete the text, and the most probable completion of “the total addressable market for XYZ is” happens to be a [confident, yet hallucinated number](https://doi.org/10.1145/3571730).\n\n[Bullshit](https://doi.org/10.1007/s10676-024-09775-5) is more apt: [an indifference to truth](/bullshit-lies-improvisation/) rather than correctness gone awry, which is what hallucination implies.\nLLMs do not care whether their claims are connected to anything real; they were never oriented towards correctness.\n\nEventually the board of directors approves the pivot.\nThe figures survive due diligence because they appear in a deck with other figures that also appear precise, and precision is contagious: [once one number carries decimals](https://doi.org/10.1016/j.jesp.2013.02.012), adjacent numbers inherit authority by proximity.\nNo one on [the board can distinguish a researched estimate from a generated one](https://doi.org/10.1093/oso/9780190861094.001.0001), which means no one can gauge the risk of the strategy they just endorsed.\n\n##\nSelective scrutiny\n\nIf a number confirms what leadership already believes, it passes without friction.\nIf it contradicts expectations, the model that generated it is suddenly blamed.\nThis tendency to scrutinize evidence that goes against our views more harshly than whatever confirms our beliefs is known as [motivated reasoning](https://doi.org/10.1037/0033-2909.108.3.480).\n\nMotivated reasoning predates LLMs, but the models amplify it when [applied without thought](/lost-in-compression/), because it is so easy to generate data that looks real but isn’t.\nThis automation misuse was already [identified back in 1997](https://doi.org/10.1518/001872097778543886), though we have ignored its lesson ever since.\nOver time, an organization generates more numbers than ever, cites more sources than ever, yet knows less than it did before, because the numbers and sources are decorative.\n\n##\nSynthetic customers\n\nInstead of recruiting participants, running interviews, and analysing transcripts, [product teams prompt a model](/synthetic-users/) for “a 32-year-old power user” and receive feedback in seconds.\n\nBut the simulated power user has no commute, no frustration with last week’s release, no competing product open in the next tab, no children interrupting the session, no unforeseen bills to pay this month, no reason to lie about how often they actually use the feature. Real users are inconvenient, yet their inconsistencies reveal what surveys and personas cannot. Synthetic personas are useful for generating hypotheses or drafting interview guides, but treating them as empirical validation is like load-testing a bridge with a photograph of a truck. When a product team ships a feature because the simulated users loved it, they will pay in churn what they saved in their research budget.\n\n##\nGoing ballistic\n\nIn recent [warfare simulations](https://www.newscientist.com/article/2516885-ais-cant-stop-recommending-nuclear-strikes-in-war-game-simulations/), multiple leading language models repeatedly recommended nuclear strikes as part of crisis decision scenarios, even when escalation would be strategically irrational for humans.\nThe models were completing text in a context where the training distribution made escalation the highest-probability continuation.\nThey do not feel fear or experience human cost, no matter how much we anthropomorphize LLMs.\n\nThe same statistical mechanism that produces “launch a warhead” produces “the total addressable market is $4.2 billion” and “users overwhelmingly prefer the new onboarding flow.” The model does not know which of these outputs will end a career, a company, or a country.\n\n##\nYou write* it, you own it (* even if you only pretend)\n\nWhen someone copy-pastes unvalidated model output, the burden to check facts and figures typically becomes the reader’s rather than the author’s, which is why [it is perfectly rational to stop reading LLM-generated rubbish from colleagues](/lost-in-compression/).\nWhenever your name is at the top of a document, you must own its contents and provenance.\nThe burden of proof always lies with the author, never the reader.\nAnd if you include Claude or whichever LLM *du jour* in the list of authors, the standard assumption is that the entire document was generated from a prompt that took only seconds to type, which is the maximum amount of attention any rational person ought to give it, too.\n\n##\nMandated sabotage\n\nAny decision-relevant number must include a source or derivation from primary sources. Any user insight influencing product direction must specify whether it is simulated or empirically observed with links to transcripts. These are the standards that existed before LLMs made it possible to bypass them at zero marginal cost.\n\nSpeed matters in business, but speed that compounds errors is simply deferred correction. We worry about adversaries weaponizing AI against us, but the more immediate risk is that we lower our own standards and call it progress. Companies destroy themselves when they confuse fluency with evidence and reward the production of numbers over the defence of numbers. The sabotage is mandated from the top and settled in severance.", "url": "https://wpnews.pro/news/llm-powered-self-sabotage", "canonical_source": "https://ianreppel.org/llm-powered-self-sabotage/", "published_at": "2026-06-28 22:00:00+00:00", "updated_at": "2026-07-13 13:52:32.195988+00:00", "lang": "en", "topics": ["large-language-models", "generative-ai", "ai-ethics", "ai-safety"], "entities": ["World Economic Forum", "ENISA"], "alternates": {"html": "https://wpnews.pro/news/llm-powered-self-sabotage", "markdown": "https://wpnews.pro/news/llm-powered-self-sabotage.md", "text": "https://wpnews.pro/news/llm-powered-self-sabotage.txt", "jsonld": "https://wpnews.pro/news/llm-powered-self-sabotage.jsonld"}}