{"slug": "adversarial-communication", "title": "Adversarial Communication", "summary": "AI systems inevitably make unpredictable mistakes, requiring verification of every output. This verification often costs as much as doing the work manually, leading companies to externalize that cost onto workers, who become 'reverse centaurs'—humans checking AI's work. The author argues that human understanding was always the bottleneck, and AI merely shifts the cost of understanding from writing to verification.", "body_md": "As I have discussed in [previous posts](https://blog.glyph.im/2025/08/futzing-fraction.html),\n*“AIs” can make mistakes*. In fact, they *do* make mistakes, and their\nmistake-making patterns are such that where and how they will make mistakes is\nboth uncertain and constantly changing.\n\nThus, in any scenario where you want to attempt to make “productive” use of\n“AI”, you must have a system in place for checking every result. Not checking\n*some* results; checking *every* result. If each result might have a\nconsequence for you (and if it didn’t have a consequence, why bother automating\nit?) and you cannot predict in advance which kinds of results will need\nverification, then verification is always required.\n\nThe verification often ends up being just as expensive as doing the work in the\nfirst place, which means that if you want your usage of “AI” to be personally\nprofitable, you have to find someone *else* to externalize the cost of\nverification onto. This person becomes your adversary, and, if you are\nsuccessful, your “AI’s” victim.\n\n## The Ladder-Climber And Their Reverse-Centaur Rungs\n\nOne way that this constellation of facts can straightforwardly assemble\nthemselves into a dystopian nightmare is the phenomenon, described by Cory\nDoctorow, of the [reverse\ncentaur](https://locusmag.com/feature/commentary-cory-doctorow-reverse-centaurs/).\nThis is when your employer non-consensually turns *you* into the verification\nsystem. The “AI” does the fun part of initially performing the work, and then\nyou do the boring part where you check if the robot is right and clean up its\nmesses, even if [everyone already knows that it would, in aggregate, be cheaper\nfor you to do the work in the first\nplace](https://fortune.com/article/why-is-the-cost-of-ai-higher-than-human-workers-nvidia-executive/).\n\nReverse centaurs can be made from any automation, not *only* “AI” automation.\nI think that there is a reason that this term happens to have emerged in the\n“age of AI”, though, and not with earlier automation technologies (even those\nwhich were\n[considerably](https://en.wikipedia.org/wiki/Cotton_gin#Effects_in_the_United_States)\nmore [viscerally\nhorrific](https://victorianweb.org/technology/ir/capuano.html)). That reason\nis: the *wrongness* of “AI” output is not merely a technical feature that must\nbe compensated for, it is a generalized externality.\n\nAs I mentioned above, if you are responsible for the entirety of the work, both\nextruding the “AI” output *and* checking it, it’s usually cheaper to have\nhumans do the entirety of the work to begin with. When humans do the writing\ndirectly, we can check as we go, and thus verification doesn’t need to be as\ncomprehensive.\n\nWhen “AI” coding advocates say “code review is the\n[bottleneck](https://lucumr.pocoo.org/2026/2/13/the-final-bottleneck/)”, what\nthey are observing is that the LLM is still rolling the dice for each PR, and a\nhuman is still necessary to verify that each of those rolls is a winner. But\ncalling this process “code review” is a bit of a\n[misnomer](https://blog.glyph.im/2026/03/what-is-code-review-for.html); it’s not really “code\nreview” in the traditional sense, it’s *human understanding*.\n\nBefore the advent of “AI”, the human understanding was implicit in the process\nof writing the code in the first place 1, and the code review was a way of\ndiffusing and extending that understanding. Now that the code can be authored\nwith no initial understanding taking place, that cost has not gone away, it has\nmoved.\n\nHuman understanding was *always* the bottleneck.\n\nHowever, this is taking a *collaborative* view of a software project, where\nsatisfying the needs and solving the problems of your customers are the goals.\nWe can see that “AI” is a bad tool to satisfy those goals, because all it’s\ndoing is converting the first half of the work, that of understanding the code\nas you write it, to understanding the agent’s output as you read it.\n\nWhat if, instead, we were to take the view that every software company is a Hobbesian nightmare, red in tooth and claw? In this view, the only goal of a software project is for the individual developers to make their promo cycles and get their bonuses. Given that there is only a certain amount of money to go around, this is a zero-sum game where each programmer wants to look more productive than their colleagues.\n\nPretty much *every* organization finds it easy to reward “productivity” as\nexpressed by lines of code emitted, but the benefits of doing\nthorough and thoughtful design, analysis, and code review *very difficult* to\nreward. In this world, an LLM is an invaluable tool for the sociopathic\nladder-climber, particularly if your legacy organization is still structuring\ntheir workflows as if the person prompting the bot is “writing” the code, and\nthen they get to foist off the act of “reviewing” the code onto someone else.\n\nHere, the prompter effectively externalizes the cost of the LLM’s failures but\ninternalizes any benefits. The prompter will vibe-code a big feature, so large\nthat the assigned reviewer can’t possibly comprehend it all effectively. When\nthis happens, the reviewer will, *eventually*, be pressured to approve it, even\nif they can try to spot a few problems along the way. The reviewer has their\nown work to get back to, after all, the obligation to review the prompter’s\n(read: the bot’s) code is a drain on their time that they are not going to get\nrewarded for.\n\nIf this feature is a big success, the prompter gets a promotion. If it causes a big issue, well, the reviewer must not have been careful enough.\n\nThis is why LLMs are “good for coding”, and also why their biggest promoters\n[keep](https://ap7i.com/posts/github-outages-vibe-coding-era/)\n[having](https://www.linkedin.com/posts/paulsf_last-weeks-massive-google-cloud-outage-activity-7340015235321278466-c0hz)\n[outages](https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f771de).\n\n## The Generative Gish Galloper\n\nCoding is the biggest “success story” of this type of adversarial\ncommunication, but it is by far not the only instance of such a thing. LLMs\ncreate a new form of leverage that can turn [Brandolini’s\nlaw](https://en.wikipedia.org/wiki/Brandolini%27s_law) from a linear advantage\ninto an exponential one. If you are engaged in a political debate where you\nwant to overwhelm the other side in nonsense, an LLM can generate bullshit\nfaster than it is physically possible for a human being to type, let alone\nrespond thoughtfully. There is an asymmetry to the utility of this weapon as\nwell: only one side of the political spectrum wants to [flood the\nzone](https://en.wikipedia.org/wiki/Flood_the_zone) and destroy trust in\ninstitutions and the concept of truth. There’s a good reason that [the\nfascists love\nit](https://newsocialist.org.uk/transmissions/ai-the-new-aesthetics-of-fascism/).\n\n## Straightforward Spam and Fraud\n\nThis is kind of obvious, but LLMs can generate lightly-customized,\nplausible-looking text much more quickly than any human being. [This\nfacilitates their use in fraud, spam, and\nscams.](https://withpersona.com/blog/llm-fraud) In a spamming or fraudulent\ninteraction, once again, the costs are externalized onto the victim: the\nrecipient of a spam message has to do all the work of “checking” the LLM’s\noutput. Spammers already expect very low hit rates from boilerplate, and if the\nLLM can increase those percentages from 1% to 5% the technology will pay for\nitself; they don’t need anything like *reliable* accuracy.\n\n## Customer “Support”\n\nIf you have any kind of commercial relationship with a company, I probably\ndon’t even need to mention this: customer “support” bots are a misery.\n[Everybody knows\nit](https://www.forbes.com/sites/terdawn-deboe/2026/04/20/customers-hate-your-ai-chatbot-small-businesses-should-listen/)\nat this point. But customer support is usually conceptualized by businesses as\nan adversarial interaction, because it is a cost center. They maintain\ninternal metrics on time-to-resolution and try to optimize them. Implicitly,\nthis creates a dynamic where the goal of the customer service agent’s job is\nnot to solve your problem, but to emit noise that will cause you to *think*\nyour problem is resolved, or to give up, as fast as possible. Unsurprisingly,\nLLMs can emit this noise faster than humans can, getting those customers off\nthe phone. But those customers will *remember* those interactions, and the\nstory *outside* the TTR metrics is horrible.\n\nSimilarly to the situation in software development, LLMs can look very good on paper for customer support, but mostly what they are doing is illuminating the problems with the industry’s existing metrics, by turning “winning the metrics battle against the customer” into a more obvious and immediate defeat for the company’s long term reputation.\n\n## “Education”\n\nIn 2026 it is sadly a fact of life that [students cheat all the time using\n“AI”](https://www.nytimes.com/2026/06/18/us/ai-apps-students-cheat.html), and\nthat this cheating is very successful, in that the teachers find it very hard\nto detect.\n\nLLMs are great for cheating on schoolwork because the student is externalizing the work of the checking onto the teachers, who are often starting at a disadvantage to begin with, at least in the US.\n\nMy view is that this is happening because of a divergence in the way that students vs. teachers (or, more accurately, “the broader educational system”) view grading.\n\nWhen a student is asked to write an essay, the teachers see the effort as both intrinsically worthwhile for the student, as well as useful as a pedagogical tool to evaluate and react to the student’s progress. The student, by contrast, sees a stumbling block designed to knock them off the path to success and into a permanent underclass. It is no wonder that the student sees “AI” as useful to their own goals and has no compunction about deploying it.\n\nThere is a bitter irony that the ability to understand the inherent value of actually writing the essay on their own is the sort of thing that students can really only learn by writing a bunch of essays. There’s no way that I can think of which makes the benefit legible as long as a shortcut is available.\n\nThe net effect here is a downward spiral, where the already-wobbling educational system is sustaining an attack that it doesn’t have the resources to recover from. The individual students’ attacks against their teachers and their schools’ grading systems might appear to momentarily succeed, but they will win the battle and lose the war.\n\n## Spamming “For Good”?\n\nUsually when we talk about someone unilaterally choosing to enter into an adversarial relationship, that’s an “attack” and for good reasons we have a negative impression of the attacker. However, I would be remiss if I did not point out that there are some cases where the relationship was already adversarial; just because you’re the attacker doesn’t mean that you are evil.\n\nFor example we might *imagine* use-cases like automatically filing appeals for\nprior authorizations against health insurance. It’s relatively\n[well-known](https://en.wikipedia.org/wiki/Delay,_Deny,_Defend) at this point\nthat the main way for-profit insurers maintain their margins is by denying\nclaims right up to the line of the policies themselves being fraud, so using a\nspamming tool to fight them might be entirely justifiable 2 in that case.\n\nSimilarly, using an LLM could be justified in a fight against a company refusing to honor a warranty. One could imagine using an LLM to immediately generate replies and escalations.\n\nHowever, even in imagined cases like these, the underlying problem is that the\ninsurers and the vendors already have a tremendous amount of structural power,\nso it is more likely that they will have the advantage in deploying a\ncommunications weapon like an LLM, as well as enacting policies to simply\nignore any LLM-based communication that you might submit. Worse, if these\nstrategies were to become widespread, they might provide an excuse to reject\n*any* communications by feeding them into an unreliable “[LLM\ndetector](https://www.npr.org/2025/12/16/nx-s1-5492397/ai-schools-teachers-students)”\nand issuing an automated “computer says no” even to hand-written\ncorrespondence.\n\nIt is also worth stressing that these cases are imagined, as compared to the very real coworker-abuse, spam, scam, fraud, and disinformation campaigns being waged in real life today.\n\nTherefore, while legitimate uses might exist, it’s hard to imagine that there’s anywhere they would be genuinely valuable and sustainable. In the best case “AI” will provide a temporary advantage for underdogs that will provoke an arms race which the resource-advantaged adversaries will win in the long run, in the worst case the arms race itself will cement permanent structural change that will make things worse.\n\n## “Search” By Stealing\n\nMost of the adversarial utility of “AI” is on the “write” side, since write-amplification is more obviously aggressive than reading. But the “read” side of LLMs — summarization and question-answering — can be a form of attack as well.\n\nTo begin with, [the act of reading\nitself](https://www.theregister.com/software/2025/08/29/ai-crawlers-destroying-websites-in-hunger-for-content/464120)\nis currently enormously destructive, but that’s arguably not a *fundamental*\naspect of this technology. They *could* set reasonable rate-limits and respect\nthings like `robots.txt`\n\n, as search engines have for decades now. They could\nalso refrain from committing [criminal\nlevels](https://www.npr.org/2025/09/05/nx-s1-5529404/anthropic-settlement-authors-copyright-ai)\n[of copyright\ninfringement](https://www.theguardian.com/technology/2025/jan/10/mark-zuckerberg-meta-books-ai-models-sarah-silverman).\nBut, today, using “AI” tools does suborn this sort of out-of-control crawling.\n\nMore insidiously, consider the scenario described in [this YouTube\nvideo](https://www.youtube.com/watch?v=8KQFgWdiudo). The LTT Bros decided to\ntry Linux again, and in the course of so doing, they had problems. When trying\nto solve these problems, they were faced with a choice: they could consult\nReddit, or they could ask an LLM. Asking an LLM would “gaslight the heck out\nof” them, but they still found it preferable, because they would at least get\nan answer without getting yelled at.\n\nInitially this sounds great. But it also means that you want to extract knowledge from a community, while mechanically eliding any values or norms that the community may want to impart as part of offering that knowledge. As someone who spent many years in a community tech support role, this is worrying. Many requests for support are people asking how to do things that will momentarily solve a superficial problem but create a long-term reliability problem or even an immediate security risk, that the question-asker doesn’t want to hear about. Consider the question “I’m tired of entering my password so much, how do I make it so my laptop unlocks automatically”. An obsequious chatbot will helpfully tell you how to do this without pushback.\n\nBut, this is also a sort of ethically murky area. The Linux community is\nsomewhat famously, for [many years\nnow](https://news.ycombinator.com/item?id=10332286), a toxic cesspool of\ngeneral hostility, misogyny, etc. It is certainly a good thing that people can\nget access to this knowledge without subjecting themselves to abuse. But it\nalso means that the people *with* the power and the privilege to change the\ncommunity for the better can just quietly withdraw, rather than fixing the\nproblems. It also means that the positive elements of culture cannot be\ntransmitted, and people will have no opportunity to learn about unknown\nunknowns.\n\nIn this case, the “adversarial” communication is with society. The thing that\nusing an LLM for search lets you do is withdraw from society and avoid forming\nany personal connections. There are some personal connections which are\npainful and annoying, and so that can feel like a momentary balm. But the need\nto make connections *in general* is, like, the concept of society itself.\n\n## Who Am I Hurting?\n\nLLMs are good at adversarial communication. They are *so* good at it, relative\nto their other benefits, that they will tend to *make* communications\nadversarial if you are not remaining vigilant about the possibility that it\nmight do so. My request to you, dear reader, if you are going to use such\ntools, is to always ask yourself, “who might I be hurting, if I use an LLM for\nthis?”\n\nIf you’re using an “AI”, who is its adversary? If you haven’t given it one\nyet, who might the “AI” *turn into* an adversary? Who might you overwhelm with\nan asymmetric amount of output, or, if you’re receiving information and not\nsending it, who are you taking that information from without consulting?\n\nFigure out the answers to these questions and conduct yourself accordingly; the answer might be “yourself”.\n\n## Acknowledgments\n\nThank you to [my patrons](/pages/patrons.html) who are supporting my writing on\nthis blog. If you like what you’ve read here and you’d\nlike to read more of it, or you’d like to support my [various open-source\nendeavors](https://github.com/glyph/), you can [support my work as a\nsponsor](/pages/patrons.html)!\n\n-\nOne of the reasons that software developers tend to prefer\n\n[greenfield](https://en.wikipedia.org/wiki/Greenfield_project)development is that when you are given a blank page, you can project your*own*specific understanding onto it. You can structure the codebase in a way that works for your brain, down to the variable naming conventions and the module layouts. LLM-assisted development makes everything into instant brownfield work, which makes developers instantly miserable; even those who are excited about the technology will frequently complain about how it feels like their agency has been stolen and their joy in the work has been diminished. But I digress.[↩](#fnref:1:adversarial-communication-2026-6) -\nModulo the massive amount of\n\n*other*externalities involved in using LLMs, of course, but I don’t have the time or energy to get into those here.[↩](#fnref:2:adversarial-communication-2026-6)", "url": "https://wpnews.pro/news/adversarial-communication", "canonical_source": "https://blog.glyph.im/2026/06/adversarial-communication.html", "published_at": "2026-06-24 01:36:05+00:00", "updated_at": "2026-06-24 03:23:39.012814+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-ethics", "ai-safety", "ai-agents", "generative-ai"], "entities": ["Cory Doctorow", "Glyph"], "alternates": {"html": "https://wpnews.pro/news/adversarial-communication", "markdown": "https://wpnews.pro/news/adversarial-communication.md", "text": "https://wpnews.pro/news/adversarial-communication.txt", "jsonld": "https://wpnews.pro/news/adversarial-communication.jsonld"}}