Why Claude Can't Read Reddit

Reddit's robots.txt file, updated on 1 July 2024, blocks Claude Code's fetch tool from accessing the site, a move that also cut off Bing, DuckDuckGo, Mojeek, and Qwant while Google reportedly pays around $60 million a year for access. The block follows Reddit's lawsuit against Anthropic filed on 4 June 2025, alleging unauthorized use of Reddit content for training Claude, which Anthropic disputes. The author notes that the error message reflects a licensing negotiation rather than a technical bug.

Why Claude Can't Read Reddit I asked Claude Code to take a look at r/AngJobs — my own subreddit. It came back with this: Then it tried old.reddit.com , the usual escape hatch. Same wall. The part I liked: it did not invent an answer. It said "Reddit is blocked from here" and went to the API instead. But it left me curious about why . The answer has nothing to do with Claude. It is robots.txt, and it is deliberate Claude Code's fetch tool obeys robots.txt . Reddit's robots.txt says no. That file changed on 1 July 2024 . Reddit's chief legal officer had announced it in r/redditdev a week earlier: if you use an automated agent to access Reddit, you need to abide by their terms and you need to talk to them first. "We believe in the open internet, but we do not believe in the misuse of public content." The block landed hard. Bing, DuckDuckGo, Mojeek and Qwant all lost Reddit the same day. One crawler kept it — Google, which reportedly pays around $60 million a year for the privilege. That is the whole story in one sentence. Reddit's asset was never the software. It is twenty years of human beings answering each other's questions, and that turns out to be some of the best training data on earth. So Reddit stopped giving it away and started selling it. The Anthropic part There is a sharper edge here, and I might as well say it plainly. On 4 June 2025 , Reddit sued Anthropic in San Francisco Superior Court. The complaint alleges that Anthropic trained Claude on Reddit content from December 2021 onward, and kept accessing the site more than 100,000 times after saying its bots had been blocked. Reddit is asking for damages and for an order barring the use of any Reddit-derived data. Anthropic disputes the claims and says it will defend itself vigorously. Reddit has since filed against Perplexity and three data-scraping firms as well. Nothing is decided. These are allegations, and courts are slow. But it means that when my terminal prints "Claude Code is unable to fetch from www.reddit.com" , I am not looking at a bug report. I am looking at the edge of a licensing negotiation, rendered as a red line of text. Three things I took from a two-line error The open web is being fenced, quickly. The assumption I built a whole career on — that a public URL is readable by anything that asks politely — is expiring. Build for the fence. The content your users write is an asset. Reddit's moat is other people's sentences. If you are building a product, what your users generate is worth more this year than it was last year, and the terms you publish today are what you get to enforce later. Prefer the official API. The workaround for a blocked crawl was never a different subdomain. It is an API, with a key, under terms someone agreed to. My agent worked that out faster than I did.