{"slug": "shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt", "title": "ShieldFont: Bludgeoning AI Scrapers that Disrespect Robots.txt", "summary": "ShieldFont, a new tool from shieldfont.org, uses ligature-based typography to replace about a quarter of words in HTML text with nonsensical but grammatically valid versions, rendering normally in browsers while confusing AI scrapers that parse raw HTML. The white paper details the method, which aims to deter LLM training scrapers that ignore robots.txt, though it also affects screen readers and legitimate search engine bots. The approach joins other anti-scraper tactics like Nepenthes and Cloudflare's AI Labyrinth.", "body_md": "In the more innocent days of the World Wide Web you could simply put a `robots.txt`\n\nfile in the root of your website that search engine indexing bots and similar would consult for the indexing wishes of the site owner. In this brave new world of LLM training data indexing such pleasantries are however rarely respected, leaving site owners to resort to increasingly more involved ways to bludgeon so-called AI scrapers, with [ShieldFont](https://shieldfont.org/) being one of the most recent methods.\n\nIts basic functioning is detailed in [the white paper](https://shieldfont.org/white-paper/), explaining their use of [ligatures.](https://en.wikipedia.org/wiki/Ligature_(writing)) These are normally used to join multiple graphemes or letters into a single glyph which are rendered in the final text. By substituting about a quarter of the words in a text with such ligature-based versions in an intelligent, dictionary-based manner, the HTML version – as typically parsed by a scraper – will read as grammatically valid but nonsensical text, while the rendered font version will look normal.\n\nNaturally, there are some disadvantages to this, such as screen readers for the visually impaired needing to also use the rendered font version, and it’s just as effective on legitimate search engine indexing bots. That said, if you apply this to static, archived content, or content marked as ‘do not follow’ in said `robots.txt`\n\n, then it might just be one way to make ChatGPT and friends spit out really funny output in the future now that the novelty of wood glue on pizza and eating rocks has somewhat worn off.\n\nWhile LLM scrapers can adapt to this by also parsing the rendered text, this makes the scraping effort significantly more expensive. Together with maze traps like [Nepenthes](https://hackaday.com/2025/01/23/trap-naughty-web-crawlers-in-digestive-juices-with-nepenthes/) and [Cloudflare’s offerings](https://hackaday.com/2025/03/24/cloudflares-ai-labyrinth-wants-bad-bots-to-get-endlessly-lost/) that seek to keep these scrapers busy scraping dynamically generated content through infinite linked pages, the tools available to combat the [menace of these scrapers](https://hackaday.com/2026/07/25/how-film-industry-data-website-the-numbers-com-got-mauled-by-bots/) keep developing.", "url": "https://wpnews.pro/news/shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt", "canonical_source": "https://hackaday.com/2026/08/14/shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt/", "published_at": "2026-08-14 11:00:48+00:00", "updated_at": "2026-08-14 11:06:04.632736+00:00", "lang": "en", "topics": ["ai-tools", "ai-policy"], "entities": ["ShieldFont", "Nepenthes", "Cloudflare"], "alternates": {"html": "https://wpnews.pro/news/shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt", "markdown": "https://wpnews.pro/news/shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt.md", "text": "https://wpnews.pro/news/shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt.txt", "jsonld": "https://wpnews.pro/news/shieldfont-bludgeoning-ai-scrapers-that-disrespect-robots-txt.jsonld"}}