{"slug": "ai-scraping-on-the-cheap", "title": "AI Scraping on the Cheap", "summary": "A developer running My Mind is Racing, which tracks over 53,000 events and 22,000 organizations, details a cost-cutting approach to AI-assisted web scraping that minimizes LLM usage. The method prioritizes JSON-LD data, uses the Lynx command-line tool to convert HTML to text, and relies on Anthropic's Haiku model for extraction, with aggressive caching to reduce token spend and electricity use.", "body_md": "# AI Scraping on the Cheap\n\n## Table of Contents\n\n[My Mind is Racing](https://mymindisracing.com) tracks over 53,000 swim/bike/run/multisport events and more than 22,000 organizations. I’ve collected these mostly without AI. In order to aggregate all of the disparate data sources, I’ve had to lean on a combination of API requests and good old-fashioned (polite) web scraping. (I wish I could say the same of the botnets that have tried to hammer me into oblivion). In the cases where I do use an LLM, I try to minimize its use. I’ll map out my basic use case below.\n\n## [#](#do-i-even-need-ai)Do I even need AI?\n\nIn a lot of cases, I can get by without an LLM for the actual data processing. In that case, the LLM just writes the custom scraper or API integration, and I can hand off the real work to a polite UserAgent. LLMs are not only slow, but they are expensive. Cutting them out of the loop whenever possible keeps costs down and reduces the time required to get the data.\n\n## [#](#does-the-page-have-json-ld)Does the page have JSON-LD?\n\nIn the case where a data source does use [JSON-LD](https://json-ld.org/), my job is a lot easier. Often this has all or most of the data that I need. So, even if the page that is being scraped is a one-off, it may already have enough data to get by. If it has JSON-LD and that’s still not enough, we can save it and use it to complement the LLM’s results.\n\n[Lynx lynx - 05](https://commons.wikimedia.org/wiki/File:Lynx_lynx_-_05.jpg) by [Carlos Delgado](https://commons.wikimedia.org/wiki/User:Kadellar), licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/).\n\n## [#](#do-i-need-to-parse-the-html)Do I need to parse the HTML?\n\nI can certainly just hand a page of HTML to the LLM, but is there a point? It’s going to have a lot of classes, attributes, inline styles, scripts and other things that don’t help me get what I need. At this point, a lot of people reach for an HTML to Markdown renderer, and that can be a perfectly serviceable solution. My past flirtations with this have not been great, particularly when it comes to turning an HTML table to Markdown. If the table has columns which contain newlines, the conversion gets a bit tricky. I assume this has been improved in the meantime, but I like to do it a different way. [lynx](https://lynx.invisible-island.net/) is a command-line tool, which is easily installable and has been tuned for decades to turn all sorts of funky HTML into human-readable text. I find that taking HTML and processing it via `lynx` has been a pretty excellent solution for my own needs.\n\n## [#](#do-i-need-the-latest-model)Do I need the latest model?\n\nIf your use case is fairly straightforward, you don’t need the biggest hammer available. For extraction of events, dates, addresses and descriptions, `haiku` has served me perfectly well, and it is relatively cheap compared with more advanced models. It’s not the new hotness, but it gets the job done. My token spend has been less than what I spend on fancy cups of coffee.\n\n## [#](#cache-aggressively)Cache aggressively\n\nLastly, I jump through a lot of hoops to cache aggressively and avoid\nre-running the same data through an LLM. The kind of data that I need does\nchange, but it doesn’t change that much. So, I set an appropriate cache\nlifetime based on the source, and I don’t ask `haiku` to do the same thing\ntwice. That’s not only kinder to my bank account, but it uses less electricity.\n\nThere are going to be lots of ways to tackle this problem, but if I’m honest, I have a soft spot for `lynx`. It reminds me of my early days in University when I was studying in [Freiburg](https://uni-freiburg.de/en/), using `talk` to communicate, `pine` to read and write email and `lynx` to try to get information from the very few available web pages. I love that `lynx` is still around, and I’m not going to apologize for using it.\n\nRelated posts:", "url": "https://wpnews.pro/news/ai-scraping-on-the-cheap", "canonical_source": "https://www.olafalders.com/2026/09/08/ai-scraping-on-the-cheap/", "published_at": "2026-09-08 00:00:00+00:00", "updated_at": "2026-09-08 04:00:18.010081+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "artificial-intelligence"], "entities": ["My Mind is Racing", "Anthropic", "Haiku", "Lynx"], "alternates": {"html": "https://wpnews.pro/news/ai-scraping-on-the-cheap", "markdown": "https://wpnews.pro/news/ai-scraping-on-the-cheap.md", "text": "https://wpnews.pro/news/ai-scraping-on-the-cheap.txt", "jsonld": "https://wpnews.pro/news/ai-scraping-on-the-cheap.jsonld"}}