AI Scraping on the Cheap

A developer running My Mind is Racing, which tracks over 53,000 events and 22,000 organizations, details a cost-cutting approach to AI-assisted web scraping that minimizes LLM usage. The method prioritizes JSON-LD data, uses the Lynx command-line tool to convert HTML to text, and relies on Anthropic's Haiku model for extraction, with aggressive caching to reduce token spend and electricity use.

AI Scraping on the Cheap Table of Contents My Mind is Racing https://mymindisracing.com tracks over 53,000 swim/bike/run/multisport events and more than 22,000 organizations. I’ve collected these mostly without AI. In order to aggregate all of the disparate data sources, I’ve had to lean on a combination of API requests and good old-fashioned polite web scraping. I wish I could say the same of the botnets that have tried to hammer me into oblivion . In the cases where I do use an LLM, I try to minimize its use. I’ll map out my basic use case below. do-i-even-need-ai Do I even need AI? In a lot of cases, I can get by without an LLM for the actual data processing. In that case, the LLM just writes the custom scraper or API integration, and I can hand off the real work to a polite UserAgent. LLMs are not only slow, but they are expensive. Cutting them out of the loop whenever possible keeps costs down and reduces the time required to get the data. does-the-page-have-json-ld Does the page have JSON-LD? In the case where a data source does use JSON-LD https://json-ld.org/ , my job is a lot easier. Often this has all or most of the data that I need. So, even if the page that is being scraped is a one-off, it may already have enough data to get by. If it has JSON-LD and that’s still not enough, we can save it and use it to complement the LLM’s results. Lynx lynx - 05 https://commons.wikimedia.org/wiki/File:Lynx lynx - 05.jpg by Carlos Delgado https://commons.wikimedia.org/wiki/User:Kadellar , licensed under CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0/ . do-i-need-to-parse-the-html Do I need to parse the HTML? I can certainly just hand a page of HTML to the LLM, but is there a point? It’s going to have a lot of classes, attributes, inline styles, scripts and other things that don’t help me get what I need. At this point, a lot of people reach for an HTML to Markdown renderer, and that can be a perfectly serviceable solution. My past flirtations with this have not been great, particularly when it comes to turning an HTML table to Markdown. If the table has columns which contain newlines, the conversion gets a bit tricky. I assume this has been improved in the meantime, but I like to do it a different way. lynx https://lynx.invisible-island.net/ is a command-line tool, which is easily installable and has been tuned for decades to turn all sorts of funky HTML into human-readable text. I find that taking HTML and processing it via lynx has been a pretty excellent solution for my own needs. do-i-need-the-latest-model Do I need the latest model? If your use case is fairly straightforward, you don’t need the biggest hammer available. For extraction of events, dates, addresses and descriptions, haiku has served me perfectly well, and it is relatively cheap compared with more advanced models. It’s not the new hotness, but it gets the job done. My token spend has been less than what I spend on fancy cups of coffee. cache-aggressively Cache aggressively Lastly, I jump through a lot of hoops to cache aggressively and avoid re-running the same data through an LLM. The kind of data that I need does change, but it doesn’t change that much. So, I set an appropriate cache lifetime based on the source, and I don’t ask haiku to do the same thing twice. That’s not only kinder to my bank account, but it uses less electricity. There are going to be lots of ways to tackle this problem, but if I’m honest, I have a soft spot for lynx . It reminds me of my early days in University when I was studying in Freiburg https://uni-freiburg.de/en/ , using talk to communicate, pine to read and write email and lynx to try to get information from the very few available web pages. I love that lynx is still around, and I’m not going to apologize for using it. Related posts: