Table of Contents #
My Mind is Racing tracks over 53,000 swim/bike/run/multisport events and more than 22,000 organizations. I’ve collected these mostly without AI. In order to aggregate all of the disparate data sources, I’ve had to lean on a combination of API requests and good old-fashioned (polite) web scraping. (I wish I could say the same of the botnets that have tried to hammer me into oblivion). In the cases where I do use an LLM, I try to minimize its use. I’ll map out my basic use case below.
#Do I even need AI? #
In a lot of cases, I can get by without an LLM for the actual data processing. In that case, the LLM just writes the custom scraper or API integration, and I can hand off the real work to a polite UserAgent. LLMs are not only slow, but they are expensive. Cutting them out of the loop whenever possible keeps costs down and reduces the time required to get the data.
#Does the page have JSON-LD? #
In the case where a data source does use JSON-LD, my job is a lot easier. Often this has all or most of the data that I need. So, even if the page that is being scraped is a one-off, it may already have enough data to get by. If it has JSON-LD and that’s still not enough, we can save it and use it to complement the LLM’s results.
Lynx lynx - 05 by Carlos Delgado, licensed under CC BY-SA 3.0.
#Do I need to parse the HTML? #
I can certainly just hand a page of HTML to the LLM, but is there a point? It’s going to have a lot of classes, attributes, inline styles, scripts and other things that don’t help me get what I need. At this point, a lot of people reach for an HTML to Markdown renderer, and that can be a perfectly serviceable solution. My past flirtations with this have not been great, particularly when it comes to turning an HTML table to Markdown. If the table has columns which contain newlines, the conversion gets a bit tricky. I assume this has been improved in the meantime, but I like to do it a different way. lynx is a command-line tool, which is easily installable and has been tuned for decades to turn all sorts of funky HTML into human-readable text. I find that taking HTML and processing it via lynx has been a pretty excellent solution for my own needs.
## [#](#do-i-need-the-latest-model)Do I need the latest model?
If your use case is fairly straightforward, you don’t need the biggest hammer available. For extraction of events, dates, addresses and descriptions, `haiku` has served me perfectly well, and it is relatively cheap compared with more advanced models. It’s not the new hotness, but it gets the job done. My token spend has been less than what I spend on fancy cups of coffee.
## [#](#cache-aggressively)Cache aggressively
Lastly, I jump through a lot of hoops to cache aggressively and avoid
re-running the same data through an LLM. The kind of data that I need does
change, but it doesn’t change that much. So, I set an appropriate cache
lifetime based on the source, and I don’t ask haiku to do the same thing
twice. That’s not only kinder to my bank account, but it uses less electricity.
There are going to be lots of ways to tackle this problem, but if I’m honest, I have a soft spot for lynx. It reminds me of my early days in University when I was studying in Freiburg, using talk to communicate, pine to read and write email and lynx to try to get information from the very few available web pages. I love that lynx is still around, and I’m not going to apologize for using it.
Related posts: