Web Scraping with Claude Code: A Working Workflow with Gaffa

A developer used Claude Code with Gaffa's browser-automation skills to build a working web-scraping workflow against books.toscrape.com, going from plain-language goal to discovery, a paginated scraper, CSV export and verification. The /gaffa-find skill located the Mystery category page, its 32 books across 2 pages and the field selectors in 9 credits, and the developer then stress-tested the scraper to surface silent failures such as missing rows that produce no exceptions.

Scraping scripts rarely fail on day one. They fail quietly on day ten. The script runs. No exception. A CSV appears. And somewhere in it, 24 rows are missing. Nobody notices until the data is already in a dashboard. If you're: This matters. I gave Claude Code a plain-language scraping goal and Gaffa's AI skills, then let it go from zero to a working web scraping workflow: discovery, a paginated scraper, CSV export and verification. Then I tried to break it. Every number, error and output below comes from real runs. Writing a scraper is the easy part. Claude can do it in one shot. The hard part is everything around it: None of these throw an error. The script finishes, the CSV looks fine, and the data is wrong. You don't find these bugs by reading code. You find them by running the scraper against the real site and checking the output against something you trust. The fix isn't a smarter prompt. It's a workflow where the agent can discover the site, build against the real pages, and debug failures from real request data. With checks that tell you when the output is wrong. Gaffa's AI skills cover exactly those three steps: /gaffa-find gaffa-authoring /gaffa-debug brq ... and it diagnoses what went wrong Under the hood, Gaffa https://gaffa.dev is a browser automation REST API: you send a URL and a list of actions click , wait , capture element , loop ... and it runs them in a real browser. 👉 You can create an account at gaffa.dev https://gaffa.dev and follow along. If you read my previous article on connecting Claude Code to Gaffa https://kevinmeneses.com/en/articles/connect-claude-code-gaffa-browser-automation , you already have this. If not, it's one command in your project folder: npx @gaffa-dev/cli install --tools=claude-code --scope=project -y Installed the gaffa skills version 0.9.0 into 1 place: ./.claude\skills project gaffa-authoring, gaffa-debug, gaffa-find, gaffa-support Registered the gaffa-docs MCP server: Claude Code ./.mcp.json added Then set GAFFA API KEY as an environment variable. The skills never read the key from anywhere else. The target for this test is books.toscrape.com https://books.toscrape.com , a sandbox catalogue built for scraping practice: 1,000 books, 50 categories, classic "next" pagination. I didn't give Claude a URL for the category, a selector or a page count. Just the homepage and a goal: /gaffa-find https://books.toscrape.com/ — Goal: I want to collect every book in the 'Mystery' category. Find the category page URL, how many books it contains, how many listing pages it spans and how pagination works, and which fields each book card exposes title, price, rating, availability, link . Rendered from the real session transcript. The skill runs a loop: recon, hypothesis, targeted extraction, validation. With hard caps: 15 requests and 50 credits per run, 5 iterations maximum. For recon, it sent a browser request that turns the homepage into Markdown. 3 credits, 5.8 seconds, and the answer was in the sidebar: - Mystery catalogue/category/books/mystery 3/index.html From there it captured the category page twice: once as Markdown to read "32 results" and "Page 1 of 2" and once as raw HTML to see the markup the Markdown strips out, like the star rating . Then it checked page 2 to confirm where pagination ends. It also wrote a reasoning log to disk, .gaffa-find-<timestamp .log , one entry per iteration: hypothesis, request id, credits, decision. You can audit exactly how it reached its answer. The final result: Terminal state: success. 32 books, 2 pages, pagination through a relative li.next a link that disappears on the last page, and a selector for each field. One detail I wouldn't have caught: the visible title text is truncated with "…", so the full title has to come from the link's title attribute. Cost of discovery: 9 credits. Same Claude session, so it kept everything it had learned. This time I asked for a real workflow: Rewrite it as books to csv.py: one single Gaffa browser request that paginates with the loop action. Price as a number, rating as an integer 1-5, absolute product URL. Write a CSV. Add --category-url. Run it on Mystery, verify 32 rows with no duplicates, then run it on a category with 3+ pages and verify the count matches the 'N results' text. Claude pulled the loop reference through the Gaffa docs MCP server approved this time , then built the request. This is the heart of the workflow. Instead of one request per page, a single request captures the results banner once and then loops: capture the product grid, click "next", repeat until there is no "next" link. "actions": {"type": "capture element", "selector": "form.form-horizontal", "custom id": "results-banner"}, { "type": "loop", "custom id": "pagination-loop", "max iterations": max pages, "timeout": loop timeout, "stop on fail": True, "continue on fail": True, the last page has no "next": that's how the loop ends "actions": {"type": "wait", "time": 800, "custom id": "settle"}, {"type": "capture element", "selector": "ol.row", "custom id": "page-cards"}, {"type": "click", "selector": "li.next a", "custom id": "next-page", "timeout": 15000}, , }, , Parsing happens in plain Python with regular expressions over the captured HTML. No LLM extraction. The skill steers Claude away from Gaffa's parse json action for structured pages like this one, because a deterministic parser is cheaper and returns the same output every run. The most valuable line in the script is not the scraper. It's this: status = "OK" if reported total == len books else "MISMATCH" print f"page says {reported total} results; scraped {len books } - {status}" if status == "MISMATCH": sys.exit 1 The workflow compares its own output with the number the website reports. If they differ, it fails loudly. Here is how phase 2 went: Attempt 1: Mystery. 32 of 32 books, 32 unique URLs, banner matched. 3 credits. Attempt 2: Nonfiction 110 books, 6 pages . request: brq VzZhqRu7EsTn47GhMyLqkSN76iJDA6 pages captured: 6 books written: 86 - nonfiction.csv page says 110 results; scraped 86 - MISMATCH All 6 pages were captured, but 24 books were missing. Claude broke it down page by page: page 1: articles=20 page 2: articles=20 page 3: articles=20 page 4: articles=12 page 5: articles=14 page 6: articles=2 Pages 4 to 6 were captured while the browser was still loading them, right after the click. Claude added a short wait at the start of each iteration and re-ran: 110 of 110 . Then re-ran Mystery to make sure nothing regressed. Without that count check, the script would have exited 0 and handed me a CSV missing 22% of the data. Cost of phase 2: 6 billed requests, 18 credits. Claude said it was done. I didn't take its word for it. I ran the script myself on categories it had never seen. Travel 11 books and Fiction 65 books, 4 pages both matched their counts. Then I ran Nonfiction again: php pages captured: 2 books written: 40 - nonfiction.csv page says 110 results; scraped 40 - MISMATCH The loop stopped after page 2. The count check caught it again. Failed and partial requests are part of scraping any live website. Pages get slow, layouts change, content loads late. What matters is how quickly you find out why. That's the job of the third skill. I passed the request id to Claude: /gaffa-debug brq VzZj6m1Mqsb31bZEevh2ZHdUNVFxyh — it only got 40 of 110 books MISMATCH . Diagnose why the loop stopped early, apply a minimal fix that does not rely on luck, then verify on Nonfiction twice and Mystery once. The diagnosis was precise: action timed out . page-2.html directly free, no Gaffa call . It has a "next" link to page 3. The patch was one value: raise the click's timeout from 5,000 to 15,000 ms, the same value Gaffa's own pagination examples use. A real last page still ends the loop, just 15 seconds later. Verification: Nonfiction 110 of 110 twice in a row with identical rows, Mystery 32 of 32. 13 credits. I ran the final script one last time on my own: | Category | Pages | Books scraped | Page says | Status | Credits | |---|---|---|---|---|---| | Mystery | 2 | 32 | 32 | OK | 3 | | Travel | 1 | 11 | 11 | OK | 2 | | Fiction | 4 | 65 | 65 | OK | 4 | | Nonfiction | 6 | 110 | 110 | OK | 3 to 5 | 218 books, 4 categories, 0 duplicates. Nonfiction, 6 pages in a single request, takes about 42 seconds end to end. The CSV is clean, typed data ready for pandas or a spreadsheet: title,price,rating,availability,url Worlds Elsewhere: Journeys Around Shakespeare’s Globe,40.3,5,In stock,https://books.toscrape.com/catalogue/worlds-elsewhere-journeys-around-shakespeares-globe 972/index.html The Five Love Languages: How to Express Heartfelt Commitment to Your Mate,31.05,3,In stock,https://books.toscrape.com/catalogue/the-five-love-languages-how-to-express-heartfelt-commitment-to-your-mate 969/index.html Reasons to Stay Alive,26.41,2,In stock,https://books.toscrape.com/catalogue/reasons-to-stay-alive 959/index.html | Phase | Gaffa credits | |---|---| | /gaffa-find discovery | 9 | | Building the workflow | 18 | | /gaffa-debug fix and verification | 13 | | Claude Code total | 40 | | My own test runs | 20 | Claude Code ran on my Claude Pro subscription. From here you can: Three lessons from this run, all of which apply outside Gaffa: Put the verification in the script, not in your head. The "N results" check caught two silent failures that would otherwise have shipped. If the site tells you how many items it has, use it. Debug from request data, not from guesses. /gaffa-debug didn't guess. It read how many iterations ran, which action failed and how long the page took to load, then checked the live page before changing a single value. Fixed waits are guesses. The 800 ms wait happened to work, and the 5 s click timeout didn't when the site was slow. When you can, wait on a selector or give timeouts real margin. loop scraped a 6-page category in about 42 seconds, and a built-in count check caught two silent data losses. Claude Code writes scrapers fast. That was never the bottleneck. The bottleneck was knowing whether the data is right. Give the agent a real browser, skills that force it to look before it writes, and a workflow that checks its own output, and web scraping with Claude Code becomes something you can actually trust. 👉 Try it: create an account at gaffa.dev https://gaffa.dev , run npx @gaffa-dev/cli install , and start with /gaffa-find on a site you need data from. Looking for technical content for your company? I can help — LinkedIn https://www.linkedin.com/in/kevin-meneses-gonzalez/ · kevinmenesesgonzalez@gmail.com mailto:kevinmenesesgonzalez@gmail.com