{"slug": "web-scraping-with-claude-code-a-working-workflow-with-gaffa", "title": "Web Scraping with Claude Code: A Working Workflow with Gaffa", "summary": "A developer used Claude Code with Gaffa's browser-automation skills to build a working web-scraping workflow against books.toscrape.com, going from plain-language goal to discovery, a paginated scraper, CSV export and verification. The /gaffa-find skill located the Mystery category page, its 32 books across 2 pages and the field selectors in 9 credits, and the developer then stress-tested the scraper to surface silent failures such as missing rows that produce no exceptions.", "body_md": "Scraping scripts rarely fail on day one.\n\nThey fail quietly on day ten.\n\nThe script runs. No exception. A CSV appears. And somewhere in it, 24 rows are missing. Nobody notices until the data is already in a dashboard.\n\nIf you're:\n\nThis matters.\n\nI gave Claude Code a plain-language scraping goal and Gaffa's AI skills, then let it go from zero to a working web scraping workflow: discovery, a paginated scraper, CSV export and verification. Then I tried to break it.\n\nEvery number, error and output below comes from real runs.\n\nWriting a scraper is the easy part. Claude can do it in one shot.\n\nThe hard part is everything around it:\n\nNone of these throw an error. The script finishes, the CSV looks fine, and the data is wrong.\n\nYou don't find these bugs by reading code. You find them by running the scraper against the real site and checking the output against something you trust.\n\nThe fix isn't a smarter prompt.\n\nIt's a workflow where the agent can **discover** the site, **build** against the real pages, and **debug** failures from real request data. With checks that tell you when the output is wrong.\n\nGaffa's AI skills cover exactly those three steps:\n\n`/gaffa-find`` gaffa-authoring``/gaffa-debug`` brq_...`) and it diagnoses what went wrong\nUnder the hood, [Gaffa](https://gaffa.dev) is a browser automation REST API: you send a URL and a list of actions (`click`, `wait`, `capture_element`, `loop`...) and it runs them in a real browser.\n\n👉 You can create an account at [gaffa.dev](https://gaffa.dev) and follow along.\n\nIf you read my previous article on [connecting Claude Code to Gaffa](https://kevinmeneses.com/en/articles/connect-claude-code-gaffa-browser-automation), you already have this. If not, it's one command in your project folder:\n\n```\nnpx @gaffa-dev/cli install --tools=claude-code --scope=project -y\nInstalled the gaffa skills (version 0.9.0) into 1 place:\n  ./.claude\\skills  (project)  gaffa-authoring, gaffa-debug, gaffa-find, gaffa-support\n\nRegistered the gaffa-docs MCP server:\n  Claude Code  ./.mcp.json  added\n```\n\nThen set `GAFFA_API_KEY` as an environment variable. The skills never read the key from anywhere else.\n\nThe target for this test is [books.toscrape.com](https://books.toscrape.com), a sandbox catalogue built for scraping practice: 1,000 books, 50 categories, classic \"next\" pagination.\n\nI didn't give Claude a URL for the category, a selector or a page count. Just the homepage and a goal:\n\n```\n/gaffa-find https://books.toscrape.com/ — Goal: I want to collect every book\nin the 'Mystery' category. Find the category page URL, how many books it\ncontains, how many listing pages it spans and how pagination works, and which\nfields each book card exposes (title, price, rating, availability, link).\n```\n\n*Rendered from the real session transcript.*\n\nThe skill runs a loop: recon, hypothesis, targeted extraction, validation. With hard caps: 15 requests and 50 credits per run, 5 iterations maximum.\n\nFor recon, it sent a browser request that turns the homepage into Markdown. 3 credits, 5.8 seconds, and the answer was in the sidebar:\n\n```\n- [Mystery](catalogue/category/books/mystery_3/index.html)\n```\n\nFrom there it captured the category page twice: once as Markdown (to read \"32 results\" and \"Page 1 of 2\") and once as raw HTML (to see the markup the Markdown strips out, like the star rating). Then it checked page 2 to confirm where pagination ends.\n\nIt also wrote a reasoning log to disk, `.gaffa-find-<timestamp>.log`, one entry per iteration: hypothesis, request id, credits, decision. You can audit exactly how it reached its answer.\n\nThe final result:\n\n**Terminal state: success.** 32 books, 2 pages, pagination through a relative `li.next a` link that disappears on the last page, and a selector for each field. One detail I wouldn't have caught: the visible title text is truncated with \"…\", so the full title has to come from the link's `title` attribute.\n\nCost of discovery: **9 credits.**\n\nSame Claude session, so it kept everything it had learned. This time I asked for a real workflow:\n\n```\nRewrite it as books_to_csv.py: one single Gaffa browser request that\npaginates with the loop action. Price as a number, rating as an integer\n1-5, absolute product URL. Write a CSV. Add --category-url. Run it on\nMystery, verify 32 rows with no duplicates, then run it on a category\nwith 3+ pages and verify the count matches the 'N results' text.\n```\n\nClaude pulled the `loop` reference through the Gaffa docs MCP server (approved this time), then built the request.\n\nThis is the heart of the workflow. Instead of one request per page, a single request captures the results banner once and then loops: capture the product grid, click \"next\", repeat until there is no \"next\" link.\n\n```\n\"actions\": [\n    {\"type\": \"capture_element\", \"selector\": \"form.form-horizontal\", \"custom_id\": \"results-banner\"},\n    {\n        \"type\": \"loop\",\n        \"custom_id\": \"pagination-loop\",\n        \"max_iterations\": max_pages,\n        \"timeout\": loop_timeout,\n        \"stop_on_fail\": True,\n        \"continue_on_fail\": True,  # the last page has no \"next\": that's how the loop ends\n        \"actions\": [\n            {\"type\": \"wait\", \"time\": 800, \"custom_id\": \"settle\"},\n            {\"type\": \"capture_element\", \"selector\": \"ol.row\", \"custom_id\": \"page-cards\"},\n            {\"type\": \"click\", \"selector\": \"li.next a\", \"custom_id\": \"next-page\", \"timeout\": 15000},\n        ],\n    },\n],\n```\n\nParsing happens in plain Python with regular expressions over the captured HTML. No LLM extraction. The skill steers Claude away from Gaffa's `parse_json` action for structured pages like this one, because a deterministic parser is cheaper and returns the same output every run.\n\nThe most valuable line in the script is not the scraper. It's this:\n\n```\nstatus = \"OK\" if reported_total == len(books) else \"MISMATCH\"\nprint(f\"page says {reported_total} results; scraped {len(books)} -> {status}\")\nif status == \"MISMATCH\":\n    sys.exit(1)\n```\n\nThe workflow compares its own output with the number the website reports. If they differ, it fails loudly.\n\nHere is how phase 2 went:\n\n**Attempt 1: Mystery.** 32 of 32 books, 32 unique URLs, banner matched. 3 credits.\n\n**Attempt 2: Nonfiction (110 books, 6 pages).**\n\n```\nrequest: brq_VzZhqRu7EsTn47GhMyLqkSN76iJDA6\npages captured: 6\nbooks written: 86 -> nonfiction.csv\npage says 110 results; scraped 86 -> MISMATCH\n```\n\nAll 6 pages were captured, but 24 books were missing. Claude broke it down page by page:\n\n```\npage 1: articles=20\npage 2: articles=20\npage 3: articles=20\npage 4: articles=12\npage 5: articles=14\npage 6: articles=2\n```\n\nPages 4 to 6 were captured while the browser was still loading them, right after the click. Claude added a short `wait` at the start of each iteration and re-ran: **110 of 110**. Then re-ran Mystery to make sure nothing regressed.\n\nWithout that count check, the script would have exited 0 and handed me a CSV missing 22% of the data.\n\nCost of phase 2: **6 billed requests, 18 credits.**\n\nClaude said it was done. I didn't take its word for it. I ran the script myself on categories it had never seen.\n\nTravel (11 books) and Fiction (65 books, 4 pages) both matched their counts. Then I ran Nonfiction again:\n\n``` php\npages captured: 2\nbooks written: 40 -> nonfiction.csv\npage says 110 results; scraped 40 -> MISMATCH\n```\n\nThe loop stopped after page 2. The count check caught it again.\n\nFailed and partial requests are part of scraping any live website. Pages get slow, layouts change, content loads late. What matters is how quickly you find out why. That's the job of the third skill.\n\nI passed the request id to Claude:\n\n```\n/gaffa-debug brq_VzZj6m1Mqsb31bZEevh2ZHdUNVFxyh — it only got 40 of 110\nbooks (MISMATCH). Diagnose why the loop stopped early, apply a minimal fix\nthat does not rely on luck, then verify on Nonfiction twice and Mystery once.\n```\n\nThe diagnosis was precise:\n\n`action_timed_out`.` page-2.html` directly (free, no Gaffa call). It has a \"next\" link to page 3. The patch was one value: raise the click's `timeout` from 5,000 to 15,000 ms, the same value Gaffa's own pagination examples use. A real last page still ends the loop, just 15 seconds later.\n\nVerification: Nonfiction 110 of 110 twice in a row with identical rows, Mystery 32 of 32. **13 credits.**\n\nI ran the final script one last time on my own:\n\n| Category | Pages | Books scraped | Page says | Status | Credits | \n|---|---|---|---|---|---|\n| Mystery | 2 | 32 | 32 | OK | 3 | \n| Travel | 1 | 11 | 11 | OK | 2 | \n| Fiction | 4 | 65 | 65 | OK | 4 | \n| Nonfiction | 6 | 110 | 110 | OK | 3 to 5 | \n\n**218 books, 4 categories, 0 duplicates.** Nonfiction, 6 pages in a single request, takes about 42 seconds end to end.\n\nThe CSV is clean, typed data ready for pandas or a spreadsheet:\n\n```\ntitle,price,rating,availability,url\nWorlds Elsewhere: Journeys Around Shakespeare’s Globe,40.3,5,In stock,https://books.toscrape.com/catalogue/worlds-elsewhere-journeys-around-shakespeares-globe_972/index.html\nThe Five Love Languages: How to Express Heartfelt Commitment to Your Mate,31.05,3,In stock,https://books.toscrape.com/catalogue/the-five-love-languages-how-to-express-heartfelt-commitment-to-your-mate_969/index.html\nReasons to Stay Alive,26.41,2,In stock,https://books.toscrape.com/catalogue/reasons-to-stay-alive_959/index.html\n```\n\n| Phase | Gaffa credits | \n|---|---|\n| `/gaffa-find` discovery | 9 | \n| Building the workflow | 18 | \n| `/gaffa-debug` fix and verification | 13 | \n| **Claude Code total** | **40** | \n| My own test runs | 20 | \n\nClaude Code ran on my Claude Pro subscription.\n\nFrom here you can:\n\nThree lessons from this run, all of which apply outside Gaffa:\n\n**Put the verification in the script, not in your head.** The \"N results\" check caught two silent failures that would otherwise have shipped. If the site tells you how many items it has, use it.\n\n**Debug from request data, not from guesses.** `/gaffa-debug` didn't guess. It read how many iterations ran, which action failed and how long the page took to load, then checked the live page before changing a single value.\n\n**Fixed waits are guesses.** The 800 ms `wait` happened to work, and the 5 s click timeout didn't when the site was slow. When you can, wait on a selector or give timeouts real margin.\n\n`loop` scraped a 6-page category in about 42 seconds, and a built-in count check caught two silent data losses.\nClaude Code writes scrapers fast. That was never the bottleneck.\n\nThe bottleneck was knowing whether the data is right.\n\nGive the agent a real browser, skills that force it to look before it writes, and a workflow that checks its own output, and web scraping with Claude Code becomes something you can actually trust.\n\n👉 Try it: create an account at [gaffa.dev](https://gaffa.dev), run `npx @gaffa-dev/cli install`, and start with `/gaffa-find` on a site you need data from.\n\n*Looking for technical content for your company? I can help — [LinkedIn](https://www.linkedin.com/in/kevin-meneses-gonzalez/) · [kevinmenesesgonzalez@gmail.com](mailto:kevinmenesesgonzalez@gmail.com)*", "url": "https://wpnews.pro/news/web-scraping-with-claude-code-a-working-workflow-with-gaffa", "canonical_source": "https://dev.to/kevin_menesesgonzlez/web-scraping-with-claude-code-a-working-workflow-with-gaffa-11po", "published_at": "2026-11-06 16:13:12+00:00", "updated_at": "2026-11-06 16:24:30.625261+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Claude Code", "Gaffa", "Anthropic", "books.toscrape.com", "gaffa.dev"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/web-scraping-with-claude-code-a-working-workflow-with-gaffa", "markdown": "https://wpnews.pro/news/web-scraping-with-claude-code-a-working-workflow-with-gaffa.md", "text": "https://wpnews.pro/news/web-scraping-with-claude-code-a-working-workflow-with-gaffa.txt", "jsonld": "https://wpnews.pro/news/web-scraping-with-claude-code-a-working-workflow-with-gaffa.jsonld"}}