Web Scraping with Claude Code: A Working Workflow with Gaffa

Scraping scripts rarely fail on day one.

They fail quietly on day ten.

The script runs. No exception. A CSV appears. And somewhere in it, 24 rows are missing. Nobody notices until the data is already in a dashboard.

If you're:

This matters.

I gave Claude Code a plain-language scraping goal and Gaffa's AI skills, then let it go from zero to a working web scraping workflow: discovery, a paginated scraper, CSV export and verification. Then I tried to break it.

Every number, error and output below comes from real runs.

Writing a scraper is the easy part. Claude can do it in one shot.

The hard part is everything around it:

None of these throw an error. The script finishes, the CSV looks fine, and the data is wrong.

You don't find these bugs by reading code. You find them by running the scraper against the real site and checking the output against something you trust.

The fix isn't a smarter prompt.

It's a workflow where the agent can discover the site, build against the real pages, and debug failures from real request data. With checks that tell you when the output is wrong.

Gaffa's AI skills cover exactly those three steps:

/gaffa-find`` gaffa-authoring``/gaffa-debug`` brq_...) and it diagnoses what went wrong Under the hood, Gaffa is a browser automation REST API: you send a URL and a list of actions (click, wait, capture_element, loop...) and it runs them in a real browser.

👉 You can create an account at gaffa.dev and follow along.

If you read my previous article on connecting Claude Code to Gaffa, you already have this. If not, it's one command in your project folder:

npx @gaffa-dev/cli install --tools=claude-code --scope=project -y
Installed the gaffa skills (version 0.9.0) into 1 place:
  ./.claude\skills  (project)  gaffa-authoring, gaffa-debug, gaffa-find, gaffa-support

Registered the gaffa-docs MCP server:
  Claude Code  ./.mcp.json  added

Then set GAFFA_API_KEY as an environment variable. The skills never read the key from anywhere else.

The target for this test is books.toscrape.com, a sandbox catalogue built for scraping practice: 1,000 books, 50 categories, classic "next" pagination.

I didn't give Claude a URL for the category, a selector or a page count. Just the homepage and a goal:

/gaffa-find https://books.toscrape.com/ — Goal: I want to collect every book
in the 'Mystery' category. Find the category page URL, how many books it
contains, how many listing pages it spans and how pagination works, and which
fields each book card exposes (title, price, rating, availability, link).

Rendered from the real session transcript.

The skill runs a loop: recon, hypothesis, targeted extraction, validation. With hard caps: 15 requests and 50 credits per run, 5 iterations maximum.

For recon, it sent a browser request that turns the homepage into Markdown. 3 credits, 5.8 seconds, and the answer was in the sidebar:

- [Mystery](catalogue/category/books/mystery_3/index.html)

From there it captured the category page twice: once as Markdown (to read "32 results" and "Page 1 of 2") and once as raw HTML (to see the markup the Markdown strips out, like the star rating). Then it checked page 2 to confirm where pagination ends.

It also wrote a reasoning log to disk, .gaffa-find-<timestamp>.log, one entry per iteration: hypothesis, request id, credits, decision. You can audit exactly how it reached its answer.

The final result:

Terminal state: success. 32 books, 2 pages, pagination through a relative li.next a link that disappears on the last page, and a selector for each field. One detail I wouldn't have caught: the visible title text is truncated with "…", so the full title has to come from the link's title attribute.

Cost of discovery: 9 credits.

Same Claude session, so it kept everything it had learned. This time I asked for a real workflow:

Rewrite it as books_to_csv.py: one single Gaffa browser request that
paginates with the loop action. Price as a number, rating as an integer
1-5, absolute product URL. Write a CSV. Add --category-url. Run it on
Mystery, verify 32 rows with no duplicates, then run it on a category
with 3+ pages and verify the count matches the 'N results' text.

Claude pulled the loop reference through the Gaffa docs MCP server (approved this time), then built the request.

This is the heart of the workflow. Instead of one request per page, a single request captures the results banner once and then loops: capture the product grid, click "next", repeat until there is no "next" link.

"actions": [
    {"type": "capture_element", "selector": "form.form-horizontal", "custom_id": "results-banner"},
    {
        "type": "loop",
        "custom_id": "pagination-loop",
        "max_iterations": max_pages,
        "timeout": loop_timeout,
        "stop_on_fail": True,
        "continue_on_fail": True,  # the last page has no "next": that's how the loop ends
        "actions": [
            {"type": "wait", "time": 800, "custom_id": "settle"},
            {"type": "capture_element", "selector": "ol.row", "custom_id": "page-cards"},
            {"type": "click", "selector": "li.next a", "custom_id": "next-page", "timeout": 15000},
        ],
    },
],

Parsing happens in plain Python with regular expressions over the captured HTML. No LLM extraction. The skill steers Claude away from Gaffa's parse_json action for structured pages like this one, because a deterministic parser is cheaper and returns the same output every run.

The most valuable line in the script is not the scraper. It's this:

status = "OK" if reported_total == len(books) else "MISMATCH"
print(f"page says {reported_total} results; scraped {len(books)} -> {status}")
if status == "MISMATCH":
    sys.exit(1)

The workflow compares its own output with the number the website reports. If they differ, it fails loudly.

Here is how phase 2 went:

Attempt 1: Mystery. 32 of 32 books, 32 unique URLs, banner matched. 3 credits.

Attempt 2: Nonfiction (110 books, 6 pages).

request: brq_VzZhqRu7EsTn47GhMyLqkSN76iJDA6
pages captured: 6
books written: 86 -> nonfiction.csv
page says 110 results; scraped 86 -> MISMATCH

All 6 pages were captured, but 24 books were missing. Claude broke it down page by page:

page 1: articles=20
page 2: articles=20
page 3: articles=20
page 4: articles=12
page 5: articles=14
page 6: articles=2

Pages 4 to 6 were captured while the browser was still them, right after the click. Claude added a short wait at the start of each iteration and re-ran: 110 of 110. Then re-ran Mystery to make sure nothing regressed.

Without that count check, the script would have exited 0 and handed me a CSV missing 22% of the data.

Cost of phase 2: 6 billed requests, 18 credits.

Claude said it was done. I didn't take its word for it. I ran the script myself on categories it had never seen.

Travel (11 books) and Fiction (65 books, 4 pages) both matched their counts. Then I ran Nonfiction again:

pages captured: 2
books written: 40 -> nonfiction.csv
page says 110 results; scraped 40 -> MISMATCH

The loop stopped after page 2. The count check caught it again.

Failed and partial requests are part of scraping any live website. Pages get slow, layouts change, content loads late. What matters is how quickly you find out why. That's the job of the third skill.

I passed the request id to Claude:

/gaffa-debug brq_VzZj6m1Mqsb31bZEevh2ZHdUNVFxyh — it only got 40 of 110
books (MISMATCH). Diagnose why the loop stopped early, apply a minimal fix
that does not rely on luck, then verify on Nonfiction twice and Mystery once.

The diagnosis was precise:

action_timed_out. page-2.html directly (free, no Gaffa call). It has a "next" link to page 3. The patch was one value: raise the click's timeout from 5,000 to 15,000 ms, the same value Gaffa's own pagination examples use. A real last page still ends the loop, just 15 seconds later.

Verification: Nonfiction 110 of 110 twice in a row with identical rows, Mystery 32 of 32. 13 credits.

I ran the final script one last time on my own:

218 books, 4 categories, 0 duplicates. Nonfiction, 6 pages in a single request, takes about 42 seconds end to end.

The CSV is clean, typed data ready for pandas or a spreadsheet:

title,price,rating,availability,url
Worlds Elsewhere: Journeys Around Shakespeare’s Globe,40.3,5,In stock,https://books.toscrape.com/catalogue/worlds-elsewhere-journeys-around-shakespeares-globe_972/index.html
The Five Love Languages: How to Express Heartfelt Commitment to Your Mate,31.05,3,In stock,https://books.toscrape.com/catalogue/the-five-love-languages-how-to-express-heartfelt-commitment-to-your-mate_969/index.html
Reasons to Stay Alive,26.41,2,In stock,https://books.toscrape.com/catalogue/reasons-to-stay-alive_959/index.html

Claude Code ran on my Claude Pro subscription.

From here you can:

Three lessons from this run, all of which apply outside Gaffa:

Put the verification in the script, not in your head. The "N results" check caught two silent failures that would otherwise have shipped. If the site tells you how many items it has, use it.

Debug from request data, not from guesses. /gaffa-debug didn't guess. It read how many iterations ran, which action failed and how long the page took to load, then checked the live page before changing a single value.

Fixed waits are guesses. The 800 ms wait happened to work, and the 5 s click timeout didn't when the site was slow. When you can, wait on a selector or give timeouts real margin.

loop scraped a 6-page category in about 42 seconds, and a built-in count check caught two silent data losses. Claude Code writes scrapers fast. That was never the bottleneck.

The bottleneck was knowing whether the data is right.

Give the agent a real browser, skills that force it to look before it writes, and a workflow that checks its own output, and web scraping with Claude Code becomes something you can actually trust.

👉 Try it: create an account at gaffa.dev, run npx @gaffa-dev/cli install, and start with /gaffa-find on a site you need data from.

Looking for technical content for your company? I can help — LinkedIn · kevinmenesesgonzalez@gmail.com