# My Scraper Said the Site Had Changed Its Layout. The Parser Was Fine. It Never Got a Page. > Source: > Published: 2026-11-12 04:10:04+00:00 *A field note from the autonomous Claude Code agent I run every day on one Windows PC. The numbers come from its own ledgers, not from memory.* **Free to read first:** a PDF sample of the book behind this setup — its first 3 chapters: [https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample](https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample) Every morning a scheduled task collects open calls from three sites: data competitions, contests, and government programs. It writes a digest. When a source returns zero items, the digest adds a hint so I know where to look. On November 8 I read the task results instead of the digest, and saw that the morning run had ended with an error. One source had returned nothing. The digest line for it said "0 items collected, structure change suspected, check the parser." The same line was in the digests for November 3, 7 and 8. There were no runs from November 4 to 6. The run on November 2 was normal. A parser that finds nothing usually means the HTML changed. So the hint was reasonable. It was also a guess: the code had never looked at what the server actually sent. On November 8 I requested the same list page by hand and looked at the response headers before the body. The status was HTTP 202, the body was empty, and one header said `x-amzn-waf-action: challenge`. The main page and a single competition page also returned 202 with an empty body. The other two sources worked that morning. That is a bot check: the site asks the client to run a script before it serves the page. My collector doesn't run scripts. A page like this reaches the parser as an empty string, and the parser correctly finds zero items. I don't know what the server sent on November 3 and 7. I only kept the digest line from those runs, not the responses. The check I saw on November 8 is the likely cause for all three, but I didn't measure the first two. I did not try to get past the check. The collector gives up on that source for the run and tries again the next morning. Until the site changes, its listings stay missing. The fetch function now raises a separate error when it sees a 202 with an empty body, or that header. The digest line now reads "bot check page, not a parser problem, can't be collected automatically" instead of sending me to the parser. The independent review I run before calling anything done asked who else calls that fetch function. Another collector reads posts from a forum through the same function. It catches errors per post with `except (URLError, TimeoutError, OSError)`, so one bad post is skipped and the rest are kept. My new error was a plain runtime error. None of those three would have caught it. One post behind a bot check would have dropped every link the forum had given, from that run. I made the new error a subclass of `OSError`, so the existing handlers catch it, and added a test for that caller. I had searched for the old hint text. I had not searched for the callers. `except`. **Where this comes from.** Every post here comes from one setup I run daily: a CLAUDE.md, memory files the agent reads before it touches anything, and a separate auditor agent that returns PASS or FAIL. The first 3 chapters of the book that walks through it are free as a PDF: [https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample](https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample) The full edition is 11 chapters plus 4 ready-to-use templates (CLAUDE.md starter, memory files, auditor checklist, measurement guide) and a hands-on section for every chapter, $19 as a PDF: [https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code](https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code) Questions about the setup are welcome in the comments — I'll answer with what actually happened, not theory.