agents.md: Teaching AI Agents How to Scrape (The Future of Web Automation)

A developer has created a Markdown file called "agents.md" that teaches AI coding agents how to work on a specific project, solving the problem of AI-generated code not matching a developer's style or conventions. By placing this file in the project root, AI agents like Claude Code first read the developer's naming conventions, architectural decisions, and error-handling patterns before generating code, eliminating the need for manual rewrites. The approach mirrors how README.md documents projects for humans, with agents.md serving as a universal instruction set for AI agents across tools like GitHub Copilot, Cursor, and OpenAI Codex CLI.

You're building a Scrapy scraper. You ask Claude Code for help. "Add pagination to my spider." Claude reads your code. It sees scraper.py, settings.py, the folder structure. Then it makes suggestions. But the suggestions don't match your style. You use specific naming conventions. Your error handling is particular. Your Scrapy middleware is custom. Claude doesn't know any of this. So it writes code that technically works but doesn't fit. You spend 30 minutes rewriting it to match your patterns. You ask again. Same problem. Claude doesn't learn. You have to explain everything every time. This is frustrating. You want the AI to understand your project. Not just the code, but the philosophy. The patterns. The conventions. The way you actually work. Then you discover agents.md. A simple Markdown file. You drop it in your project root. It teaches AI agents how you work. Your naming conventions. Your architectural decisions. Your scraping patterns. Your error handling approach. Everything. Now when you ask Claude Code for help, it reads agents.md first. It understands your project. The code it writes matches your style immediately. No rewriting. No repeating yourself. You're not building a scraper anymore. You're building a scraper that teaches AI how to build scrapers. This is agents.md. And it's changing how developers work with AI. Let me show you. agents.md is a Markdown file that teaches AI coding agents how to work on your project. Think of it like this: README.md teaches humans about your project. What it does. How to install it. How to contribute. agents.md teaches AI about your project. The patterns you use. The decisions you made. How to write code that fits your style. Both live in your project. Both are important. They serve different audiences. Scrapy projects are complex. You have spiders, pipelines, middleware, settings. You have patterns for error handling. Conventions for naming. Architectural decisions about how data flows. When you ask AI for help, it sees all this complexity. But it doesn't understand the patterns. The philosophy. The way you think about scraping. agents.md fixes this. You document your patterns once. AI reads it. Now every suggestion, every piece of generated code, follows your patterns automatically. There are several variants of this concept: agents.md: Open standard for any AI coding agent. Works with GitHub Copilot, Cursor, OpenAI Codex CLI, and more. SKILL.md: Anthropic's format for specific reusable skills. More structured. Separate folder per skill. CLAUDE.md: Claude Code's specific variant newer, more features . agents.md is the universal format. It works everywhere. This blog focuses on agents.md. agents.md has two parts: Here's a minimal agents.md file: --- name: ecommerce-scraper description: Web scraper for e-commerce product data --- E-commerce Scraper Project Structure - spiders/ - Scrapy spiders for different sites - pipelines.py - Data cleaning and storage - settings.py - Scrapy configuration - items.py - Item definitions Naming Conventions Spiders are named: spider <site name .py Examples: spider amazon.py, spider ebay.py Error Handling All network errors are caught and logged That's it. Simple. The agent reads this. Understands your project. Follows your style. name: Your project's identifier used internally description: Short description 50-100 chars . Tells the agent when to use this. Optional fields: --- name: ecommerce-scraper description: Web scraper for e-commerce product data version: 1.0.0 author: Your Name tags: scrapy, web-scraping, ecommerce license: MIT --- These fields help the agent understand the project scope. They're optional but helpful for teams. Here's how to build an effective agents.md for a scraper. In your project root: touch agents.md --- name: product-scraper description: Scrapy spider for scraping product data version: 1.0.0 author: Your Name tags: scrapy, web-scraping, python --- Project Structure - scrapers/spiders/ - All Scrapy spiders - One spider per website - Named spider <site .py - scrapers/pipelines.py - Data processing - TextCleaningPipeline - Normalize text - DuplicateRemovalPipeline - Remove duplicates - DatabaseStoragePipeline - Save to PostgreSQL - scrapers/items.py - Data schema definitions - scrapers/settings.py - Scrapy configuration - scrapers/middlewares.py - Custom middleware Naming Conventions Spiders: spider <site name .py - Example: spider amazon.py, spider ebay.py - Class: <SiteName Spider CamelCase - Example: AmazonSpider, EbaySpider Pipelines: <FunctionName Pipeline - Example: TextCleaningPipeline, DuplicateRemovalPipeline Items: <SiteName Item - Example: ProductItem, ReviewItem Methods: snake case - parse product , extract price , clean text Include code examples showing your approach: Spider Patterns Basic Spider Structure Every spider follows this pattern: class <SiteName Spider scrapy.Spider : name = '<site-identifier ' allowed domains = '<domain.com ' start urls = '<url ' def parse self, response : Extract items for item in response.css '<selector ' : yield { 'name': item.css '.name::text' .get , 'price': item.css '.price::text' .get , } Handle pagination next page = response.css '<next-page-selector ' .get if next page: yield response.follow next page, self.parse Error Handling Always wrap network calls in try-except: def parse product self, response : try: name = response.css '.name::text' .get if not name: self.logger.warning f"No name found on {response.url}" return price = response.css '.price::text' .get process... except Exception as e: self.logger.error f"Error parsing {response.url}: {e}" return Price Extraction Prices should be extracted as floats: def clean price self, price str : import re match = re.search r'\d+\.?\d ', price str return float match.group 0 if match else None Data Validation Check for required fields before yielding: def parse self, response : for item in response.css '.product' : product = { 'name': item.css '.name::text' .get , 'price': self.clean price item.css '.price::text' .get , 'url': response.url, } Only yield if required fields present if product 'name' and product 'price' : yield product Scrapy Settings Key settings in settings.py: CONCURRENT REQUESTS = 16 CONCURRENT REQUESTS PER DOMAIN = 2 DOWNLOAD DELAY = 2 RANDOMIZE DOWNLOAD DELAY = True USER AGENT = 'Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36' Pipelines in order : 1. TextCleaningPipeline normalize text 2. DuplicateRemovalPipeline remove duplicates 3. DatabaseStoragePipeline save to database Common Tasks Adding a New Spider 1. Create scrapers/spiders/spider <sitename .py 2. Inherit from scrapy.Spider 3. Define name, allowed domains, start urls 4. Implement parse method 5. Return items with required fields name, price, url 6. Test locally: scrapy crawl <spider-name Adding a New Pipeline 1. Create new class in pipelines.py 2. Implement process item method 3. Add to ITEM PIPELINES in settings.py 4. Assign integer priority lower = earlier Debugging a Spider Debug specific spider: scrapy crawl spider name -a debug=True See rendered HTML: scrapy shell 'https://example.com' response.css '.selector' .get Check all items before pipeline: ITEM PIPELINES = {} Dependencies Core: - Scrapy 2.9+ - Python 3.8+ Data Processing: - pandas data cleaning - sqlalchemy database - psycopg2 PostgreSQL Testing: - pytest - pytest-scrapy Install all: pip install -r requirements.txt Document what your items actually look like: Item Format All items follow this structure: { 'name': str, Product name required 'price': float, Price in USD required 'original price': float, Before discount optional 'rating': float, 0-5 stars optional 'review count': int, Number of reviews optional 'url': str, Source URL required 'site': str, Which site amazon, ebay, etc 'scraped at': str, ISO datetime } Claude Code automatically reads agents.md in your project root. Just ask for help and Claude will follow your patterns. Example: "Add a spider for Target.com following the existing patterns" Claude reads agents.md. Knows your naming conventions. Your error handling. Your spider template. Generates code that fits perfectly. Create agents.md in your repo root. GitHub Copilot reads it in VS Code/GitHub. When you start typing a new spider, Copilot autocompletes following your patterns. Same as GitHub Copilot. Cursor reads agents.md automatically. You can reference agents.md when using Codex. Start small. Add more as you need it. --- name: my-scraper description: Scrapy scraper for product data --- Project Structure - spiders/ - Scrapy spiders - pipelines.py - Data processing - items.py - Data schema Naming Spiders: spider <site .py Methods: snake case Error Handling Wrap in try-except. Log errors with URL. Start with this. Get working. Then expand. Add spider template, pipeline patterns, common tasks. Add full reference, all patterns, all conventions, troubleshooting. Add complex patterns, edge cases, performance tips. You write a 5,000-word agents.md. Too much. AI can't process it all. Fix: Keep it under 2,000 words. Use progressive disclosure. Reference external files: "See docs/advanced-patterns.md for complex scraping scenarios." "Use good names." Too vague. AI doesn't know what good means. Fix: Be specific. Show examples. "Spiders: spider .py. Classes: Spider. Methods: snake case." You update your patterns but forget to update agents.md. AI follows the old patterns from the file. Fix: Update agents.md whenever you change patterns. Commit it to Git. Shows example but no context. Fix: Explain why. Show both example and reasoning. "Remember to use kwargs for flexibility." New developers don't understand this. Fix: Explain for beginners. Show what it does. When multiple developers work on the scraper, agents.md becomes crucial. All developers read the same agents.md. Everyone follows the same patterns. No more debates: "Should this be snake case or CamelCase?" It's in agents.md. New developer joins. You say: "Read agents.md. It explains how we work." They read one file. Now they can contribute following your patterns immediately. Reviewer checks if code follows agents.md patterns. Reduces comments about style. Focus on logic, not formatting. Add this to agents.md: "AI will check: When you ask Claude Code for help, it checks these automatically. Here's a complete agents.md for an e-commerce scraper: --- name: ecommerce-scraper description: Scrapy spider for scraping product data version: 1.0.0 author: Your Team tags: scrapy, web-scraping, ecommerce, python license: MIT --- E-commerce Product Scraper This Scrapy project scrapes product data from multiple e-commerce websites. Quick Facts - Framework: Scrapy 2.9+ - Python: 3.8+ - Database: PostgreSQL - Sites: Amazon, eBay, Walmart, Best Buy - Update Frequency: Daily via cron job Project Structure ecommerce-scraper/ ├── scrapers/ │ ├── spiders/ │ │ ├── spider amazon.py │ │ ├── spider ebay.py │ │ └── spider walmart.py │ ├── items.py │ ├── pipelines.py │ ├── settings.py │ └── middlewares.py ├── requirements.txt └── agents.md Data Flow Start URLs ↓ Parse Product Page ↓ Extract: name, price, rating, reviews, url ↓ Validation Pipeline ↓ Text Cleaning ↓ Deduplication ↓ Database Storage Item Structure All items follow this structure: { 'name': str, Product name required 'price': float, Price in USD required 'original price': float, Before discount optional 'rating': float, 0-5 stars optional 'review count': int, Number of reviews optional 'url': str, Source URL required 'site': str, Which site 'scraped at': str, ISO datetime } Naming Conventions Spiders: spider <site .py Classes: <SiteName Spider CamelCase Methods: snake case Examples: - Spider file: spider amazon.py - Class: AmazonSpider - Method: extract price Spider Template class <SiteName Spider scrapy.Spider : name = '<site-slug ' allowed domains = '<domain.com ' start urls = '<start-url ' def parse self, response : try: Extract products for product in response.css '<selector ' : yield self.extract product product, response Handle pagination next page = response.css '<next-selector ' .get if next page: yield response.follow next page, self.parse except Exception as e: self.logger.error f"Error parsing {response.url}: {e}" def extract product self, product, response : return { 'name': product.css '.name::text' .get '' .strip , 'price': self.extract price product , 'url': response.urljoin product.css 'a::attr href ' .get , 'site': self.name, } def extract price self, product : import re price str = product.css '.price::text' .get '' match = re.search r'\d+\.?\d ', price str return float match.group 0 if match else None Error Handling Always wrap in try-except. Log errors with URL. Scrapy Settings CONCURRENT REQUESTS = 16 CONCURRENT REQUESTS PER DOMAIN = 2 DOWNLOAD DELAY = 2 RANDOMIZE DOWNLOAD DELAY = True USER AGENT = 'Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36' ITEM PIPELINES = { 'scrapers.pipelines.TextCleaningPipeline': 100, 'scrapers.pipelines.DuplicateRemovalPipeline': 200, 'scrapers.pipelines.DatabaseStoragePipeline': 300, } Common Tasks Adding a New Spider: 1. Create scrapers/spiders/spider <sitename .py 2. Inherit from scrapy.Spider 3. Define name, allowed domains, start urls 4. Implement parse method 5. Test: scrapy crawl <spider-name Testing a Spider: scrapy shell 'https://example.com' response.css '.selector' .get Running All Spiders: scrapy crawl amazon && scrapy crawl ebay && scrapy crawl walmart Dependencies - Scrapy 2.9+ - Python 3.8+ - sqlalchemy - psycopg2-binary - pandas Install: pip install -r requirements.txt This is a complete agents.md. When Claude Code reads this, it understands everything about your project. Every suggestion follows your patterns. agents.md teaches AI agents how your project works. Markdown file in project root. YAML frontmatter + Markdown body. Describes patterns, conventions, architecture. Read automatically by Claude Code, GitHub Copilot, Cursor. AI writes code matching your style. No rewriting AI-generated code. Faster onboarding for new developers. Consistent project standards. Works with all AI coding agents. Keep it under 2,000 words. Be specific, not vague. Update when you change patterns. Use progressive disclosure. Explain the why, not just examples. One agents.md = everyone on same page. Faster code review. Easier onboarding. Enforce standards without debate. You're not just building a scraper. You're building a scraper that teaches AI how to build scrapers. The future of development is humans and AI working together. agents.md is how you teach the AI to be a good collaborator. Next Steps: The investment in documenting patterns pays dividends every time you ask for AI help.