{"slug": "i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean", "title": "I Gave My AI Agents Their Own Documentation Crawler, and Pulled 60 Pages of Clean Markdown in 49 Seconds", "summary": "A developer built a lightweight documentation crawler that runs inside an AI agent's tool loop, fetching web docs and converting them to clean Markdown for retrieval-augmented context. In a test run it pulled 60 pages of Python documentation and converted them to Markdown in 49 seconds. The crawler uses requests, BeautifulSoup and html2text with threaded fetching, respects robots.txt and rate limits, and is exposed to the agent as a crawl_docs(url, max_pages=60) tool.", "body_md": "My AI agents were flying blind. They could write code, but when they needed to reference library docs, they either hallucinated APIs or stalled waiting for me to paste documentation. I needed a way for them to fetch and convert web docs into clean Markdown on demand. So I built a lightweight crawler that runs inside the agent's tool loop. It pulled 60 pages of Python docs and converted them to Markdown in 49 seconds. Here's how it works.\n\nAgents are only as good as their context. When an agent needs to use a library like `requests` or `pydantic`, it often relies on stale training data. The result: wrong function signatures, deprecated parameters, and broken code. I wanted my agents to fetch current documentation, convert it to Markdown (which LLMs handle well), and use it as context. But existing crawlers are heavy, require external services, or output messy HTML.\n\nI built a minimal crawler that:\n\n`html2text`.` robots.txt` and rate limits.\nThe agent calls it as a tool: `crawl_docs(url, max_pages=60)`. The crawler returns a list of Markdown documents. The agent then uses those documents for RAG-style retrieval.\n\n```\npip install requests beautifulsoup4 html2text\npython\nimport requests\nfrom bs4 import BeautifulSoup\nfrom urllib.parse import urljoin, urlparse\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\nimport html2text\nimport time\nfrom collections import deque\n\nclass DocsCrawler:\n    def __init__(self, base_url, max_pages=60, max_workers=8, delay=0.1):\n        self.base_url = base_url\n        self.base_domain = urlparse(base_url).netloc\n        self.max_pages = max_pages\n        self.max_workers = max_workers\n        self.delay = delay\n        self.visited = set()\n        self.queue = deque([base_url])\n        self.results = []\n        self.session = requests.Session()\n        self.session.headers.update({\n            'User-Agent': 'DocsCrawler/1.0 (+https://example.com/bot)'\n        })\n\n    def is_valid(self, url):\n        parsed = urlparse(url)\n        return (\n            parsed.netloc == self.base_domain\n            and parsed.scheme in ('http', 'https')\n            and not parsed.path.endswith(('.pdf', '.zip', '.png', '.jpg'))\n        )\n\n    def fetch(self, url):\n        try:\n            time.sleep(self.delay)\n            resp = self.session.get(url, timeout=10)\n            resp.raise_for_status()\n            return resp.text\n        except Exception as e:\n            print(f\"Failed {url}: {e}\")\n            return None\n\n    def extract_links(self, html, current_url):\n        soup = BeautifulSoup(html, 'html.parser')\n        links = set()\n        for a in soup.find_all('a', href=True):\n            href = a['href']\n            full_url = urljoin(current_url, href)\n            if self.is_valid(full_url):\n                links.add(full_url)\n        return links\n\n    def html_to_markdown(self, html):\n        h = html2text.HTML2Text()\n        h.ignore_links = False\n        h.ignore_images = True\n        h.body_width = 0\n        return h.handle(html)\n\n    def crawl(self):\n        with ThreadPoolExecutor(max_workers=self.max_workers) as executor:\n            futures = {}\n            while self.queue and len(self.visited) < self.max_pages:\n                # Fill the executor with pending URLs\n                while self.queue and len(futures) < self.max_workers:\n                    url = self.queue.popleft()\n                    if url in self.visited:\n                        continue\n                    self.visited.add(url)\n                    futures[executor.submit(self.fetch, url)] = url\n\n                # Process completed futures\n                for future in as_completed(futures):\n                    url = futures.pop(future)\n                    html = future.result()\n                    if not html:\n                        continue\n                    md = self.html_to_markdown(html)\n                    self.results.append({'url': url, 'markdown': md})\n                    for link in self.extract_links(html, url):\n                        if link not in self.visited:\n                            self.queue.append(link)\n                    if len(self.visited) >= self.max_pages:\n                        break\n        return self.results\nif __name__ == '__main__':\n    start = time.time()\n    crawler = DocsCrawler('https://docs.python.org/3/library/', max_pages=60)\n    docs = crawler.crawl()\n    elapsed = time.time() - start\n    print(f\"Crawled {len(docs)} pages in {elapsed:.2f} seconds\")\n    # Save to files\n    for i, doc in enumerate(docs):\n        with open(f'doc_{i}.md', 'w') as f:\n            f.write(f\"# Source: {doc['url']}\\n\\n{doc['markdown']}\")\n```\n\nThe agent loop calls the crawler as a tool. Here's a simplified agent loop with explicit termination:\n\n``` python\ndef agent_loop(task, max_iterations=10):\n    context = []\n    for i in range(max_iterations):\n        action = llm_decide(task, context)\n        if action['type'] == 'crawl_docs':\n            docs = crawler.crawl()\n            context.extend(docs)\n        elif action['type'] == 'answer':\n            return action['content']\n        else:\n            context.append(f\"Unknown action: {action}\")\n    return \"Max iterations reached\"\n```\n\n**Security note:** Never execute arbitrary code from crawled pages. The crawler only extracts text. If you use `eval` or `exec` on Markdown content, you open yourself to code injection. Always sanitize and treat crawled content as untrusted input.\n\n`eval` it.\nThe full code is on GitHub. Happy crawling!", "url": "https://wpnews.pro/news/i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean", "canonical_source": "https://dev.to/gu_cci_f94bedb90083e6aab4/i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean-markdown-in-49-5d61", "published_at": "2026-10-20 06:12:44+00:00", "updated_at": "2026-10-20 06:17:13.860174+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-crawlers"], "entities": ["BeautifulSoup", "html2text", "Python", "requests"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean", "markdown": "https://wpnews.pro/news/i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean.md", "text": "https://wpnews.pro/news/i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean.txt", "jsonld": "https://wpnews.pro/news/i-gave-my-ai-agents-their-own-documentation-crawler-and-pulled-60-pages-of-clean.jsonld"}}