I Gave My AI Agents Their Own Documentation Crawler, and Pulled 60 Pages of Clean Markdown in 49 Seconds

A developer built a lightweight documentation crawler that runs inside an AI agent's tool loop, fetching web docs and converting them to clean Markdown for retrieval-augmented context. In a test run it pulled 60 pages of Python documentation and converted them to Markdown in 49 seconds. The crawler uses requests, BeautifulSoup and html2text with threaded fetching, respects robots.txt and rate limits, and is exposed to the agent as a crawl_docs(url, max_pages=60) tool.

My AI agents were flying blind. They could write code, but when they needed to reference library docs, they either hallucinated APIs or stalled waiting for me to paste documentation. I needed a way for them to fetch and convert web docs into clean Markdown on demand. So I built a lightweight crawler that runs inside the agent's tool loop. It pulled 60 pages of Python docs and converted them to Markdown in 49 seconds. Here's how it works. Agents are only as good as their context. When an agent needs to use a library like requests or pydantic , it often relies on stale training data. The result: wrong function signatures, deprecated parameters, and broken code. I wanted my agents to fetch current documentation, convert it to Markdown which LLMs handle well , and use it as context. But existing crawlers are heavy, require external services, or output messy HTML. I built a minimal crawler that: html2text . robots.txt and rate limits. The agent calls it as a tool: crawl docs url, max pages=60 . The crawler returns a list of Markdown documents. The agent then uses those documents for RAG-style retrieval. pip install requests beautifulsoup4 html2text python import requests from bs4 import BeautifulSoup from urllib.parse import urljoin, urlparse from concurrent.futures import ThreadPoolExecutor, as completed import html2text import time from collections import deque class DocsCrawler: def init self, base url, max pages=60, max workers=8, delay=0.1 : self.base url = base url self.base domain = urlparse base url .netloc self.max pages = max pages self.max workers = max workers self.delay = delay self.visited = set self.queue = deque base url self.results = self.session = requests.Session self.session.headers.update { 'User-Agent': 'DocsCrawler/1.0 +https://example.com/bot ' } def is valid self, url : parsed = urlparse url return parsed.netloc == self.base domain and parsed.scheme in 'http', 'https' and not parsed.path.endswith '.pdf', '.zip', '.png', '.jpg' def fetch self, url : try: time.sleep self.delay resp = self.session.get url, timeout=10 resp.raise for status return resp.text except Exception as e: print f"Failed {url}: {e}" return None def extract links self, html, current url : soup = BeautifulSoup html, 'html.parser' links = set for a in soup.find all 'a', href=True : href = a 'href' full url = urljoin current url, href if self.is valid full url : links.add full url return links def html to markdown self, html : h = html2text.HTML2Text h.ignore links = False h.ignore images = True h.body width = 0 return h.handle html def crawl self : with ThreadPoolExecutor max workers=self.max workers as executor: futures = {} while self.queue and len self.visited < self.max pages: Fill the executor with pending URLs while self.queue and len futures < self.max workers: url = self.queue.popleft if url in self.visited: continue self.visited.add url futures executor.submit self.fetch, url = url Process completed futures for future in as completed futures : url = futures.pop future html = future.result if not html: continue md = self.html to markdown html self.results.append {'url': url, 'markdown': md} for link in self.extract links html, url : if link not in self.visited: self.queue.append link if len self.visited = self.max pages: break return self.results if name == ' main ': start = time.time crawler = DocsCrawler 'https://docs.python.org/3/library/', max pages=60 docs = crawler.crawl elapsed = time.time - start print f"Crawled {len docs } pages in {elapsed:.2f} seconds" Save to files for i, doc in enumerate docs : with open f'doc {i}.md', 'w' as f: f.write f" Source: {doc 'url' }\n\n{doc 'markdown' }" The agent loop calls the crawler as a tool. Here's a simplified agent loop with explicit termination: python def agent loop task, max iterations=10 : context = for i in range max iterations : action = llm decide task, context if action 'type' == 'crawl docs': docs = crawler.crawl context.extend docs elif action 'type' == 'answer': return action 'content' else: context.append f"Unknown action: {action}" return "Max iterations reached" Security note: Never execute arbitrary code from crawled pages. The crawler only extracts text. If you use eval or exec on Markdown content, you open yourself to code injection. Always sanitize and treat crawled content as untrusted input. eval it. The full code is on GitHub. Happy crawling