I Gave My AI Agents Their Own Documentation Crawler, and Pulled 60 Pages of Clean Markdown in 49 Seconds

My AI agents were flying blind. They could write code, but when they needed to reference library docs, they either hallucinated APIs or stalled waiting for me to paste documentation. I needed a way for them to fetch and convert web docs into clean Markdown on demand. So I built a lightweight crawler that runs inside the agent's tool loop. It pulled 60 pages of Python docs and converted them to Markdown in 49 seconds. Here's how it works.

Agents are only as good as their context. When an agent needs to use a library like requests or pydantic, it often relies on stale training data. The result: wrong function signatures, deprecated parameters, and broken code. I wanted my agents to fetch current documentation, convert it to Markdown (which LLMs handle well), and use it as context. But existing crawlers are heavy, require external services, or output messy HTML.

I built a minimal crawler that:

html2text. robots.txt and rate limits. The agent calls it as a tool: crawl_docs(url, max_pages=60). The crawler returns a list of Markdown documents. The agent then uses those documents for RAG-style retrieval.

pip install requests beautifulsoup4 html2text
python
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from concurrent.futures import ThreadPoolExecutor, as_completed
import html2text
import time
from collections import deque

class DocsCrawler:
    def __init__(self, base_url, max_pages=60, max_workers=8, delay=0.1):
        self.base_url = base_url
        self.base_domain = urlparse(base_url).netloc
        self.max_pages = max_pages
        self.max_workers = max_workers
        self.delay = delay
        self.visited = set()
        self.queue = deque([base_url])
        self.results = []
        self.session = requests.Session()
        self.session.headers.update({
            'User-Agent': 'DocsCrawler/1.0 (+https://example.com/bot)'
        })

    def is_valid(self, url):
        parsed = urlparse(url)
        return (
            parsed.netloc == self.base_domain
            and parsed.scheme in ('http', 'https')
            and not parsed.path.endswith(('.pdf', '.zip', '.png', '.jpg'))
        )

    def fetch(self, url):
        try:
            time.sleep(self.delay)
            resp = self.session.get(url, timeout=10)
            resp.raise_for_status()
            return resp.text
        except Exception as e:
            print(f"Failed {url}: {e}")
            return None

    def extract_links(self, html, current_url):
        soup = BeautifulSoup(html, 'html.parser')
        links = set()
        for a in soup.find_all('a', href=True):
            href = a['href']
            full_url = urljoin(current_url, href)
            if self.is_valid(full_url):
                links.add(full_url)
        return links

    def html_to_markdown(self, html):
        h = html2text.HTML2Text()
        h.ignore_links = False
        h.ignore_images = True
        h.body_width = 0
        return h.handle(html)

    def crawl(self):
        with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
            futures = {}
            while self.queue and len(self.visited) < self.max_pages:
                while self.queue and len(futures) < self.max_workers:
                    url = self.queue.popleft()
                    if url in self.visited:
                        continue
                    self.visited.add(url)
                    futures[executor.submit(self.fetch, url)] = url

                for future in as_completed(futures):
                    url = futures.pop(future)
                    html = future.result()
                    if not html:
                        continue
                    md = self.html_to_markdown(html)
                    self.results.append({'url': url, 'markdown': md})
                    for link in self.extract_links(html, url):
                        if link not in self.visited:
                            self.queue.append(link)
                    if len(self.visited) >= self.max_pages:
                        break
        return self.results
if __name__ == '__main__':
    start = time.time()
    crawler = DocsCrawler('https://docs.python.org/3/library/', max_pages=60)
    docs = crawler.crawl()
    elapsed = time.time() - start
    print(f"Crawled {len(docs)} pages in {elapsed:.2f} seconds")
    for i, doc in enumerate(docs):
        with open(f'doc_{i}.md', 'w') as f:
            f.write(f"# Source: {doc['url']}\n\n{doc['markdown']}")

The agent loop calls the crawler as a tool. Here's a simplified agent loop with explicit termination:

def agent_loop(task, max_iterations=10):
    context = []
    for i in range(max_iterations):
        action = llm_decide(task, context)
        if action['type'] == 'crawl_docs':
            docs = crawler.crawl()
            context.extend(docs)
        elif action['type'] == 'answer':
            return action['content']
        else:
            context.append(f"Unknown action: {action}")
    return "Max iterations reached"

Security note: Never execute arbitrary code from crawled pages. The crawler only extracts text. If you use eval or exec on Markdown content, you open yourself to code injection. Always sanitize and treat crawled content as untrusted input.

eval it. The full code is on GitHub. Happy crawling!