{"slug": "stop-reaching-for-a-headless-browser-to-scrape-documentation-sites", "title": "Stop Reaching for a Headless Browser to Scrape Documentation Sites", "summary": "A developer outlines lightweight alternatives to headless browsers for scraping documentation sites, noting that static site generators like Next.js and Docusaurus leave content in predictable places such as __NEXT_DATA__ tags or pre-rendered HTML. The post recommends checking the initial HTML response, using curl to verify content presence, and sparse-cloning markdown from public repos for cleaner input, while cautioning about license checks.", "body_md": "Every few days someone in a scraping forum asks a version of the same question: *\"I'm collecting documentation text for an AI tool, but the pages render with JavaScript. What's the lightest way to get the content?\"*\n\nThe answers are always the same — open DevTools, check the Network tab, find the XHR. That advice is correct, and for documentation sites specifically it's usually unnecessary work.\n\nDocumentation sites are not arbitrary web apps. They're overwhelmingly built by a handful of static site generators, and those generators leave the content sitting in predictable places. Three routes cover most of what you'll hit, and none of them need a browser.\n\nNext.js-based docs (which includes a large share of company developer portals) embed the full page payload in a `__NEXT_DATA__`\n\nscript tag. It's in the initial HTML response — no JavaScript execution needed.\n\n``` python\nimport json, re, httpx\n\nhtml = httpx.get(url, follow_redirects=True).text\nm = re.search(r'<script id=\"__NEXT_DATA__\" type=\"application/json\">(.*?)</script>', html, re.S)\nif m:\n    data = json.loads(m.group(1))\n    # content location varies by site; dump the tree once and look around\n    print(json.dumps(data[\"props\"][\"pageProps\"], indent=2)[:2000])\n```\n\nDocusaurus (the other big one) doesn't use `__NEXT_DATA__`\n\n, but it pre-renders the full article text into the static HTML. A plain `httpx.get`\n\nplus a `main`\n\nor `article`\n\nselector gets you everything. The \"JavaScript rendering\" you see in the browser is hydration for navigation and search — the prose was already there in the initial response.\n\n**Thirty-second check:** `curl -s <url> | grep -c \"some sentence you can see on the page\"`\n\n. If that returns 1 or more, the content is in the HTML and you're done. This one check resolves the majority of \"it's JavaScript-rendered\" cases, because people judge from DevTools' Elements panel — which shows the *hydrated* DOM, not what the server actually sent.\n\nMost open-source project documentation is markdown in the same repository as the code, usually under `docs/`\n\n. Scraping the rendered HTML means fetching pages one at a time, parsing them, and stripping navigation chrome you didn't want. Cloning gets you the clean source in one shot:\n\n```\ngit clone --depth 1 --filter=blob:none --sparse https://github.com/org/project\ncd project && git sparse-checkout set docs\n```\n\nYou get the original markdown — headings intact, code fences intact, no nav sidebars, no cookie banners, no per-page rate limiting. For a RAG pipeline this is strictly better input than parsed HTML, and it's one request instead of several hundred.\n\nWorth saying plainly: check the license before you ingest. Documentation is frequently licensed separately from the code, and \"the repo is public\" is not the same as \"you may redistribute this.\"\n\nA growing number of documentation hosts expose plain-text views:\n\n`/llms.txt`\n\n`/sitemap.xml`\n\n| Signal | Route |\n|---|---|\n`curl` output contains visible page text |\nParse the static HTML — done |\n`__NEXT_DATA__` in the HTML |\nExtract and walk the JSON |\nPublic repo with a `docs/` directory |\nSparse-clone the markdown |\n`/llms.txt` returns 200 |\nStart there |\n| ReadTheDocs / GitBook host | Look for the download build |\n| None of the above |\nNow open DevTools |\n\nSome cases are genuinely dynamic, and it's worth knowing them so you don't over-apply the above:\n\nFor a few hundred pages of prose, though, these are the exception. The default assumption should be that the text is already reachable, and the browser is the fallback.\n\nThe reason this matters isn't purity — it's that headless browsers change the shape of your project. You go from a script anyone can run to a pipeline with a browser binary, a memory ceiling, per-page startup cost, and a new class of flaky failures that only reproduce sometimes. On a few hundred documentation pages, Route 1 or Route 2 typically finishes before a browser-based run has finished launching.\n\nCheck whether the content is already sitting there in plain text. Most of the time, it is.\n\n*We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.*", "url": "https://wpnews.pro/news/stop-reaching-for-a-headless-browser-to-scrape-documentation-sites", "canonical_source": "https://dev.to/roamproxy/stop-reaching-for-a-headless-browser-to-scrape-documentation-sites-1l8n", "published_at": "2026-08-11 23:45:28+00:00", "updated_at": "2026-08-12 00:14:47.296592+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Next.js", "Docusaurus", "ReadTheDocs", "GitBook", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/stop-reaching-for-a-headless-browser-to-scrape-documentation-sites", "markdown": "https://wpnews.pro/news/stop-reaching-for-a-headless-browser-to-scrape-documentation-sites.md", "text": "https://wpnews.pro/news/stop-reaching-for-a-headless-browser-to-scrape-documentation-sites.txt", "jsonld": "https://wpnews.pro/news/stop-reaching-for-a-headless-browser-to-scrape-documentation-sites.jsonld"}}