Show HN: CLI for crawling documentation sites into Markdown with defuddle

Docrawl, a lightweight Node.js CLI tool for crawling documentation sites and converting them into Markdown, has been released. The tool supports static and server-rendered docs platforms like Docusaurus, VitePress, and MkDocs, and is designed to feed content into LLM contexts, RAG pipelines, and local knowledge bases without requiring a browser or JavaScript execution.

docrawl is a lightweight Node.js CLI for crawling documentation sites and converting them into Markdown with defuddle https://github.com/kepano/defuddle . It is built for static and server-rendered docs sites such as Docusaurus, VitePress, MkDocs, GitBook exports, and Obsidian Publish. It does not run a browser and does not execute page JavaScript. docrawl is useful when you want to: - turn docs sites into Markdown for LLM context - build local knowledge bases - feed content into RAG pipelines - archive clean docs content without a browser dependency - Node.js = 20 Run without installing: npx docrawl --help Install globally: npm install -g docrawl Then run: docrawl --help npm install Build: npm run build Run the CLI from the project workspace: npm run start -- --help Run tests: npm test docrawl crawl <url options Examples: Crawl a docs section into ./output docrawl crawl https://docs.example.com/guide/ Run a smaller smoke test first docrawl crawl https://docs.example.com/guide/ --max-pages 10 --depth 1 --verbose Merge everything into one file docrawl crawl https://docs.example.com/guide/ --single-file --output ./context.md Crawl the full hostname, not only the seed path subtree docrawl crawl https://docs.example.com --domain --max-pages 200 Options: -o, --output <path Output directory or file path -s, --single-file Merge all pages into one Markdown file --domain Crawl the whole hostname, not just the seed path --depth <n Maximum crawl depth --max-pages <n Maximum pages to process default: 500 --concurrency <n Concurrent requests default: 3 --delay <ms Delay between requests per worker default: 500 --lang <code Preferred language, BCP 47 --no-sitemap Disable sitemap discovery --include <glob Include URL glob pattern, repeatable --exclude <glob Exclude URL glob pattern, repeatable --verbose Verbose progress logging docrawl parse <url options Examples: Parse one page as Markdown docrawl parse https://docs.example.com/guide/intro Parse one page as JSON docrawl parse https://docs.example.com/guide/intro --json Options: -j, --json Output full JSON response --lang <code Preferred language, BCP 47 By default, docrawl crawl writes one Markdown file per successful page and a manifest.json . Example layout: output/ ├── getting-started/ │ ├── introduction.md │ └── quickstart.md └── manifest.json Each Markdown file includes frontmatter with fields such as: title sourceUrl finalUrl canonicalUrl crawledAt depth wordCount contentHash With --single-file , docrawl writes: - one merged Markdown file - one adjacent manifest file named like <name .manifest.json The merged file includes a table of contents and one section per successful page. Example: docrawl crawl https://docs.example.com --single-file --output ./context.md Produces: context.md context.manifest.json docrawl currently does not handle: - JavaScript-rendered SPAs that need browser execution - login-gated or authenticated content - asset downloading robots.txt compliance- resumable crawls - incremental recrawls - full navigation reconstruction