Show HN: Doc-scraper, offline searchable docs corpora for coding agents (Go)

Doc-scraper, a configurable, concurrent, and resumable web crawler written in Go, has been released as an open-source tool on GitHub to scrape technical documentation websites and convert them into clean Markdown for ingestion by large language models (LLMs). The tool, created by Sriram-PR, supports features such as content extraction via CSS selectors, HTML-to-Markdown conversion, resumable crawls, rate limiting, and optional JSONL output for RAG systems, with a stated goal of automating the collection and cleaning of web-based documentation for LLM training and retrieval-augmented generation.

A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models LLMs , and save the results locally. This project provides a powerful command-line tool to crawl documentation sites based on settings defined in a config.yaml file. It navigates the site structure, extracts content from specified HTML sections using CSS selectors, and converts it into clean Markdown files. - Built for LLM Training & RAG Systems - Creates clean, consistent Markdown optimized for ingestion - Preserves Documentation Structure - Maintains the original site hierarchy for context preservation - Production-Ready Features - Offers resumable crawls, rate limiting, and graceful error handling - High Performance - Uses Go's concurrency model for efficient parallel processing The main objective of this tool is to automate the often tedious process of gathering and cleaning web-based documentation for use with Large Language Models. By converting structured web content into clean Markdown, it aims to provide a dataset that is: - Text-Focused: Prioritizes the textual content extracted via CSS selectors - Structured: Maintains the directory hierarchy of the original documentation site, preserving context - Cleaned: Converts HTML to Markdown, removing web-specific markup and clutter - Locally Accessible: Provides the content as local files for easier processing and pipeline integration | Feature | Description | |---|---| | Configurable Crawling | Uses YAML for global and site-specific settings | | Scope Control | Limits crawling by domain, path prefix, and disallowed path patterns regex | | Content Extraction | Extracts main content using CSS selectors | | HTML-to-Markdown | Converts extracted HTML to clean GitHub-Flavored Markdown tables, task lists, strikethrough | | Image Handling | Opt-in downloading and local rewriting of image links with domain and size filtering disabled by default; doc-scraper is text-first | | Link Rewriting | Rewrites internal links to relative paths for local structure | | JSONL Output | Optional one-record-per-page JSONL with a trailing crawl-summary record, for RAG ingestion | | Concurrency | Configurable worker pools and semaphore-based request limits global and per-host | | Rate Limiting | Configurable per-host delays with jitter | | Robots.txt & Sitemaps | Respects robots.txt and processes discovered sitemaps | | State Persistence | Uses BadgerDB for state; supports resuming crawls via crawl --resume | | Graceful Shutdown | Handles SIGINT / SIGTERM with proper cleanup | | HTTP Retries | Exponential backoff with jitter for transient errors | | Observability | Structured logging log/slog ; optional pprof endpoint build with -tags pprof | | Modular Code | Organized into packages for clarity and maintainability | | CLI Utilities | Built-in config validate and config list commands for configuration management | | MCP Server Mode | Expose as Model Context Protocol server for Claude Code/Cursor integration | | Full-Text Search | Offline BM25 search over crawled docs SQLite FTS5 via the search docs MCP tool | | Auto Content Detection | Automatic framework detection Docusaurus, MkDocs, Sphinx, GitBook, ReadTheDocs with readability fallback | | Parallel Site Crawling | Crawl multiple sites concurrently with shared resource management | | Watch Mode | Scheduled periodic re-crawling with state persistence | - Go: Version 1.26 or later - Git: For cloning the repository - Disk Space: Sufficient for storing crawled content and state database Option 1: Direct Installation Recommended Install the latest version directly from GitHub: go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest This installs the doc-scraper binary to your GOPATH/bin directory usually ~/go/bin or %USERPROFILE%\go\bin . Make sure this directory is in your PATH . Option 2: Clone and Build 1. Clone the repository: git clone https://github.com/Sriram-PR/doc-scraper.git cd doc-scraper 2. Install Dependencies: go mod tidy 3. Build the Binary: make build or: go build -o doc-scraper ./cmd/doc-scraper This creates an executable named doc-scraper in the project root. Create a minimal config.yaml in the project root: output base dir: "./crawled docs" state dir: "./crawler state" enable jsonl output: true sites: rust cli book: start urls: - "https://rust-cli.github.io/book/index.html" allowed domain: "rust-cli.github.io" allowed path prefix: "/book/" content selector: " content, main" max depth: 2 seed plus one level; set 0 for the whole book Run the crawl: ./doc-scraper crawl -site rust cli book -loglevel info The Markdown, plus pages.jsonl , llms.txt , and llms-full.txt , lands under ./crawled docs/rust cli book/ output is organized by site key . A small book like this finishes in a few seconds; large sites can take minutes, so start with a low max depth to gauge size before removing the bound. A config.yaml file is required to run the crawler. Create this file in the project root or specify its path using the -config flag. When configuring for LLM documentation processing, pay special attention to these settings: - sites.<your site key .content selector : Define precisely to capture only relevant text - sites.<your site key .allowed domain / allowed path prefix : Define scope accurately - skip images : Images are not downloaded by default text-first . Set to false globally or per-site to download and localize images for offline consumption - Adjust concurrency/delay settings based on the target site and your resources Global settings applied if not overridden by site default delay per host: 500ms num workers: 8 num image workers: 8 max requests: 48 max requests per host: 4 output base dir: "./crawled docs" state dir: "./crawler state" max retries: 4 initial retry delay: 1s max retry delay: 30s global crawl timeout: 0s skip images: true Default. Set to false to download and localize images max image size bytes: 10485760 10 MiB applies only when images are downloaded enable jsonl output: true jsonl output filename: "pages.jsonl" HTTP Client Settings http client settings: timeout: 45s max idle conns per host: 6 Site-specific configurations sites: Key used with -site flag pytorch docs: start urls: - "https://pytorch.org/docs/stable/" allowed domain: "pytorch.org" allowed path prefix: "/docs/stable/" content selector: "article.pytorch-article .body" max depth: 0 0 for unlimited depth skip images: false Opt in to downloading images for this site disallowed path patterns: - "/docs/stable/. / modules/. " - "/docs/stable/. \.html . " tensorflow docs: start urls: - "https://www.tensorflow.org/guide" - "https://www.tensorflow.org/tutorials" allowed domain: "www.tensorflow.org" allowed path prefix: "/" content selector: ".devsite-article-body" max depth: 0 delay per host: 1s Site-specific override Disable JSONL output for this site, overriding global enable jsonl output: false disallowed path patterns: - "/install/. " - "/js/. " | Option | Type | Description | Default | |---|---|---|---| | default user agent | String | Default User-Agent header for requests | "" Go default | | default delay per host | Duration | Time to wait between requests to the same host | 0s no delay | | num workers | Integer | Number of concurrent crawl workers | 4 | | num image workers | Integer | Number of concurrent image download workers | same as num workers | | max requests | Integer | Maximum concurrent requests global | 10 | | max requests per host | Integer | Maximum concurrent requests per host | 2 | | output base dir | String | Base directory for crawled content | "./crawled docs" | | state dir | String | Directory for BadgerDB state data | "./crawler state" | | max retries | Integer | Maximum retry attempts for HTTP requests. To disable retries, set this to 0 together with a non-zero initial retry delay ; max retries: 0 on its own is treated as unset and falls back to the default | 3 | | initial retry delay | Duration | Initial delay for retry backoff | 1s | | max retry delay | Duration | Maximum delay for retry backoff | 30s | | global crawl timeout | Duration | Overall timeout for the entire crawl | 0s no timeout | | per page timeout | Duration | Timeout for processing a single page | 0s no timeout | | skip images | Boolean | Whether to skip downloading images. Image downloading is opt-in | true skip | | max image size bytes | Integer | Maximum allowed image size applies only when images are downloaded | 0 unlimited | | max page size bytes | Integer | Maximum HTML page body size | 52428800 50 MiB | | enable jsonl output | Boolean | Enable JSONL page output one record per page plus a trailing crawl meta record for RAG pipelines | false | | jsonl output filename | String | Filename for JSONL output | "pages.jsonl" | | enable incremental | Boolean | Enable incremental crawling globally | false | | crawl history retention | Integer | Number of past crawls per site kept in the SQLite history index powers get freshness / diff crawl | 10 | | http client settings | Object | HTTP client configuration | see below | | sites | Map | Site-specific configurations | required | HTTP Client Settings: Global; cannot be overridden per site. Pool, dialer, and TLS timings are baked into pkg/fetch with sane defaults and are not exposed as config knobs. - timeout : Overall request timeout default 45s - max idle conns per host : Idle connections per host default 2 - allow private networks : Disables the SSRF guard that blocks dials to loopback / private / link-local / CGNAT / multicast addresses. Default false . Set to true only if you intentionally crawl internal documentation servers reachable via private IPs. Site-Specific Configuration Options: - start urls : Array of starting URLs for crawling Required - allowed domain : Restrict crawling to this domain Required - allowed path prefix : Restrict crawling to URLs under this path prefix Optional; defaults to / , the whole domain . Setting it is strongly recommended to bound scope - content selector : CSS selector for main content extraction, or "auto" for automatic detection Required - max depth : Exclusive upper bound on crawl depth from start URLs. Start pages are depth 0, so 1 crawls only the start pages, 2 adds their directly-linked pages, and so on. 0 = unlimited. URLs discovered from a sitemap.xml are seeded at depth 1 one hop from the site root , so they are still bounded by max depth : max depth: 1 stays start-only and skips sitemap expansion - delay per host : Override global delay setting for this site - disallowed path patterns : Array of regex patterns for URLs to skip - link extraction selectors : Array of CSS selectors for additional link extraction areas - respect nofollow : Boolean. Whether to respect rel="nofollow" links - user agent : String. Override global user agent for this site - skip images : Override the global image setting for this site. Images are skipped unless this or the global skip images is set to false - max image size bytes : Integer. Override global max image size for this site - allowed image domains : Array of domains from which to download images - disallowed image domains : Array of domains to block image downloads from - enable jsonl output : true or false . Override global JSONL output enablement for this site - jsonl output filename : String. Override global JSONL output filename for this site Execute the compiled binary from the project root directory: ./doc-scraper <command options | Command | Description | |---|---| | crawl | Start a crawl add --resume to continue an interrupted one | | add | Probe a docs site and draft a config entry for it: detects the framework, proposes crawl scope from the sitemap, previews one extracted page, and writes only after confirmation | | config validate | Validate configuration file without crawling | | config list | List available site keys from config | | mcp-server | Start MCP server for AI tool integration | | search | Ranked full-text search over the crawled corpus BM25, stemming, section anchors | | watch | Watch sites and re-crawl on schedule | | version | Show version information | | run | Read a JSON task spec from stdin and dispatch a crawl or watch for orchestration/automation | crawl: | Flag | Description | Default | |---|---|---| | -config <path | Path to config file | config.yaml | | -site <key | Site key from config single site | - | | -sites <keys | Comma-separated site keys for parallel crawling | - | | --all-sites | Crawl all configured sites in parallel | false | | --resume | Resume an interrupted crawl from existing state | false | | -loglevel <level | Log level debug , info , warn , error | info | | -json | Emit logs as JSON one record per line instead of text | false | | -pprof <addr | pprof server address. Only effective in builds with -tags pprof ; default builds log a warning and ignore the flag | "" disabled | | -incremental | Enable incremental crawling skip unchanged pages | false | | -full | Force full crawl ignore incremental settings | false | Note: One of -site , -sites , or --all-sites is required. add: doc-scraper add https://vitepress.dev/guide/what-is-vitepress Probes the site with a handful of polite requests the page, robots.txt, llms.txt, the sitemap , then shows what it found before anything is written: the detected framework and content selector validated against the fetched page , a crawl scope clustered from the sitemap with page counts as evidence, sibling version/locale trees proposed as exclusions, and a markdown preview of the extracted page with code-block fidelity numbers. The entry is appended to your config only after you confirm; the rest of the file is preserved byte-for-byte, comments included. | Flag | Description | Default | |---|---|---| | -config <path | Path to config file created if missing | config.yaml | | -site <key | Site key to use instead of the derived one | - | | -selector <css | Content CSS selector, skipping auto-detection | - | | -depth <n | Override the proposed max depth | - | | -yes | Write without prompting | false | | -dry-run | Draft only, never write exit code 2 | false | | -json | Emit the draft as JSON on stdout human text goes to stderr | false | Exit codes: 0 written, 1 error, 2 drafted but not written. For agents and scripts: add -dry-run -json <url inspects, then add -yes <url commits; with no terminal attached the command fails fast instead of waiting on stdin. Sites whose robots.txt disallows crawling the given path are refused, and robots rules that restrict AI crawlers are surfaced as a warning. config validate: | Flag | Description | Default | |---|---|---| | -config <path | Path to config file | config.yaml | | -site <key | Site key to validate optional, validates all if empty | - | | -json | Emit a single JSON object instead of human-readable text | false | config list: | Flag | Description | Default | |---|---|---| | -config <path | Path to config file | config.yaml | | -json | Emit a single JSON object instead of human-readable text | false | mcp-server: stdio transport only; the SSE transport was removed in v2.x | Flag | Description | Default | |---|---|---| | -config <path | Path to config file | config.yaml | | -loglevel <level | Log level debug , info , warn , error | info | watch: | Flag | Description | Default | |---|---|---| | -config <path | Path to config file | config.yaml | | -site <key | Site key to watch single site | - | | -sites <keys | Comma-separated site keys to watch | - | | --all-sites | Watch all configured sites | false | | -interval <duration | Crawl interval e.g., 1h , 24h , 7d | 24h | | -loglevel <level | Log level debug , info , warn , error | info | | -json | Emit logs as JSON one record per line instead of text | false | Note: One of -site , -sites , or --all-sites is required. Basic Crawl: ./doc-scraper crawl -site tensorflow docs -loglevel info Resume a Large Crawl: ./doc-scraper crawl -site pytorch docs --resume -loglevel info Validate Configuration: ./doc-scraper config validate -config config.yaml ./doc-scraper config validate -site pytorch docs Validate specific site List Available Sites: ./doc-scraper config list High Performance Crawl with Profiling: ./doc-scraper crawl -site small docs -loglevel warn -pprof localhost:6060 Debug Mode for Troubleshooting: ./doc-scraper crawl -site test site -loglevel debug Parallel Crawl of Multiple Sites: ./doc-scraper crawl -sites pytorch docs,tensorflow docs,langchain docs Crawl All Configured Sites: ./doc-scraper crawl --all-sites Start MCP Server for Claude Desktop: ./doc-scraper mcp-server -config config.yaml crawl -incremental which implies --resume , and is also what watch mode uses re-fetches every previously-crawled page and re-checks it for changes: - Change detection is content-scoped : it hashes the extracted content-selector region, not the raw page. Churn in the page shell navigation, analytics, build timestamps, CSRF tokens outside the content selector does not count as a change. - Pages whose content region is unchanged are skipped without re-converting, re-downloading images, or rewriting output. - Pages whose content region changed are fully reprocessed and their output is rewritten. - A page that now returns an error e.g. 404 on re-crawl leaves its previously-crawled output as-is ; nothing is pruned. Because there is no conditional-request support yet, incremental mode still performs the HTTP fetch for each known page; the savings come from skipping the downstream processing of unchanged pages. Crawled content is saved under the output base dir defined in the config, organized by site key and preserving the site structure. Keying by site key rather than domain keeps two site configs that target the same domain in separate trees: <output base dir / └── <sanitized site key / e.g., flask docs ├── images/ Always created; only populated when skip images: false │ ├── image1.png │ └── image2.jpg ├── index.md Markdown for the root path ├── <jsonl output filename If enable jsonl output: true ├── llms.txt Manifest of pages auto-generated, when JSONL is enabled ├── llms-full.txt Full content concatenated auto-generated, when JSONL is enabled ├── topic one/ │ ├── index.md │ └── subtopic a.md └── topic two.md When JSONL output is enabled, the crawler also emits llms.txt and llms-full.txt following the llmstxt.org https://llmstxt.org/ convention. llms.txt is a markdown manifest H1 + summary blockquote + Pages list of every crawled page with title and URL . llms-full.txt concatenates the full markdown content of every page, with section separators. Both files are regenerated on every crawl from the JSONL source of truth, so resumed crawls produce a complete updated manifest. Each generated Markdown file begins with a YAML frontmatter block carrying page metadata, followed by the converted content: - YAML frontmatter delimited by --- with title , url source URL , crawled at RFC3339 timestamp , content hash SHA-256 of the content, matching the JSONL record , and depth - Clean content converted from HTML to GitHub-Flavored Markdown, preserving tables - Relative links to other pages when within the allowed domain - Local image references if images are enabled Example: --- title: 'Authentication' url: https://docs.example.com/api/auth crawled at: "2026-08-09T12:00:00Z" content hash: 9f2b...c1a4 depth: 2 --- Authentication ...page content as Markdown... When enabled, the crawler writes one JSON object per line to a JSONL file. This format is designed for ingestion into RAG pipelines and downstream indexers. Enable it: enable jsonl output: true jsonl output filename: "pages.jsonl" default The file mixes two record kinds, distinguished by the record type field: - page records, one per crawled page. - A single crawl meta record as the final line, holding the crawl-level summary. Resuming rewrites the file to drop any leftover crawl meta record before appending a fresh one at close, so a closed file always contains exactly one crawl meta record. page record fields from PageJSONL : | Field | Description | |---|---| | record type | Always "page" | | url | Final absolute URL of the page | | title | Page title | | content | Full markdown content | | headings | Array of headings extracted from the page | | links | Array of links found in the content | | images | Array of image URLs found in the content | | content hash | SHA-256 hash of the content used for incremental crawling | | crawled at | Timestamp of when the page was crawled | | depth | Crawl depth from the start URL | crawl meta record fields from CrawlMetaJSONL : | Field | Description | |---|---| | record type | Always "crawl meta" | | site key | Site key from the config | | allowed domain | The crawled domain | | crawl started at | Crawl start timestamp | | crawl ended at | Crawl end timestamp | | total pages | Number of pages recorded in this crawl | The output file is written to each site's output directory. Both the enable flag and filename can be overridden per site. When you set content selector: "auto" for a site, the crawler automatically detects the documentation framework and applies the appropriate content selector. Detection recognizes 30+ documentation generators and hosted platforms, checked in three tiers of decreasing trust: the <meta name="generator" tag, structural DOM signatures attributes, ids, classes , and asset path patterns. Covered families include Docusaurus, VitePress, VuePress, Starlight/Astro, Nextra, Fumadocs, Mintlify, GitBook, MkDocs Material, ReadTheDocs theme, and plain , Sphinx furo, pydata, book, RTD, and classic themes , Antora, Docsy, hugo-book, Geekdoc, just-the-docs, mdBook, rustdoc, pkg.go.dev, Javadoc, Doxygen, TypeDoc, Writerside, ReadMe.com, Intercom, and Docus. Every detected selector is validated against the live page before it is trusted: if it matches nothing or captures too little text, the crawler falls back instead of extracting empty content. Client-rendered shells Docsify, Swagger UI, Redoc, Scalar, Document360, and generic empty-body SPAs are recognized and reported as needing JavaScript rendering rather than silently producing an empty crawl. If no known framework is detected or the detected selectors do not match the page , the crawler uses Mozilla's Readability algorithm to extract the main content. This works well on classic server-rendered docs, but can drop code blocks on some modern sites, so doc-scraper add 's preview reports code-block fidelity before you commit a config. sites: pytorch docs: start urls: - "https://pytorch.org/docs/stable/" allowed domain: "pytorch.org" allowed path prefix: "/docs/stable/" content selector: "auto" Auto-detect framework max depth: 0 Crawl multiple documentation sites concurrently with shared resource management. The orchestrator coordinates multiple crawlers while respecting global rate limits and semaphores. Crawl specific sites in parallel ./doc-scraper crawl -sites pytorch docs,tensorflow docs,langchain docs Crawl all configured sites ./doc-scraper crawl --all-sites Resume parallel crawl ./doc-scraper crawl -sites pytorch docs,tensorflow docs --resume When running parallel crawls, the following resources are shared across all site crawlers: - Global semaphore : Limits total concurrent requests across all sites - HTTP client : Shared connection pooling - Rate limiter : Respects per-host delays Each site still maintains its own: - BadgerDB store for state persistence - Output directory for crawled content - Per-host semaphores for domain-specific limiting After all sites complete, the orchestrator outputs a summary: =========================================== Parallel crawl completed in 2m30s Site Results: pytorch docs: SUCCESS - 1500 pages in 1m20s tensorflow docs: SUCCESS - 2000 pages in 2m15s langchain docs: FAILED - 0 pages in 3s Error: initial fetch failed for start URL see logs ------------------------------------------- Total: 3 sites 2 success, 1 failed , 3500 pages processed =========================================== Unknown or misspelled site keys are rejected before the crawl starts, so they never appear as a FAILED row in this summary. For example, crawl -sites pytorch docs,typo key exits immediately non-zero with: Invalid site keys: site 'typo key' not found. Available sites: pytorch docs tensorflow docs langchain docs The FAILED rows in the summary are for sites that exist in the config but errored during the crawl itself. Watch mode enables scheduled periodic re-crawling of documentation sites. The scheduler tracks the last run time for each site and automatically triggers crawls when the configured interval has elapsed. Watch a single site with 24-hour interval ./doc-scraper watch -site pytorch docs -interval 24h Watch multiple sites ./doc-scraper watch -sites pytorch docs,tensorflow docs -interval 12h Watch all configured sites weekly ./doc-scraper watch --all-sites -interval 7d The interval supports standard Go duration format plus day units: - 30m - 30 minutes - 1h - 1 hour - 24h - 24 hours - 7d - 7 days - 1d12h - 1 day and 12 hours Watch mode persists state to <state dir /watch state.json , tracking: - Last run time for each site - Success/failure status - Pages processed - Error messages if any This allows the scheduler to resume correctly after restarts, only running sites when their interval has elapsed. INFO Starting watch mode for 2 sites with interval 24h0m0s INFO Watch schedule: INFO pytorch docs: last run 2024-01-15T10:30:00Z success, 1500 pages , next run 2024-01-16T10:30:00Z INFO tensorflow docs: never run, will run immediately INFO Running crawl for 1 due sites: tensorflow docs ... INFO Next crawl: pytorch docs in 23h45m at 10:30:00 Watch mode handles SIGINT/SIGTERM gracefully: it stops the scheduler and cancels any in-progress crawl, letting the crawler flush its BadgerDB state and partial output first, so the interrupted crawl resumes cleanly on the next run. The run command reads a single JSON object from stdin and dispatches the equivalent crawl or watch . It is meant for orchestration agents that would rather build a JSON payload than assemble shell flags. Unknown fields are rejected so typos surface immediately; logs go to stderr and the exit code matches the equivalent flag-driven subcommand. { "command": "crawl" | "watch", // required "config": "config.yaml", // optional, defaults to config.yaml "site": "site key", // exactly one of site | sites | all sites "sites": "a", "b" , "all sites": true, "resume": false, // crawl only "incremental": false, // crawl only implies resume "full": false, // crawl only mutually exclusive with incremental "interval": "24h", // watch only, defaults to 24h "loglevel": "info", // defaults to info "json logs": false, // emit slog records as JSON on stderr "pprof": "" // crawl only, e.g. localhost:6060 } Examples: echo '{"command":"crawl","site":"pytorch docs"}' | doc-scraper run echo '{"command":"crawl","all sites":true,"incremental":true,"json logs":true}' | doc-scraper run echo '{"command":"watch","sites": "pytorch docs","tensorflow docs" ,"interval":"6h"}' | doc-scraper run The crawler can run as a Model Context Protocol MCP https://modelcontextprotocol.io/ server, enabling integration with AI assistants like Claude Code and Cursor. | Tool | Description | |---|---| | describe server | Orientation manifest: server identity + sites + recent jobs in one call call this first | | list sites | List all configured sites from config file | | get page | Fetch a single URL live over the network and return content as markdown | | crawl site | Start a background crawl for a site returns job ID | | get job status | Check the status of a background crawl job | | cancel crawl | Cancel a running or pending crawl job by job ID | | list pages | Enumerate crawled pages for a site paginated, metadata only | | read page | Return a crawled page's markdown from the stored output, without network access | | search docs | Full-text search across crawled docs BM25, stemming, snippets , without network access | | get freshness | Report how stale a site's latest crawl is, from the crawl-history index | | diff crawl | Report pages added, removed, or changed since a given timestamp | The MCP server uses the stdio transport, compatible with Claude Desktop, Claude Code, and Cursor. ./doc-scraper mcp-server -config config.yaml Add to your Claude Code configuration claude code config.json : { "mcpServers": { "doc-scraper": { "command": "/path/to/doc-scraper", "args": "mcp-server", "-config", "/path/to/config.yaml" } } } List available sites: Tool: list sites Result: Returns all configured sites with their domains and crawl status Fetch a single page: Tool: get page Arguments: { "url": "https://docs.example.com/guide", "content selector": "article" } Result: Returns page content as markdown with metadata Start a background crawl: Tool: crawl site Arguments: { "site key": "pytorch docs", "incremental": true } Result: Returns job ID for tracking progress Check crawl progress: Tool: get job status Arguments: { "job id": "abc-123-def" } Result: Returns status, pages processed, and completion info Enumerate crawled pages: Tool: list pages Arguments: { "site key": "pytorch docs", "max results": 50, "offset": 0 } Result: Returns up to 50 page entries URL, title, depth, crawled at, content length , sorted by URL. Use offset for pagination. Cancel a running crawl: Tool: cancel crawl Arguments: { "job id": "abc-123-def" } Result: Returns cancelled: true/false and the job's current status. Has no effect on jobs already in a terminal state. Contributions are welcome Please feel free to open an issue to discuss bugs, suggest features, or propose changes. Pull Request Process: 1. Fork the repository 2. Create a feature branch git checkout -b feature/amazing-feature 3. Commit your changes git commit -m 'Add some amazing feature' 4. Push to the branch git push origin feature/amazing-feature 5. Open a Pull Request Please ensure code adheres to Go best practices and includes appropriate documentation. doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: PRIVACY.md https://github.com/Sriram-PR/doc-scraper/blob/main/PRIVACY.md . This project is licensed under the Apache-2.0 License https://github.com/Sriram-PR/doc-scraper/blob/main/LICENSE . - GoQuery https://github.com/PuerkitoBio/goquery for HTML parsing - html-to-markdown https://github.com/JohannesKaufmann/html-to-markdown for conversion - BadgerDB https://github.com/dgraph-io/badger for state persistence - mcp-go https://github.com/mark3labs/mcp-go for MCP server implementation - go-readability https://github.com/go-shiori/go-readability for content extraction fallback - modernc.org/sqlite https://gitlab.com/cznic/sqlite for the pure-Go crawl-history index