{"slug": "show-hn-rust-crawler-for-seo-and-content-audits-with-optional-llm-review", "title": "Show HN: Rust crawler for SEO and content audits, with optional LLM review", "summary": "Developer Jan Regeš released SiteOne Crawler, an open-source Rust-based desktop and command-line tool that audits websites for SEO, performance, security, accessibility and best practices, scoring each crawl on a 0–10 quality scale and flagging a ranked \"Fix first\" list of issues. The crawler supports a CI/CD quality gate (`--ci`) with configurable thresholds, JUnit XML and GitHub annotations that exits with code 10 to fail a build, can generate static offline site clones, and can convert an entire website to Markdown for feeding LLMs or generating `llms.txt`. Example audit reports published for astro.build, netlify.com and nextjs.org were each capped at the first 1,000 URLs via `--max-visited-urls=1000`, and the tool installs via Homebrew, apt, dnf, zypper, apk or a Windows x64 zip.", "body_md": "Desktop Application\n\n*Netlify.com*:\n\n[📊 Audit report](https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0oi6qxkr-ku79.html)\n\n[🚀 Web clone](https://crawler.siteone.io/examples-exports/netlify.com/)\n\nThe audit report is limited to the first 1 000 URLs using [`--max-visited-urls=1000`](https://crawler.siteone.io/configuration/command-line-options/#--max-visited-urls).\n\n \n\nEverything you need for your website\n\n…or install the command-line version:\n\n```\ncurl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.deb.sh | sudo -E bash\nsudo apt install siteone-crawler\nsiteone-crawler --url=https://your-site.com\ncurl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh | sudo -E bash\nsudo dnf install siteone-crawler\nsiteone-crawler --url=https://your-site.com\ncurl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh | sudo -E bash\nsudo zypper install siteone-crawler\nsiteone-crawler --url=https://your-site.com\ncurl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.alpine.sh | sudo -E bash\nsudo apk add siteone-crawler\nsiteone-crawler --url=https://your-site.com\n```\n\nPrefer Homebrew? `brew install janreges/tap/siteone-crawler`\n\n```\n# Homebrew (macOS)\nbrew install janreges/tap/siteone-crawler\nsiteone-crawler --url=https://your-site.com\n```\n\n`siteone-crawler-*-win-x64.zip` from the `cmd` or PowerShell in that folder and run:\n\n```\nsiteone-crawler.exe --url=https://your-site.com\n```\n\nCommand-line Interface\n\n *Astro.build*: [📊 Audit report](https://crawler.siteone.io/html/2024-08-24/forever/hwzxj1-qrs69-1fqlxbd.html) [🚀 Web clone](https://crawler.siteone.io/examples-exports/docs.astro.build/)\n\n *Nextjs.org*: [📊 Audit report](https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html) [🚀 Web clone](https://crawler.siteone.io/examples-exports/nextjs.org/) \n\nThe audit reports are limited to the first 1 000 URLs using [`--max-visited-urls=1000`](https://crawler.siteone.io/configuration/command-line-options/#--max-visited-urls).\n\nNextjs.org website clone was done with the [`--disable-javascript`](https://crawler.siteone.io/configuration/command-line-options/#--disable-javascript) setting to suppress unwanted features that can't work without the backend.\n\nIndexing of example website clones by search engines is disabled using [robots.txt](https://crawler.siteone.io/robots.txt).\n\nAudit report + quality score\n\nEvery crawl ends with an interactive [HTML audit report](https://crawler.siteone.io/features/audit-report/) and a **0–10 [quality score](https://crawler.siteone.io/quality/quality-scoring/)** across Performance, SEO, Security, Accessibility & Best Practices — plus a **“Fix first”** panel that ranks the highest-impact issues so you know exactly where to start.\n\n💡 See a live report for [astro.build](https://crawler.siteone.io/html/2024-08-24/forever/hwzxj1-qrs69-1fqlxbd.html), [netlify.com](https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0oi6qxkr-ku79.html) or [nextjs.org](https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html).\n\nBlock bad deploys (CI/CD gate)\n\nWire it into your pipeline with the built-in [CI/CD quality gate](https://crawler.siteone.io/quality/ci-cd-integration/) (`--ci`): configurable thresholds, JUnit XML and GitHub annotations, with **exit code 10** on failure to fail the build automatically. Doubles as a post-deploy **cache warmer**.\n\nClone any website for offline use\n\nGenerate a static [offline clone (mirror)](https://crawler.siteone.io/features/offline-website-generator/) of your whole site — assets, fonts, documents and 3rd-party resources included — to browse without a server. Great for archival and **disaster-recovery** plans.\n\n💡 Browse a real clone of [docs.astro.build](https://crawler.siteone.io/examples-exports/docs.astro.build/) or [netlify.com](https://crawler.siteone.io/examples-exports/netlify.com/).\n\nConvert a website to Markdown for AI\n\n[Convert an entire website to clean Markdown](https://crawler.siteone.io/features/website-to-markdown-converter/) (or one large file) — ideal for feeding **LLMs / AI models**, generating `llms.txt`, or documentation. Includes standalone HTML-to-Markdown and a built-in viewer.\n\nCrawl JavaScript / SPA sites\n\nOptional [`--browser` mode](https://crawler.siteone.io/browser-rendering/browser-rendering/) renders each page in a **real Chromium** (CDP), so React, Vue, Svelte, Angular & Next.js sites are crawled, exported and converted with their **post-render DOM**.\n\nRun it **headful (`--browser-headful`) to watch the browser open and click through every page live**, **capture a screenshot of every crawled page** (viewport or full-page, as PNG/JPG/WebP) and **stitch them into a GIF or MP4 video** time-lapse, and collect console / JavaScript / network diagnostics for each URL.\n\nBring your own LLM (AI assistant)\n\nPlug **any LLM into the analysis — commercial (OpenAI, Anthropic, Google Gemini) or fully local & private** (vLLM, Ollama, LM Studio, or any OpenAI-compatible endpoint), so your data never has to leave your machine.\n\nThe opt-in [AI assistant](https://crawler.siteone.io/ai-assistant/ai-assistant/) adds AI-graded **SEO**, an executive **summary across security, accessibility, performance & infrastructure**, spelling/grammar checks, `llms.txt` and your own custom prompts — with strict cost controls and a `--ai-dry-run` preview.\n\n…and that’s not all: deep crawling, [SEO & OpenGraph](https://crawler.siteone.io/features/seo-and-opengraph-analysis/), [accessibility](https://crawler.siteone.io/features/accessibility-analysis/), [security](https://crawler.siteone.io/features/security-analysis/), [performance](https://crawler.siteone.io/features/performance-analysis/), [broken links & 404s](https://crawler.siteone.io/features/redirect-and-404-analysis/), [sitemaps](https://crawler.siteone.io/features/sitemap-generator/), [stress testing](https://crawler.siteone.io/features/stress-testing/), [email reports](https://crawler.siteone.io/features/mailer/) and more. [See all features →](https://crawler.siteone.io/introduction/key-features/#list-of-features)\n\nGrab the desktop app or CLI from the top of this page — or explore every option: [ready-to-use packages](https://crawler.siteone.io/installation-and-requirements/ready-to-use-packages/) (Homebrew, apt, dnf, zypper, apk), [system requirements](https://crawler.siteone.io/installation-and-requirements/system-requirements/), [build from source](https://crawler.siteone.io/installation-and-requirements/manual-installation/) and the [desktop application](https://crawler.siteone.io/installation-and-requirements/desktop-application/).\n\nStar us on GitHub ⭐\n\nIf the tool saves you time, a star helps others find it. Star the [command-line interface](https://github.com/janreges/siteone-crawler) and the [desktop app](https://github.com/janreges/siteone-crawler-gui) repos.\n\nMissing a feature? [Tell us](https://crawler.siteone.io/introduction/contact-and-community/#contact-and-community-links) or [open a feature request](https://github.com/janreges/siteone-crawler/issues).\n\nShare it", "url": "https://wpnews.pro/news/show-hn-rust-crawler-for-seo-and-content-audits-with-optional-llm-review", "canonical_source": "https://crawler.siteone.io/", "published_at": "2026-11-01 09:52:30+00:00", "updated_at": "2026-11-01 10:23:43.049980+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "generative-engine-optimization"], "entities": ["SiteOne Crawler", "Jan Regeš", "Rust", "Netlify", "Astro", "Next.js", "GitHub", "Homebrew"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-rust-crawler-for-seo-and-content-audits-with-optional-llm-review", "markdown": "https://wpnews.pro/news/show-hn-rust-crawler-for-seo-and-content-audits-with-optional-llm-review.md", "text": "https://wpnews.pro/news/show-hn-rust-crawler-for-seo-and-content-audits-with-optional-llm-review.txt", "jsonld": "https://wpnews.pro/news/show-hn-rust-crawler-for-seo-and-content-audits-with-optional-llm-review.jsonld"}}