Show HN: Rust crawler for SEO and content audits, with optional LLM review

Developer Jan Regeš released SiteOne Crawler, an open-source Rust-based desktop and command-line tool that audits websites for SEO, performance, security, accessibility and best practices, scoring each crawl on a 0–10 quality scale and flagging a ranked "Fix first" list of issues. The crawler supports a CI/CD quality gate (`--ci`) with configurable thresholds, JUnit XML and GitHub annotations that exits with code 10 to fail a build, can generate static offline site clones, and can convert an entire website to Markdown for feeding LLMs or generating `llms.txt`. Example audit reports published for astro.build, netlify.com and nextjs.org were each capped at the first 1,000 URLs via `--max-visited-urls=1000`, and the tool installs via Homebrew, apt, dnf, zypper, apk or a Windows x64 zip.

Desktop Application Netlify.com : 📊 Audit report https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0oi6qxkr-ku79.html 🚀 Web clone https://crawler.siteone.io/examples-exports/netlify.com/ The audit report is limited to the first 1 000 URLs using --max-visited-urls=1000 https://crawler.siteone.io/configuration/command-line-options/ --max-visited-urls . Everything you need for your website …or install the command-line version: curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.deb.sh | sudo -E bash sudo apt install siteone-crawler siteone-crawler --url=https://your-site.com curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh | sudo -E bash sudo dnf install siteone-crawler siteone-crawler --url=https://your-site.com curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh | sudo -E bash sudo zypper install siteone-crawler siteone-crawler --url=https://your-site.com curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.alpine.sh | sudo -E bash sudo apk add siteone-crawler siteone-crawler --url=https://your-site.com Prefer Homebrew? brew install janreges/tap/siteone-crawler Homebrew macOS brew install janreges/tap/siteone-crawler siteone-crawler --url=https://your-site.com siteone-crawler- -win-x64.zip from the cmd or PowerShell in that folder and run: siteone-crawler.exe --url=https://your-site.com Command-line Interface Astro.build : 📊 Audit report https://crawler.siteone.io/html/2024-08-24/forever/hwzxj1-qrs69-1fqlxbd.html 🚀 Web clone https://crawler.siteone.io/examples-exports/docs.astro.build/ Nextjs.org : 📊 Audit report https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html 🚀 Web clone https://crawler.siteone.io/examples-exports/nextjs.org/ The audit reports are limited to the first 1 000 URLs using --max-visited-urls=1000 https://crawler.siteone.io/configuration/command-line-options/ --max-visited-urls . Nextjs.org website clone was done with the --disable-javascript https://crawler.siteone.io/configuration/command-line-options/ --disable-javascript setting to suppress unwanted features that can't work without the backend. Indexing of example website clones by search engines is disabled using robots.txt https://crawler.siteone.io/robots.txt . Audit report + quality score Every crawl ends with an interactive HTML audit report https://crawler.siteone.io/features/audit-report/ and a 0–10 quality score https://crawler.siteone.io/quality/quality-scoring/ across Performance, SEO, Security, Accessibility & Best Practices — plus a “Fix first” panel that ranks the highest-impact issues so you know exactly where to start. 💡 See a live report for astro.build https://crawler.siteone.io/html/2024-08-24/forever/hwzxj1-qrs69-1fqlxbd.html , netlify.com https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0oi6qxkr-ku79.html or nextjs.org https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html . Block bad deploys CI/CD gate Wire it into your pipeline with the built-in CI/CD quality gate https://crawler.siteone.io/quality/ci-cd-integration/ --ci : configurable thresholds, JUnit XML and GitHub annotations, with exit code 10 on failure to fail the build automatically. Doubles as a post-deploy cache warmer . Clone any website for offline use Generate a static offline clone mirror https://crawler.siteone.io/features/offline-website-generator/ of your whole site — assets, fonts, documents and 3rd-party resources included — to browse without a server. Great for archival and disaster-recovery plans. 💡 Browse a real clone of docs.astro.build https://crawler.siteone.io/examples-exports/docs.astro.build/ or netlify.com https://crawler.siteone.io/examples-exports/netlify.com/ . Convert a website to Markdown for AI Convert an entire website to clean Markdown https://crawler.siteone.io/features/website-to-markdown-converter/ or one large file — ideal for feeding LLMs / AI models , generating llms.txt , or documentation. Includes standalone HTML-to-Markdown and a built-in viewer. Crawl JavaScript / SPA sites Optional --browser mode https://crawler.siteone.io/browser-rendering/browser-rendering/ renders each page in a real Chromium CDP , so React, Vue, Svelte, Angular & Next.js sites are crawled, exported and converted with their post-render DOM . Run it headful --browser-headful to watch the browser open and click through every page live , capture a screenshot of every crawled page viewport or full-page, as PNG/JPG/WebP and stitch them into a GIF or MP4 video time-lapse, and collect console / JavaScript / network diagnostics for each URL. Bring your own LLM AI assistant Plug any LLM into the analysis — commercial OpenAI, Anthropic, Google Gemini or fully local & private vLLM, Ollama, LM Studio, or any OpenAI-compatible endpoint , so your data never has to leave your machine. The opt-in AI assistant https://crawler.siteone.io/ai-assistant/ai-assistant/ adds AI-graded SEO , an executive summary across security, accessibility, performance & infrastructure , spelling/grammar checks, llms.txt and your own custom prompts — with strict cost controls and a --ai-dry-run preview. …and that’s not all: deep crawling, SEO & OpenGraph https://crawler.siteone.io/features/seo-and-opengraph-analysis/ , accessibility https://crawler.siteone.io/features/accessibility-analysis/ , security https://crawler.siteone.io/features/security-analysis/ , performance https://crawler.siteone.io/features/performance-analysis/ , broken links & 404s https://crawler.siteone.io/features/redirect-and-404-analysis/ , sitemaps https://crawler.siteone.io/features/sitemap-generator/ , stress testing https://crawler.siteone.io/features/stress-testing/ , email reports https://crawler.siteone.io/features/mailer/ and more. See all features → https://crawler.siteone.io/introduction/key-features/ list-of-features Grab the desktop app or CLI from the top of this page — or explore every option: ready-to-use packages https://crawler.siteone.io/installation-and-requirements/ready-to-use-packages/ Homebrew, apt, dnf, zypper, apk , system requirements https://crawler.siteone.io/installation-and-requirements/system-requirements/ , build from source https://crawler.siteone.io/installation-and-requirements/manual-installation/ and the desktop application https://crawler.siteone.io/installation-and-requirements/desktop-application/ . Star us on GitHub ⭐ If the tool saves you time, a star helps others find it. Star the command-line interface https://github.com/janreges/siteone-crawler and the desktop app https://github.com/janreges/siteone-crawler-gui repos. Missing a feature? Tell us https://crawler.siteone.io/introduction/contact-and-community/ contact-and-community-links or open a feature request https://github.com/janreges/siteone-crawler/issues . Share it