# Show HN: Rust crawler for SEO and content audits, with optional LLM review

> Source: <https://crawler.siteone.io/>
> Published: 2026-11-01 09:52:30+00:00

Desktop Application

*Netlify.com*:

[📊 Audit report](https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0oi6qxkr-ku79.html)

[🚀 Web clone](https://crawler.siteone.io/examples-exports/netlify.com/)

The audit report is limited to the first 1 000 URLs using [`--max-visited-urls=1000`](https://crawler.siteone.io/configuration/command-line-options/#--max-visited-urls).

 

Everything you need for your website

…or install the command-line version:

```
curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.deb.sh | sudo -E bash
sudo apt install siteone-crawler
siteone-crawler --url=https://your-site.com
curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh | sudo -E bash
sudo dnf install siteone-crawler
siteone-crawler --url=https://your-site.com
curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh | sudo -E bash
sudo zypper install siteone-crawler
siteone-crawler --url=https://your-site.com
curl -1sLf https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.alpine.sh | sudo -E bash
sudo apk add siteone-crawler
siteone-crawler --url=https://your-site.com
```

Prefer Homebrew? `brew install janreges/tap/siteone-crawler`

```
# Homebrew (macOS)
brew install janreges/tap/siteone-crawler
siteone-crawler --url=https://your-site.com
```

`siteone-crawler-*-win-x64.zip` from the `cmd` or PowerShell in that folder and run:

```
siteone-crawler.exe --url=https://your-site.com
```

Command-line Interface

 *Astro.build*: [📊 Audit report](https://crawler.siteone.io/html/2024-08-24/forever/hwzxj1-qrs69-1fqlxbd.html) [🚀 Web clone](https://crawler.siteone.io/examples-exports/docs.astro.build/)

 *Nextjs.org*: [📊 Audit report](https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html) [🚀 Web clone](https://crawler.siteone.io/examples-exports/nextjs.org/) 

The audit reports are limited to the first 1 000 URLs using [`--max-visited-urls=1000`](https://crawler.siteone.io/configuration/command-line-options/#--max-visited-urls).

Nextjs.org website clone was done with the [`--disable-javascript`](https://crawler.siteone.io/configuration/command-line-options/#--disable-javascript) setting to suppress unwanted features that can't work without the backend.

Indexing of example website clones by search engines is disabled using [robots.txt](https://crawler.siteone.io/robots.txt).

Audit report + quality score

Every crawl ends with an interactive [HTML audit report](https://crawler.siteone.io/features/audit-report/) and a **0–10 [quality score](https://crawler.siteone.io/quality/quality-scoring/)** across Performance, SEO, Security, Accessibility & Best Practices — plus a **“Fix first”** panel that ranks the highest-impact issues so you know exactly where to start.

💡 See a live report for [astro.build](https://crawler.siteone.io/html/2024-08-24/forever/hwzxj1-qrs69-1fqlxbd.html), [netlify.com](https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0oi6qxkr-ku79.html) or [nextjs.org](https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html).

Block bad deploys (CI/CD gate)

Wire it into your pipeline with the built-in [CI/CD quality gate](https://crawler.siteone.io/quality/ci-cd-integration/) (`--ci`): configurable thresholds, JUnit XML and GitHub annotations, with **exit code 10** on failure to fail the build automatically. Doubles as a post-deploy **cache warmer**.

Clone any website for offline use

Generate a static [offline clone (mirror)](https://crawler.siteone.io/features/offline-website-generator/) of your whole site — assets, fonts, documents and 3rd-party resources included — to browse without a server. Great for archival and **disaster-recovery** plans.

💡 Browse a real clone of [docs.astro.build](https://crawler.siteone.io/examples-exports/docs.astro.build/) or [netlify.com](https://crawler.siteone.io/examples-exports/netlify.com/).

Convert a website to Markdown for AI

[Convert an entire website to clean Markdown](https://crawler.siteone.io/features/website-to-markdown-converter/) (or one large file) — ideal for feeding **LLMs / AI models**, generating `llms.txt`, or documentation. Includes standalone HTML-to-Markdown and a built-in viewer.

Crawl JavaScript / SPA sites

Optional [`--browser` mode](https://crawler.siteone.io/browser-rendering/browser-rendering/) renders each page in a **real Chromium** (CDP), so React, Vue, Svelte, Angular & Next.js sites are crawled, exported and converted with their **post-render DOM**.

Run it **headful (`--browser-headful`) to watch the browser open and click through every page live**, **capture a screenshot of every crawled page** (viewport or full-page, as PNG/JPG/WebP) and **stitch them into a GIF or MP4 video** time-lapse, and collect console / JavaScript / network diagnostics for each URL.

Bring your own LLM (AI assistant)

Plug **any LLM into the analysis — commercial (OpenAI, Anthropic, Google Gemini) or fully local & private** (vLLM, Ollama, LM Studio, or any OpenAI-compatible endpoint), so your data never has to leave your machine.

The opt-in [AI assistant](https://crawler.siteone.io/ai-assistant/ai-assistant/) adds AI-graded **SEO**, an executive **summary across security, accessibility, performance & infrastructure**, spelling/grammar checks, `llms.txt` and your own custom prompts — with strict cost controls and a `--ai-dry-run` preview.

…and that’s not all: deep crawling, [SEO & OpenGraph](https://crawler.siteone.io/features/seo-and-opengraph-analysis/), [accessibility](https://crawler.siteone.io/features/accessibility-analysis/), [security](https://crawler.siteone.io/features/security-analysis/), [performance](https://crawler.siteone.io/features/performance-analysis/), [broken links & 404s](https://crawler.siteone.io/features/redirect-and-404-analysis/), [sitemaps](https://crawler.siteone.io/features/sitemap-generator/), [stress testing](https://crawler.siteone.io/features/stress-testing/), [email reports](https://crawler.siteone.io/features/mailer/) and more. [See all features →](https://crawler.siteone.io/introduction/key-features/#list-of-features)

Grab the desktop app or CLI from the top of this page — or explore every option: [ready-to-use packages](https://crawler.siteone.io/installation-and-requirements/ready-to-use-packages/) (Homebrew, apt, dnf, zypper, apk), [system requirements](https://crawler.siteone.io/installation-and-requirements/system-requirements/), [build from source](https://crawler.siteone.io/installation-and-requirements/manual-installation/) and the [desktop application](https://crawler.siteone.io/installation-and-requirements/desktop-application/).

Star us on GitHub ⭐

If the tool saves you time, a star helps others find it. Star the [command-line interface](https://github.com/janreges/siteone-crawler) and the [desktop app](https://github.com/janreges/siteone-crawler-gui) repos.

Missing a feature? [Tell us](https://crawler.siteone.io/introduction/contact-and-community/#contact-and-community-links) or [open a feature request](https://github.com/janreges/siteone-crawler/issues).

Share it