Show HN: Crawlie – Free open-source SEO audit tool for humans and agents

Spronta released Crawlie, a free open-source SEO audit tool that crawls websites for broken links, redirects, missing metadata, and over 40 SEO and generative-engine optimization checks. The tool includes a CLI, a desktop app, and an MCP server for AI agents, providing plain-English guidance on fixes and scoring technical SEO and AI-search readiness.

The fast, free, open-source technical SEO + GEO crawler — built for humans and agents. Crawl any site for broken links, redirects, missing metadata, and 40+ SEO & Generative-Engine checks — with plain-English guidance on every fix. Runs locally, ships a CLI and an MCP server, and costs nothing. Setup setup · CLI how-to-use-cli · MCP & agents use-with-agents-mcp · Use cases use-cases · Why I built this why-i-built-this · Desktop app desktop-app · Checks what-it-checks · Compare how-it-compares · Architecture architecture by Spronta The easy way — npm installs the crawlie CLI and the crawlie-mcp server : npm i -g @spronta/crawlie The macOS app — grab the signed .dmg from Releases https://github.com/spronta/crawlie/releases . From source — needs Rust https://rustup.rs engine/CLI/MCP and, for the desktop app, pnpm https://pnpm.io + Node: git clone https://github.com/spronta/crawlie cd crawlie cargo build --release → target/release/crawlie and target/release/crawlie-mcp or install onto your PATH: cargo install --path crates/crawlie-cli installs crawlie cargo install --path crates/crawlie-mcp installs crawlie-mcp How it ships:theCLI + MCPcomeonlythrough npm — the right native binary installs automatically as a platform package nothing to download or unblock . Thedesktop appis the only direct download: a Spronta-signed, notarized .dmg on Releases . Crawl a whole site respects robots.txt, seeds from sitemap.xml crawlie crawl https://example.com --format pretty Audit a single page, or a specific set of pages crawlie audit https://example.com/pricing crawlie audit https://example.com/a https://example.com/b Save a shareable, self-contained HTML report crawlie crawl https://example.com --format html -o report.html Clean JSON on stdout perfect for piping / scripting / agents crawlie crawl https://example.com --format json -o report.json Learn why any finding matters and how to fix it crawlie explain geo-not-answerable Output formats: pretty terminal , json machine-readable, the default , csv issues , html shareable file . Common flags: | Flag | What it does | |---|---| --max-pages <n | Cap pages fetched default 500 | --max-depth <n | Max click depth from the seed | --concurrency <n | Parallel requests default 16 | --include <glob / --exclude <glob | Scope the crawl by URL pattern | --no-robots / --no-sitemap / --no-external | Turn off robots.txt, sitemap seeding, external link checks | --severity error|warning|notice | Only output findings at/above a level | --save | Save to local report history crawlie reports , crawlie report <id | --fail-on error|warning | Non-zero exit code for CI gating | Every crawl returns two scores: a Health score technical SEO and a GEO score AI-search readiness . crawlie ships a Model Context Protocol https://modelcontextprotocol.io server so an LLM agent can run a full audit and act on it — no human in the loop. This is the part most SEO tools don't have. After npm i -g @spronta/crawlie , crawlie-mcp is on your PATH . For Claude Desktop , edit claude desktop config.json : { "mcpServers": { "crawlie": { "command": "crawlie-mcp" } } } For Claude Code : claude mcp add crawlie crawlie-mcp If you built from source instead, use the absolute path to target/release/crawlie-mcp . Any MCP-compatible client works — Cursor, Cline, your own agent. It speaks JSON-RPC over stdio. | Tool | Purpose | |---|---| crawl site | Crawl + audit a whole site SEO + GEO , returns scores, issues, per-page data | audit url | Audit a single page | audit urls | Audit an explicit list of pages | explain issue | Why a rule matters + how to fix it | list rules | The full catalogue of checks | list reports / get report | Read saved crawl history | "Crawl spronta.com, then give me the top 5 fixes that would most improve my GEO score, with the exact change for each." "Audit these three landing pages and tell me which is least ready to be cited by AI search, and why." "Run a crawl with --fail-on error semantics — are there any broken links or 5xx pages blocking launch?" The agent calls crawl site , reads the structured issues, and uses explain issue to turn findings into a prioritized, actionable plan. Pre-launch QA — catch broken links, redirects, 4xx/5xx, and missing metadata before you ship. GEO optimization — make pages citable by AI search: structured data, semantic HTML, answer-ready content, authorship/E-E-A-T. Agent workflows — let a marketing/SEO agent audit a site and propose fixes autonomously via MCP. CI/CD gating — crawlie crawl … --fail-on error in a pipeline to block regressions. Client reporting — generate a polished, shareable HTML report in one command. Auditing AI-generated sites — verify that the site your agent just built is actually built for search. I'm Sean Ryan . I've spent 6+ years at Pendo.io as a Lead Marketing Engineer and lead engineer, and on the side I'm building Spronta — AI for marketers. With AI, it's faster than ever to ship a marketing site — but most of what gets generated is slop that was never built to be found. And the tools meant to catch that fall short: most SEO auditors cost money, don't play nicely with your agents, or tell you what's wrong without telling you how to actually rank for SEO and GEO Generative Engine Optimization — being cited by AI search like ChatGPT, Perplexity, and Google AI Overviews . crawlie fixes that. It's free, it's local-first, it's agent-native, and every issue it finds comes with why it matters and how to fix it . If this is useful to you, connect with me on LinkedIn → — I share what I'm learning building AI for marketers and SEO/GEO tooling, and I'd love to hear how you're using crawlie. A beautiful Tauri + React app Geist design, light/dark, seamless window chrome : cd apps/desktop pnpm install pnpm tauri dev live native crawls pnpm dev preview the UI in a browser demo data, no backend Whole-site / single-page / URL-list modes, live progress, Health & GEO score rings, issues with built-in why-it-matters guidance, a sortable pages table, a per-page drawer GEO signals, headers, schema, hreflang… , auto-saved report history, and one-click shareable HTML export. First run, generate the icon set: cd src-tauri/icons && python3 generate.py && cd .. && pnpm tauri icon icons/source.png 46 rules and counting. Technical SEO — broken links · 4xx/5xx · redirects & chains · titles & meta descriptions missing / duplicate / length · H1s · canonicals · noindex / nofollow / X-Robots-Tag · robots.txt blocking · images missing alt · thin & duplicate content · orphan & deep pages Performance & security — slow responses · large pages · missing compression · HTTPS · mixed content · HSTS Mobile, international & social — viewport · lang · hreflang · Open Graph · Twitter cards · structured data GEO — Generative Engine Optimization — structured data, semantic HTML, answer-readiness, authorship/E-E-A-T, dated content, question-style headings, and extractable blocks, rolled into a per-page GEO score . Every finding links to plain-English guidance: why it matters , how to fix it , and what happens if you ignore it . crawlie | Screaming Frog | Sitebulb | | |---|---|---|---| | Price | Free & open-source | £259/yr to unlock | from £13.50/mo | | Engine | Rust, async, tiny binary | Java JVM | .NET | | CLI with JSON output | ✅ | partial | ❌ | MCP server agent-native | ✅ | ❌ | ❌ | GEO — AI/answer-engine audit | ✅ | ❌ | ❌ | "Why it matters" built in | ✅ every issue | ❌ | partial | | Shareable HTML report | ✅ | paid | ✅ | | Source you can read & extend | ✅ | ❌ | ❌ | crates/ crawlie-core the engine — crawl, audit, score, knowledge base, reports crawlie-cli crawlie — JSON / pretty / CSV / HTML output crawlie-mcp crawlie-mcp — Model Context Protocol server stdio apps/ desktop Tauri v2 + React Geist desktop app crawlie-core has zero host dependencies — the same audited engine drops straight into a cloud worker it already targets wasm32 . One engine, every surface, identical results. - Cloud workers shared Rust core for scheduled/remote crawls - JavaScript rendering for SPA-heavy sites - Crawl-to-crawl comparison & regression alerts - Internal-link graph visualization MIT © Sean Ryan / Spronta https://spronta.com . Built by Sean Ryan — Lead Marketing Engineer at Pendo.io, building AI for marketers at Spronta on the side. Connect on LinkedIn → https://linkedin.com/in/sean-exe If crawlie saves you time, a ⭐ on the repo and a hello on LinkedIn mean a lot.