Free Website SEO Audit: Finding What's Actually Broken

A developer detailed a free website SEO audit process that uncovered 31 broken pages and 900 phantom URLs on an inherited site, leading to improved crawl efficiency after fixes. The audit uses free tools like Google Search Console and PageSpeed Insights, and the developer emphasized fixing underlying causes rather than symptoms. The article also advises preparing content for AI search engines by checking robots.txt for AI crawler access and using structured data like Article and FAQPage schema.

Your site can rank for nothing and you won't know why — unless you run this free audit. Here is the exact checklist I run, plus what the AI search engines want in 2026. Six months ago, I opened Search Console for a site I'd inherited from a client handoff and found something ugly: 31 pages returning 404s, most of them indexed. And that was only the visible damage. The crawl data showed roughly 900 URLs Google had discovered that I had never built — phantom links, pagination fragments, and a parameter mess that made the site look three times larger than it was. That audit cost nothing. No paid tool, no agency report. Just the free stack — Search Console, a crawl log, and page-speed data — plus the knowledge of what to look at first. The fix took a week, and the site's crawl efficiency jumped visibly within two cycles. Here is how to get found and cited in 2026: fix what is actually broken, then make your content trivially quotable for the AI crawlers. This article walks both halves — the free technical audit and the platform-by-platform checklist for AI search visibility. Run these in order. Each one needs zero dollars and catches a different class of failure. Open Google Search Console → Pages report. Filter by the "Not indexed" tab and read the reasons. The four that matter: If your money pages sit in any of these, fix the reason, not the symptom. A "crawled not indexed" page needs better content or a better internal link; a "discovered" page needs fewer junk URLs so the crawler reaches it. Grab your server access log or use Google Search Console's "Page indexing" report for the URLs Google has already flagged. A free crawler like Screaming Frog's free tier 500 URLs or a quick log analysis will surface the 404s that matter. Then hunt orphans — pages with zero internal links. I found 900 phantom URLs because of bad pagination and filters. The fix was noindex,follow on parameter pages, clean canonical tags, and an XML sitemap that lists only the 200 pages worth ranking. The discipline here is to fix the reason behind each status, not the page. A page Google "discovered but never crawled" needs fewer junk URLs competing for crawl budget, not a new meta description. A page "crawled but not indexed" needs better content and a stronger internal link. If you only chase symptoms, the same URLs come back in the next cycle and you have learned nothing. Google's thresholds are public, and they are the same thresholds the AI crawlers use as a rough quality signal: fetchpriority="high" on the hero. Use PageSpeed Insights free and run your top ten pages. For most WordPress or Next.js sites, the top three fixes are: serve next-gen images, defer non-critical JavaScript, and self-host or remove tracking scripts. The basics, checked in five minutes: one H1 per page, a unique title and meta description, a canonical tag pointing to itself, no meta robots "noindex" on pages you want ranked, and internal links from at least one relevant page. Every money page should be reachable within three clicks of the homepage. Run a site: search on your domain for a few core terms. If two URLs return for the same query, that is a duplicate-signal problem. Consolidate with 301s or canonical tags and merge the content into a single stronger page. Use Google's Rich Results Test free on your article and product pages. In 2026, the schema that earns you AI citations is: Article with datePublished , dateModified , author , FAQPage , HowTo , and Organization . Structured data is how you hand the AI crawlers a clean map of who wrote what, when, and what it answers. This step did not exist three years ago and it matters now. Several high-profile sites have spent years blocking AI crawlers, then wondered why they vanished from AI answers. Check your robots.txt for the major user agents — GPTBot , PerplexityBot , Google-Extended the Gemini crawler , ClaudeBot , CCBot Common Crawl, which trains and is read by several systems , and Bytespider TikTok . If you want AI citations, they need permission to fetch your pages: User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: ClaudeBot Allow: / You can scope it by path if some areas are off-limits — Disallow: /admin/ , for instance — but blocking an AI crawler from your public article pages is a choice with a measurable cost. It is one of the cheapest wins on this list: most sites are blocked by default because a template shipped with broad Disallow rules, not because anyone decided deliberately. Then confirm your sitemap is submitted and clean. An XML sitemap listing only canonical, indexable pages is the fastest way to tell every crawler — Google's and the AI ones — which URLs are real and worth reading. The old SEO checklist gets you indexed and ranked in classic results. The 2026 question is whether AI assistants cite you at all. They behave differently, and each platform has its own biases. AI Overviews pulls heavily from pages that rank top-three for the query, plus pages with strong structured data. The playbook: rank well traditionally first, because the Overview is generated from the same index — then reinforce with FAQPage schema and a clear, direct answer in the first paragraph. Checklist: FAQPage or Article schema validated with Rich Results Test dateModified updates when content changes, so the AI treats it as fresh ChatGPT's browse mode reads full pages and prefers content it can quote verbatim. That rewards clear, self-contained answers with named facts, dates, and numbers — things a model can lift as a citation. Perplexity cites sources inline and ranks by a mix of authority and recency. It aggressively values freshness — a slightly lower-authority page published this month beats a stale authority page. Update your money pages on a schedule and say so in the page. Gemini's grounded answers pull from a blend of web results and structured data, and it gives extra weight to sources that machine-read cleanly. Semantic HTML, logical heading order, and schema are not optional here. h1 → h2 → h3 , no skipping <article , <time , <address for authors Organization + After the fixes, watch four things in Search Console and analytics for a month: A fifth signal is free if your analytics tool splits crawler traffic: watch whether GPTBot , PerplexityBot , and ClaudeBot are actually hitting your site, and at what frequency. If a page gets crawled by Google but never by an AI crawler, check your robots.txt rules again. Visibility into crawler behavior turns the AI work from guesswork into a measurable loop. If you want the whole thing automated, I built a free audit tool at MisarSEO https://misarseo.com that runs the crawl-and-indexability pass for you in one click — it reads Search Console and surfaces the "crawled not indexed" and 404 lists without the spreadsheet work. Free tier handles a starter site, and you keep the paid-free workflow above as the manual checklist. The brutal truth of technical SEO: your site almost certainly has a handful of obvious failures — 404s, duplicate canonicals, noindex mistakes, slow images — and you cannot rank well until you clear them. The good news is the entire audit costs an afternoon and nothing else. Run the six steps, fix the reasons behind the reports, and give the AI crawlers the clean, quotable pages they are looking for. That is the whole game. One more habit worth stealing: schedule the audit, do not wait for a traffic drop. A quarterly free pass through these six steps takes two hours and turns technical SEO from a firefight into maintenance. The sites that win the AI-citation game are rarely the biggest — they are the ones that are structurally clean, freshly updated, and allowed to be read by every crawler that matters. That is a competitive advantage you can build for the price of a lunch break. Gulshan Yad