A few days ago my site stopped showing up. It was gone from the Google cards it used to appear in, and ChatGPT and Perplexity stopped citing it when I asked about my own work.
The code was fine. robots.txt allowed every crawler, every page had the right tags, and the structured data passed every test. The problem was the host. On and off for about a day, our server kept failing when Cloudflare connected to it, so new visitors and crawlers got error pages (Cloudflare’s 520, 522 and 525) instead of the site. People who had visited before still saw it, because the site saves a copy in their browser. Googlebot and the AI crawlers have no such copy. When they came during an outage, they got an error.
That is the part worth writing about. From the outside, everything looked right. The only way to see the problem was to ask for the page the way each crawler does and look at the answer it got. Most site owners have no easy way to do that, so I built one: the AI Visibility Checker. It is free, needs no sign-up, and gives a score out of 100 with the fixes in order.
When someone asks ChatGPT, Claude, Perplexity or Google’s AI Overviews a question, the answer is built from pages the tool can fetch and read, often at that moment. Before your page can be one of them, three things have to be true. They became the three parts of the score.
Reach carries the most weight because it is a hard gate. A page nobody can fetch scores zero in every answer, however good it is.
Most of the work was deciding what the score should mean. These are the decisions that shaped it.
Each AI company runs more than one crawler. OpenAI has GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for when a person asks ChatGPT to read a page. Anthropic and Perplexity have the same split. Google-Extended controls Gemini training and does not affect Google Search.
Blocking training crawlers is a fair business choice, and many publishers make it. It does not stop AI assistants from citing you. So the checker shows the training crawlers but gives them no points. A site that blocks GPTBot and lets OAI-SearchBot in can still score 100.
Most AI visibility checks stop at robots.txt. That would have missed my own problem, and it misses a common one: a firewall, a bot protection setting or a security plugin that turns crawlers away no matter what robots.txt says.
So the checker asks for your page 13 times: once as a browser and once as each of 12 crawlers, using each crawler’s own user agent. It reports what each one got, and names a Cloudflare challenge when it sees one. If a browser gets the page and PerplexityBot gets a 403, you know where to look.
There is one limit, and the page says so. Some firewalls check a crawler’s network address, not only its name. The checker sends each crawler’s name from Cloudflare’s network, so a firewall like that may treat the real crawler differently. Your server logs have the final word.
Google runs JavaScript before it indexes a page. Most AI crawlers do not. A site built as a JavaScript app can look perfect in a browser and send crawlers an empty <div id="root">.
The checker counts the words in the HTML as sent, before any script runs, and shows the first lines it found. Under 50 words is an error, because that is what an AI crawler sees.
The Recognized checks depend on outside services. On launch day, Common Crawl’s index kept timing out on our requests. If the checker counted that as a fail, every site’s score would have dropped for a reason that had nothing to do with the site.
So when a source does not answer, its check is left out of the total and the page says “did not answer, not scored.” A score should only move when something about your site changes.
The obvious version of this tool asks ChatGPT and Gemini what they say about your brand. That is useful, but every check would cost money, and a free tool that costs money per use either gets switched off or gets a sign-up form.
So the first version calls no AI model at all. It uses a small Cloudflare Worker on the free plan, Wikidata and Common Crawl (both free public services), keeps each result for an hour so repeat checks cost nothing, and limits checks per visitor. Asking Gemini directly is the next step, on Google’s free tier with no billing account attached, so it can never charge anything. When the daily limit runs out, that part will say “back tomorrow” and the rest will keep working.
A tool that fetches any address a visitor types can be pointed at places it should never reach, like a company’s internal network. The checker only fetches public addresses. It checks each address as written, then looks up where the name points before connecting, and refuses private and local addresses, including tricks like a public name that points back to the server itself. It follows 10 redirects at most, waits 8 seconds at most, and reads a limited amount of each page.
Its server also logs nothing. The address you check is fetched, scored and kept for an hour, and the deploy fails if logging is ever turned on for the Worker.
The site already had a Social Preview Debugger that fetches pages safely and reads their tags, robots.txt and structured data. The checker reuses that code rather than copying it, so a fix in one tool fixes both.
Each scoring rule has a test, and I broke each rule on purpose to make sure its test fails. One test passed when it should have failed: no test covered a crawler that robots.txt blocks and the server also refuses, which the score should count only once. I added that case before shipping.
It went live in two steps: first the Worker on its own, with no page pointing to it, then the page. The first Worker upload failed because the bundled file had the wrong name. The old version stayed live, nothing broke for visitors, and the fix was one line.
The live site check now runs a real check of robertoiriondo.com through the tool every time it runs, and fails if no score comes back. The page scores 99 on mobile and 100 on desktop in Lighthouse, with 100 for accessibility, best practices and SEO.
With the host back to normal, robertoiriondo.com scores 89, “Ready for AI search.” Every crawler gets in, the homepage has about 2,900 words readable without JavaScript, and the structured data names me, my company and my profiles.
The biggest gap is outside the site: no Wikidata item lists robertoiriondo.com as its official website. Google and AI models use Wikidata to tell people and companies with similar names apart. That is now on my list.
If the checker cannot reach a site, the page lists how to check the same things by hand. Why did my site disappear from ChatGPT or Google AI answers? Usually because the AI tool could not reach or read the page when it looked. The common causes are a server or firewall that turns crawlers away, robots.txt rules that block the search crawlers behind AI answers, a noindex tag, or a page that only shows its text after JavaScript runs. Outages on your host can do it too, even when your code is fine.
Does blocking GPTBot remove my site from ChatGPT answers? No. GPTBot collects pages for model training. ChatGPT search uses OAI-SearchBot, and ChatGPT reading a page for a user uses ChatGPT-User. You can block GPTBot and still be cited, as long as those two can reach the page.
Is the AI Visibility Checker free? Yes. It needs no sign-up and calls no AI model, so it costs nothing to run. It uses a small Cloudflare Worker on the free plan plus Wikidata and Common Crawl, which are free public services.
What score should my site get? 85 or more means search engines and AI assistants can reach, read and recognize the site. Most of the points are about reach, so a low score usually means a crawler is being turned away, which is also the quickest thing to fix.
Roberto Iriondo is a marketing leader for AI companies, full-time or fractional. He co-founded Towards AI, acquired in 2022, and builds AI agents and tools at Generative AI Lab, which keeps his technical skills sharp.
Let’s connect on LinkedIn or see how I work with AI teams on marketing. Originally published at https://robertoiriondo.com on November 11, 2026.
My Site Disappeared From AI Answers. So I Built a Free Checker was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.