cd /news/generative-engine-optimization/how-to-get-found-by-chatgpt-perplexi… · home › topics › generative-engine-optimization › article
[ARTICLE · art-172816] src=dev.to ↗ pub= topic=generative-engine-optimization verified=true sentiment=· neutral

How to get found by ChatGPT, Perplexity and Google AI: the lipp.dev case

A developer relaunched the personal site lipp.dev as a static Astro build designed to be read and cited by AI assistants, explicitly allowing named AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in robots.txt while still blocking /api/. The work also included disabling Cloudflare email obfuscation so addresses appear in raw HTML, submitting to Bing and IndexNow, and adding structured data and consistent author identity; the developer reports no citation results yet, only that Google had indexed about 38 pages and Bing 2 the day after launch.

key takeaway

Name each AI crawler explicitly in robots.txt (GPTBot for training, OAI-SearchBot for ChatGPT search, Google-Extended for Gemini) so you can block training while staying eligible for AI search answers.

by read7 min views8 publishedNov 19, 2026

Originally published on lipp.dev.

My site has been live for one day. This is what I did so AI assistants can read it, find it and cite it: every step is in the site's code, and you can check it.

In short. AI assistants that search the web cite pages they can read, that they find in an index (Google's, Bing's or their own) and whose author they can identify. On lipp.dev I worked on those three things: AI crawlers welcomed by name and HTML that reads without JavaScript; Bing, IndexNow and the sitemap; structured data and the same identity everywhere. No tricks: technical SEO done with care.

The new lipp.dev went live on 18 November 2026. I am writing this the day after, so I have no results to show you yet: no charts, no "+300% citations". I'll tell you what I did and why, with links to the official documentation.

The starting point, measured on 19 November: Google had indexed about 38 pages of the site, Bing only 2. I'll measure the results over the coming weeks and update this guide with the real numbers, even if they're disappointing.

When an assistant searches the web to answer you, it reads pages that someone has already discovered and put into an index. There are three main routes:

OAI-SearchBot. OpenAI says sites that opt out of it are not shown in ChatGPT search answers (PerplexityBot to surface and link sites in its results, and Perplexity-User when a user asks a question ( In practice, a site with solid SEO basics already has a head start. The work "for AI" is mostly: don't block anyone, get indexed beyond Google, and be clear about who you are.

Many sites block AI bots without knowing it, through copied rules or a CDN setting. In my robots.txt I listed them one by one, so the intent is unambiguous:

User-agent: *
Allow: /
Disallow: /api/

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /api/

Sitemap: https://lipp.dev/sitemap-index.xml

A useful distinction: GPTBot is for model training, OAI-SearchBot for search. You can block the first, allow the second, and still appear in ChatGPT search. Likewise Google-Extended is about Gemini and doesn't touch Google Search: Google says it affects neither inclusion nor ranking (Google's crawlers). I let them all in: for a freelancer who wants to be known, being read pays off. It's a choice, not a requirement.

lipp.dev is built with Astro, as a static site: every page reaches the browser as complete HTML, with real text and real links (<a href>). React only wakes up in the contact form. I can't assume a crawler runs JavaScript, so everything that matters has to be in the HTML.

The site sits behind Cloudflare, which has an option that hides email addresses from spammers: it replaces the address in the HTML and rebuilds it with a script (Cloudflare docs). People see no difference, but anything reading only the HTML finds a placeholder. I turned it off on 19 November: hello@lipp.dev is now in the page source, exactly as people read it. The cost is a bit more spam.

No AI asks for a minimum speed score, but Google lists page experience among the basics that also apply to its AI features. On lipp.dev: static pages on Cloudflare Pages, a single self-hosted typeface, WebP images. In internal tests on 12 November, mobile Lighthouse scored 100 on the home, service and project pages, and on 19 November the server answered in about 0.19 seconds (TTFB).

The sitemap is generated on every build. Every page has its canonical URL and the Italian/English pair declared with hreflang, so no engine mixes up the two versions or takes them for duplicates.

IndexNow is a protocol that lets a site tell search engines a page is new or changed, instead of waiting for them to come by. Bing, Naver, Seznam, Yandex and Yep support it (indexnow.org). On lipp.dev a script runs after every production build, reads the sitemap and submits every URL:

// after the production build: every URL of the sitemap, in one call
const urlList = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
await fetch('https://api.indexnow.org/indexnow', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json; charset=utf-8' },
  body: JSON.stringify({ host: 'lipp.dev', key: KEY, keyLocation: `https://lipp.dev/${KEY}.txt`, urlList }),
});

The key is public by design: it sits in a text file on the domain and proves the site is mine. If the call fails, the deploy goes on anyway.

Most people only think of Google Search Console, which of course I set up. But Bing is the base of Copilot and Bing's AI summaries, and since February 2026 Bing Webmaster Tools has a report, still in preview, that shows how often your pages are cited in Copilot and Bing's summaries (Bing's announcement). I verified the domain with a DNS record and submitted the sitemap. For ChatGPT, Bing's weight isn't documented: OpenAI runs its own crawler and doesn't publish how much other indexes count. I cover both routes, since it costs little.

I also submitted the main pages to Brave Search.

llms.txt is a 2024 proposal by Jeremy Howard: a Markdown file at the site root that gives assistants a short, ordered map of the content (llmstxt.org). llms-full.txt isn't part of the proposal: it's a later convention, the long version with all the useful text.

On lipp.dev both are generated on every build from the same data as the pages (services, projects with their results, FAQ), so they can't say anything the site doesn't. I'm cautious about their weight: Google states plainly that its AI features need no such files, and no major engine has confirmed reading them systematically. I keep them because they cost nothing.

Every page of lipp.dev carries a JSON-LD graph with the same core nodes, joined by stable identifiers, so engines understand it is always the same business and the same person:

ProfessionalService + LocalBusiness: Lipp, with VAT number, area served (Rome, Lazio, Italy), opening hours, contacts, and in sameAs the Google Maps listing and the LinkedIn page;Person: Luca Piacentini, with the job title and the LinkedIn, GitHub and Instagram profiles;Service on service pages, FAQPage only where the questions are actually on the page, BreadcrumbList everywhere;ProfilePage on the TechArticle with me as the author.

{
  "@type": "Person",
  "@id": "https://lipp.dev/#luca",
  "name": "Luca Piacentini",
  "jobTitle": "Web designer e sviluppatore freelance",
  "worksFor": { "@id": "https://lipp.dev/#business" },
  "sameAs": ["https://www.linkedin.com/in/lucapiacentini/", "https://github.com/l-piacentini", "…"]
}

One rule I always keep: the structured data only says what the page also says.

On 19 November I asked ChatGPT "Who is Lipp?". It described me well, but as an "independent studio": the words of my old site. AI assistants repeat what they find almost word for word. That same day I aligned the title, description, structured data and llms.txt on "freelance web designer and developer".

Then the same sentence and the same details wherever I can be found: Google Business Profile, Bing Places (imported from Google), LinkedIn profile and page, GitHub, Instagram, AddLance, FreelanceDEV. "Luca Piacentini" is a name with plenty of namesakes, and "Lipp" alone is ambiguous: so name, brand and job always travel together.

AI assistants quote concrete sentences: who you are, where you work, what you do, with what results. That's why there is an About page with dates and background, an FAQ with direct answers, and project pages with measured results where there are some.

You can check almost all of it yourself in an afternoon:

yoursite.com/robots.txt: there must be no Disallow: / for User-agent: *). Organization), the business, the services, and

curl -s https://lipp.dev/ | grep -o 'hello@lipp.dev' | head -1
curl -s https://lipp.dev/robots.txt
curl -s https://lipp.dev/llms.txt | head -20

In two weeks, and again in two to three months, I'll look at: indexed pages in Google Search Console and Bing Webmaster Tools, Bing's AI citations report, and the same questions asked to ChatGPT, Perplexity and Google AI Mode, with date and answer. I'll update the guide on lipp.dev with what I find.

If you have a site, new or in need of fixing, I can do this work for you too: technical audit, robots.txt, structured data, Bing and IndexNow, llms.txt, and the same identity across your site and profiles. I'm a freelancer based in Rome working with businesses all over Italy, and you always talk to me: write to me or see the SEO, GEO & AEO service.

Written with the help of AI tools; every fact and link was checked by me.

── more in #generative-engine-optimization 4 stories · sorted by recency
── more on @lipp.dev 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-get-found-by-…] indexed:0 read:7min 2026-11-19 · —