{"slug": "ai-discovery-index", "title": "AI Discovery Index", "summary": "Mojodojo.io's AI Discovery Index, tracking 73 newly published pages since 6 June 2026, found that Bingbot was first to reach 51% of pages and Anthropic's ClaudeBot first on 49%, with a median time to first visit of 14 hours for both. The site reported that 38% of new pages received a visit from an AI crawler within a day, while OpenAI's GPTBot reached 43 of 73 pages at a median of 38 hours and PerplexityBot and Meta-ExternalAgent lagged at 3.8 and 4.0 days respectively. Live user fetches were rare: ChatGPT-User reached 14 of 73 pages at a median of 19 hours, and DuckDuckBot, Claude-User and DuckAssistBot reached none of the 73 pages.", "body_md": "# We publish a page. How long until AI finds it?\n\nEvery new page on mojodojo.io starts a clock. We time the first verified visit from each search engine and AI crawler, and the first time an assistant reads the page live for a person. These are aggregates across every new page published in the period.\n\n14 blog posts, 15 city pages, 40 service pages\n\nmedian. Bingbot was first on 51% of pages\n\nmedian. ClaudeBot was first on 49%\n\nof new pages had a visit from an AI crawler within a day\n\n## Hours from publish to first visit\n\nThe dot is the median across the new pages the crawler has reached (log scale), and the bar covers the middle half of those pages. Diamonds are live fetches for a person. Hollow dots mean fewer than 3 pages reached, so treat them as anecdotes.\n\n- Search engine\n- AI crawler\n- Live user fetch\n\n| Crawler | Type | Pages reached | Still waiting | Median | Middle half | \n|---|---|---|---|---|---|\n| Bingbot | Search engine | 56 of 73 | 23% | 14h | 14h to 14h | \n| ClaudeBot | AI crawler | 55 of 73 | 25% | 14h | 13h to 14h | \n| GoogleOther | Search engine | 49 of 73 | 33% | 36h | 14h to 38h | \n| Googlebot | Search engine | 45 of 73 | 38% | 38h | 14h to 2.2d | \n| PetalBot | Search engine | 48 of 73 | 34% | 38h | 38h to 2.3d | \n| OAI-SearchBot | AI crawler | 45 of 73 | 38% | 38h | 23h to 2.1d | \n| GPTBot | AI crawler | 43 of 73 | 41% | 38h | 38h to 38h | \n| Amazonbot | AI crawler | 52 of 73 | 29% | 2.4d | 38h to 2.8d | \n| Applebot | Search engine | 50 of 73 | 32% | 3.8d | 3.2d to 4.3d | \n| PerplexityBot | AI crawler | 52 of 73 | 29% | 3.8d | 12h to 5.4d | \n| Meta-ExternalAgent | AI crawler | 57 of 73 | 22% | 4.0d | 3.9d to 4.1d | \n| ChatGPT-User | Live user fetch | 14 of 73 | 81% | 19h | 9.6h to 2.8d | \n| DuckDuckBot | Search engine | 0 of 73 | 100% | n/a | n/a | \n| Claude-User | Live user fetch | 0 of 73 | 100% | n/a | n/a | \n| DuckAssistBot | Live user fetch | 0 of 73 | 100% | n/a | n/a | \n\n## Share of new pages found, over time since publishing\n\nOf the new pages published in the period, the share each crawler had visited by that point after publishing. Pages published near the end of the period have had less time.\n\n- Bingbot\n- ClaudeBot\n- GoogleOther\n- Googlebot\n- PetalBot\n- OAI-SearchBot\n\n## Daily median time to discovery\n\nMedian time from publish for the pages published in each bucket (log scale). The number under each bucket is how many new pages it contains. Gaps are buckets with no new pages.\n\n- First crawl, any bot\n- First AI crawler\n- First live user fetch\n\n**About this data.** 330 pages were published since tracking began on 6 Jun 2026.\n\n**219 are excluded** because their URL was already live before they were published. Visits since 12 Oct 2026 carry exact timestamps (89% of the pages in this view have at least one). Before that only the day is known, so first visits are estimated at the middle of the remaining day.\n\n25 blog posts, 15 city pages, 45 service pages\n\nmedian. Bingbot was first on 49% of pages\n\nmedian. ClaudeBot was first on 40%\n\nof new pages had a visit from an AI crawler within a day\n\n## Hours from publish to first visit\n\nThe dot is the median across the new pages the crawler has reached (log scale), and the bar covers the middle half of those pages. Diamonds are live fetches for a person. Hollow dots mean fewer than 3 pages reached, so treat them as anecdotes.\n\n- Search engine\n- AI crawler\n- Live user fetch\n\n| Crawler | Type | Pages reached | Still waiting | Median | Middle half | \n|---|---|---|---|---|---|\n| Bingbot | Search engine | 67 of 89 | 25% | 14h | 12h to 14h | \n| ClaudeBot | AI crawler | 66 of 89 | 26% | 14h | 14h to 14h | \n| PerplexityBot | AI crawler | 63 of 89 | 29% | 32h | 9.6h to 5.4d | \n| Googlebot | Search engine | 55 of 89 | 38% | 38h | 14h to 3.8d | \n| GoogleOther | Search engine | 57 of 89 | 36% | 38h | 14h to 2.1d | \n| OAI-SearchBot | AI crawler | 56 of 89 | 37% | 38h | 22h to 2.1d | \n| GPTBot | AI crawler | 43 of 89 | 52% | 38h | 38h to 38h | \n| PetalBot | Search engine | 58 of 89 | 35% | 2.1d | 38h to 2.6d | \n| Amazonbot | AI crawler | 63 of 89 | 29% | 2.2d | 38h to 2.8d | \n| Applebot | Search engine | 61 of 89 | 31% | 3.8d | 3.4d to 4.3d | \n| Meta-ExternalAgent | AI crawler | 68 of 89 | 24% | 3.9d | 3.5d to 4.1d | \n| ChatGPT-User | Live user fetch | 24 of 89 | 73% | 2.4d | 12h to 3.1d | \n| Claude-User | Live user fetch | 2 of 89 | 98% | 8.4d | 6.9d to 10.0d | \n| DuckDuckBot | Search engine | 0 of 89 | 100% | n/a | n/a | \n| DuckAssistBot | Live user fetch | 0 of 89 | 100% | n/a | n/a | \n\n## Share of new pages found, over time since publishing\n\nOf the new pages published in the period, the share each crawler had visited by that point after publishing. Pages published near the end of the period have had less time.\n\n- Bingbot\n- ClaudeBot\n- PerplexityBot\n- Googlebot\n- GoogleOther\n- OAI-SearchBot\n\n## Weekly median time to discovery\n\nMedian time from publish for the pages published in each bucket (log scale). The number under each bucket is how many new pages it contains. Gaps are buckets with no new pages.\n\n- First crawl, any bot\n- First AI crawler\n- First live user fetch\n\n**About this data.** 330 pages were published since tracking began on 6 Jun 2026.\n\n**219 are excluded** because their URL was already live before they were published. Visits since 12 Oct 2026 carry exact timestamps (82% of the pages in this view have at least one). Before that only the day is known, so first visits are estimated at the middle of the remaining day.\n\n33 blog posts, 15 city pages, 50 service pages, 6 playbooks\n\nmedian. Bingbot was first on 48% of pages\n\nmedian. ClaudeBot was first on 33%\n\nof new pages had a visit from an AI crawler within a day\n\n## Hours from publish to first visit\n\nThe dot is the median across the new pages the crawler has reached (log scale), and the bar covers the middle half of those pages. Diamonds are live fetches for a person. Hollow dots mean fewer than 3 pages reached, so treat them as anecdotes.\n\n- Search engine\n- AI crawler\n- Live user fetch\n\n| Crawler | Type | Pages reached | Still waiting | Median | Middle half | \n|---|---|---|---|---|---|\n| Bingbot | Search engine | 86 of 108 | 20% | 14h | 10h to 14h | \n| ClaudeBot | AI crawler | 85 of 108 | 21% | 14h | 12h to 14h | \n| PerplexityBot | AI crawler | 82 of 108 | 24% | 32h | 11h to 5.4d | \n| GoogleOther | Search engine | 62 of 108 | 43% | 38h | 14h to 2.1d | \n| OAI-SearchBot | AI crawler | 74 of 108 | 31% | 38h | 31h to 2.9d | \n| GPTBot | AI crawler | 61 of 108 | 44% | 38h | 38h to 38h | \n| Googlebot | Search engine | 73 of 108 | 32% | 2.2d | 24h to 5.6d | \n| PetalBot | Search engine | 77 of 108 | 29% | 2.2d | 38h to 10.6d | \n| Amazonbot | AI crawler | 82 of 108 | 24% | 2.3d | 36h to 2.8d | \n| Meta-ExternalAgent | AI crawler | 86 of 108 | 20% | 4.0d | 3.9d to 4.4d | \n| Applebot | Search engine | 80 of 108 | 26% | 4.1d | 3.5d to 5.7d | \n| DuckDuckBot | Search engine | 1 of 108 | 99% | 25.4d | 25.4d to 25.4d | \n| ChatGPT-User | Live user fetch | 38 of 108 | 65% | 3.1d | 25h to 15.9d | \n| Claude-User | Live user fetch | 5 of 108 | 95% | 40.1d | 11.5d to 40.1d | \n| DuckAssistBot | Live user fetch | 0 of 108 | 100% | n/a | n/a | \n\n## Share of new pages found, over time since publishing\n\nOf the new pages published in the period, the share each crawler had visited by that point after publishing. Pages published near the end of the period have had less time.\n\n- Bingbot\n- ClaudeBot\n- PerplexityBot\n- GoogleOther\n- OAI-SearchBot\n- GPTBot\n\n## Weekly median time to discovery\n\nMedian time from publish for the pages published in each bucket (log scale). The number under each bucket is how many new pages it contains. Gaps are buckets with no new pages.\n\n- First crawl, any bot\n- First AI crawler\n- First live user fetch\n\n**About this data.** 330 pages were published since tracking began on 6 Jun 2026.\n\n**219 are excluded** because their URL was already live before they were published. Visits since 12 Oct 2026 carry exact timestamps (78% of the pages in this view have at least one). Before that only the day is known, so first visits are estimated at the middle of the remaining day.\n\n35 blog posts, 15 city pages, 50 service pages, 7 playbooks\n\nmedian. Bingbot was first on 47% of pages\n\nmedian. ClaudeBot was first on 32%\n\nof new pages had a visit from an AI crawler within a day\n\n## Hours from publish to first visit\n\nThe dot is the median across the new pages the crawler has reached (log scale), and the bar covers the middle half of those pages. Diamonds are live fetches for a person. Hollow dots mean fewer than 3 pages reached, so treat them as anecdotes.\n\n- Search engine\n- AI crawler\n- Live user fetch\n\n| Crawler | Type | Pages reached | Still waiting | Median | Middle half | \n|---|---|---|---|---|---|\n| Bingbot | Search engine | 89 of 111 | 20% | 14h | 10h to 14h | \n| ClaudeBot | AI crawler | 88 of 111 | 21% | 14h | 12h to 15h | \n| PerplexityBot | AI crawler | 85 of 111 | 23% | 30h | 10.0h to 5.4d | \n| GoogleOther | Search engine | 62 of 111 | 44% | 38h | 14h to 2.1d | \n| OAI-SearchBot | AI crawler | 77 of 111 | 31% | 38h | 26h to 2.9d | \n| GPTBot | AI crawler | 64 of 111 | 42% | 38h | 38h to 38h | \n| Googlebot | Search engine | 76 of 111 | 32% | 2.2d | 19h to 5.6d | \n| Amazonbot | AI crawler | 85 of 111 | 23% | 2.2d | 34h to 2.8d | \n| PetalBot | Search engine | 80 of 111 | 28% | 2.3d | 38h to 29.0d | \n| Applebot | Search engine | 83 of 111 | 25% | 4.0d | 3.5d to 5.5d | \n| Meta-ExternalAgent | AI crawler | 89 of 111 | 20% | 4.1d | 3.9d to 6.3d | \n| DuckDuckBot | Search engine | 2 of 111 | 98% | 74.3d | 49.8d to 98.7d | \n| ChatGPT-User | Live user fetch | 41 of 111 | 63% | 3.1d | 12h to 14.4d | \n| Claude-User | Live user fetch | 5 of 111 | 95% | 40.1d | 11.5d to 40.1d | \n| DuckAssistBot | Live user fetch | 1 of 111 | 99% | 123.2d | 123.2d to 123.2d | \n\n## Share of new pages found, over time since publishing\n\nOf the new pages published in the period, the share each crawler had visited by that point after publishing. Pages published near the end of the period have had less time.\n\n- Bingbot\n- ClaudeBot\n- PerplexityBot\n- GoogleOther\n- OAI-SearchBot\n- GPTBot\n\n## Monthly median time to discovery\n\nMedian time from publish for the pages published in each bucket (log scale). The number under each bucket is how many new pages it contains. Gaps are buckets with no new pages.\n\n- First crawl, any bot\n- First AI crawler\n- First live user fetch\n\n**About this data.** 330 pages were published since tracking began on 6 Jun 2026.\n\n**219 are excluded** because their URL was already live before they were published. Visits since 12 Oct 2026 carry exact timestamps (77% of the pages in this view have at least one). Before that only the day is known, so first visits are estimated at the middle of the remaining day.\n\n**Method.** Publish times come from the CMS (`published_at`). First visits come from our own server log and leave out any hit that fails verification against the operator's published IP ranges or reverse DNS, so fake bots never count. Pages whose URL was already being crawled before their publish date (republished or migrated pages) are excluded, so the clock only starts for pages that were genuinely new. Medians use only the pages a crawler has reached; \"still waiting\" counts the rest.\n\n**Cite this.** Mojo Dojo, “AI Discovery Index”, mojodojo.io/ai-discovery-index, accessed 20 Oct 2026. Free to reuse under CC BY 4.0 with a link back.", "url": "https://wpnews.pro/news/ai-discovery-index", "canonical_source": "https://mojodojo.io/ai-discovery-index", "published_at": "2026-10-20 00:38:55+00:00", "updated_at": "2026-10-20 00:46:11.551306+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-search", "generative-engine-optimization"], "entities": ["Mojodojo.io", "Bingbot", "ClaudeBot", "GPTBot", "OAI-SearchBot", "PerplexityBot", "Meta-ExternalAgent", "ChatGPT-User"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-discovery-index", "markdown": "https://wpnews.pro/news/ai-discovery-index.md", "text": "https://wpnews.pro/news/ai-discovery-index.txt", "jsonld": "https://wpnews.pro/news/ai-discovery-index.jsonld"}}