{"type": "article", "title": "robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained", "publisher": "Web Pulse", "url": "https://wpnews.pro/news/robots-txt-for-ai-crawlers-gptbot-claudebot-perplexitybot-explained", "original_source": "https://dev.to/citescore/robotstxt-for-ai-crawlers-gptbot-claudebot-perplexitybot-explained-46mc", "published": "2026-10-10T20:43:44+00:00", "accessed": "2026-10-10", "id": "robots-txt-for-ai-crawlers-gptbot-claudebot-perplexitybot-explained"}