Bot Preference Sync

mentions 2 type Person feed RSS

// recent coverage 2 mentions

14:30

2026-09-18

searchenginejournal.com

ai-crawlers

Cloudflare Will Write Your Robots.txt, And It Has A Point

Cloudflare announced Bot Preference Sync on August 21, a product that generates a website's robots.txt from three dashboard settings — Search, Agent, and Training — and prepends the entries between "#…

23:39

2026-08-21

runtimewire.com

ai-policy

Cloudflare syncs AI bot rules with robots.txt, closing its own policy gap

Cloudflare product manager Jin-Hee Lee will roll out Bot Preference Sync, a feature that writes AI crawler preferences from Cloudflare's dashboard into robots.txt, closing the gap between published co…

// co-occurs with top 6 entities

Cloudflare 2 Bytespider 1 Googlebot 1 Jin-Hee Lee 1 Cloudflare Radar 1 BotBase 1

// topics top 5 topics

ai policy 2 ai crawlers 1 generative engine optimization 1 ai tools 1 ai infrastructure 1