10:37
2026-10-08
dev.to
ai-crawlers
Keeping AI search bots in and AI training bots out on a small Next.js site
A developer maintaining the Next.js site for Capital Complete Solutions, a Birmingham property inventory company, configured robots.txt named user-agent groups to block AI training crawlers such as GPā¦