Updates to managing AI crawlers on your account

About a year ago, we launched Content Independence Day and introduced a simple way to block AI bots from scraping your sites without your explicit permission.

We're writing to let you know about changes to our controls for AI crawlers taking effect today, and rolling out over the coming week. These changes give you more precise control over how different types of AI crawlers interact with your content.

What's changing?

- Smarter security setting options for Training crawlers: Starting September 15, the recommended setting for AI Training crawlers will shift from Block to Disallow AI Training. Disallow AI Training allows major search crawlers - like Applebot, Googlebot and Bingbot - to index your site for search results while instructing them not to use your content for training.

- Automatic migration: If you previously had "Block AI Bots" or "Managed Robots.txt" enabled, your settings will be migrated automatically, over the next week, to the new Search, Training, and Agent controls. If Block AI Bots was set to Block, you'll now have Search: Allow, Training: Disallow AI Training, and Agent: Block on pages with ads. These changes take effect today – the UI changes will roll out in a slower controlled release over the coming week. If you make manual changes to your settings during this time, your manual changes will be preserved.

- No change if you haven't configured anything: If you have never adjusted these settings, your configuration remains Allow - nothing changes for you.

What do you need to do?

Most customers won't need to change anything. The new controls are more precise than the old Block AI Bots switch, so if you had it enabled it's worth a quick look at your new Search, Training, and Agent settings.

These changes will roll out to all Cloudflare customers over the coming  week. If you make changes to your settings before the migration completes, your changes will be preserved. You will know the migration is complete when the Block AI Bots switch is removed from your dashboard.

If you want to review or adjust your settings, you can do so in your Security Settings. Our documentation explains all available controls and how they work. Please reach out to crawlercontrols@cloudflare.com.

Thank you for being part of the Cloudflare network and helping us build a better Internet!

Best regards,

The Cloudflare Team