Cloudflare AI Control: Check Your Settings Before You're Blocked

Cloudflare's new AI crawl controls, which now block AI training bots by default for many sites, risk cutting you off from AI-powered search unless you actively review your settings. Since a policy shift on 15 September 2026, announced by Cloudflare, any new site on Cloudflare, or any existing site that displays ads, is likely telling bots from OpenAI and Google not to use its content for model training. A default setting might be making you invisible to the next generation of AI.

According to data from Cloudflare, while fewer than 1% of sites on its network block traditional search crawlers, a reported 17% now restrict AI training. This means there is a one-in-six chance your own site is blocking these critical bots without your knowledge, putting you at a disadvantage. This is a choice that should be made deliberately, not by a default you never saw. Here is how to check your settings and ensure you are not accidentally blocking the crawlers that determine your AI visibility.

What Changed with Cloudflare's AI Controls?

On 15 September 2026, Cloudflare replaced its single AI bot switch with three granular controls for 'Search', 'AI Training', and 'AI Agents', ending the all-or-nothing choice for site owners. The move, announced in a press release, was designed to give site owners separate controls for bots that index for live answers versus those that feed the machine's brain for future models.

Previously, blocking AI bots was a blunt instrument. Now, a site owner can theoretically allow a bot like Googlebot to crawl for inclusion in search results while simultaneously telling it not to use that same content to train the next version of its Gemini model. This added control, however, comes with a new and potentially costly default setting.

The New Default: Are You Blocking AI Crawlers by Mistake?

The most significant change is not the controls themselves, but how they are configured out of the box. For any new domain added to Cloudflare after 15 September 2026, or for pages on any site that display ads, the new default settings now block 'AI Training' and 'AI Agent' crawlers. Only 'Search' crawlers are permitted.

If you have not touched your settings, you may be blocking the very bots that feed engines like ChatGPT. The result is a slow erosion of what is AI visibility, as your brand and its expertise are quietly excluded from the knowledge base of future models. Fortunately, checking your configuration is straightforward.

To review your settings, log in to your Cloudflare dashboard, select your domain, and navigate to 'Security' > 'Bots'. You will find the 'AI Crawl Control' panel there. This is where you can see which categories of bots are currently allowed or blocked. If you see 'AI Training' is blocked, you need to make a conscious decision about whether that is the right strategy for your business.

Within the dashboard, you will see three toggles. If the 'AI Training' toggle is off (greyed out), you are actively blocking bots like GPTBot from using your content to train future versions of ChatGPT. While this might seem like a good way to protect your intellectual property, it is a decision with long-term consequences for your brand's presence in AI. Leaving the 'Search' toggle on (green) is essential for immediate visibility in tools like Perplexity and Google AI Overviews, but the 'Training' toggle governs your brand's future.

Search vs. Training: The Choice You Must Make

The choice Cloudflare presents is between 'Search' crawlers, which gather data for immediate answers, and 'AI Training' crawlers, which feed future models. 'Search' crawlers are those that gather information to answer a user's immediate query, like Perplexity's bots or the ones that feed Google's AI Overviews. 'AI Training' crawlers, such as OpenAI's GPTBot, collect vast amounts of data to build and update the foundational models themselves. 'AI Agents' are an emerging category of bots that act on a user's behalf to perform tasks.

The distinction is not always clean. Cloudflare itself notes that as of September 2026, 36.6% of verified crawler traffic comes from mixed-use bots. This means a simple block on 'training' is a blunt tool that could unexpectedly reduce your visibility in search answers today. Data from Cloudflare Radar shows that in Q3 2026, 47.62% of all AI crawler requests were for training. Blocking them means cutting yourself off from nearly half of all AI-related web activity.

Want to get recommended by AI?

Check your AI search visibility, then let the Honeyb agent write, fix, and earn what gets you recommended. Free to start.

Free AI visibility checker

Blocking training bots might feel like a safe way to protect your content, but it carries a long-term risk. AI models that are not trained on your data cannot learn your brand's name, its products, or its expertise. By opting out of training, you may be ensuring you are never recommended by the next generation of AI assistants, ceding the ground to competitors who chose to remain visible.

The stakes are higher than many realise. AI answers are no longer a niche feature. In a recent capture of 1,825 commercial searches, our own measurement at Honeyb found that Google showed an AI Overview 93% of the time. When a customer asks a question about your industry, an AI is now the default intermediary. If that AI has not been trained on your content because of a restrictive crawl setting, it cannot recommend you. It will recommend a competitor who was included in the training data instead.

How to Enforce Your Choices with robots.txt

Bot Preference Sync is a Cloudflare feature that automatically updates your site's robots.txt file to reflect your dashboard settings, ensuring your preferences are communicated in a standard format. This is important because the robots.txt file is the long-standing, universally recognised way for websites to give instructions to web crawlers, independent of Cloudflare's own infrastructure.

As of September 2026, major players like Google and Apple already support these more granular `robots.txt` directives, allowing them to distinguish between crawling for search and crawling for training. By using Bot Preference Sync, you ensure your preferences are communicated clearly and in a standardised format that compliant bots will respect. This provides an extra layer of control and a public declaration of your site's policy, independent of Cloudflare's own infrastructure.

Manually editing your robots.txt file remains an option for those who need even more specific rules, such as blocking a single misbehaving bot while allowing others in the same category. However, for most site owners, ensuring the Cloudflare dashboard settings are correct and allowing Bot Preference Sync to manage the robots.txt file is the most effective strategy. It combines ease of use with adherence to web standards, a crucial combination for managing your AI visibility.

Which AI Crawlers Should You Allow?

For most businesses, the goal is to be seen and recommended by AI. This means you should generally allow both 'Search' and 'AI Training' crawlers from reputable sources. Cloudflare has said it designates crawlers as 'accountable' if they respect content use instructions, reportedly including those from Apple, Google, and Microsoft.

Here is a practical table of the most important AI crawlers and our recommended settings for maximum visibility. You can manage these in your Cloudflare settings and by using a correctly formatted robots.txt file, which Cloudflare's Bot Preference Sync can help generate.

This configuration ensures you are discoverable in today's AI-powered search results while also allowing your content to inform the foundational models of tomorrow. For a deeper dive into specific crawlers, our guide to GPTBot is a useful reference. The crucial step is to move away from the default and make an active choice that aligns with your marketing goals.

A default setting should not dictate your company's future in AI search. The control is there, but you have to use it; the quickest way to find out where you stand is to run a free AI visibility check and see which answers name you today.