Cloudflare’s New Disallow AI Training Setting Ends the Search vs. Training Tradeoff

Cloudflare Forces the Mixed-Use Crawler Reckoning

On September 15, Cloudflare removed one of the most persistent either-or decisions in web publishing.

The new Disallow AI Training setting lets site owners refuse AI training access while preserving search crawlability for the same mixed-use bots.

Applebot, Googlebot, and Bingbot can now receive a training opt-out without being blocked from organic search.

That change, documented in Cloudflare’s announcement, moves crawler governance from blunt blocking to intent-based control.

The underlying data makes the stakes explicit.

Fewer than 1% of Cloudflare-hosted sites block search bots.

Seventeen percent activate some mechanism to block training.

That gap is not indifference; it is a rational response to advertising and subscription models that only pay when a human arrives.

For years, robots.txt could express a preference but could not identify the crawler, classify its purpose, or stop noncompliant actors.

Cloudflare’s network-level control now handles identification, classification, blocking, and reporting in one place.

Inside the Accountable Crawler Standards

Cloudflare classifies bot behavior into three categories: Search, Training, and Agent.

A mixed-use crawler performs Search and Training in a single pass, which created the original tradeoff.

The new setting publishes a robots.txt Disallow preference for training while keeping Accountable mixed-use crawlers available for search.

It is available only for Training traffic, not for Search or Agent controls.

Agent behavior lacks a mature directive standard, so Cloudflare will revisit that layer as protocols such as ai-prefs develop.

The updated setting logic now looks like this:

• Allow — all crawlers pass unless another rule or WAF policy blocks them.
• Disallow AI Training — a no-training preference is published; Accountable mixed-use crawlers stay searchable.
• Block on pages with ads — all crawlers, including mixed-use crawlers, are blocked only on pages serving ads.
• Block — all crawlers, including mixed-use crawlers, are blocked.

Importantly, Block and Block on pages with ads now apply to mixed-use crawlers as well.

That means a decision to block training-only crawlers no longer silently spares Googlebot, Bingbot, or Applebot.

The legacy Block AI Bots toggle and Managed Robots.txt are being retired in favor of Bot Preference Sync.

Existing configurations carry over automatically, and new ad-supported domains receive a more restrictive default: Search stays allowed, Training is set to Disallow AI Training, and Agent is blocked on ad pages.

An ads-only version of Disallow AI Training is not possible because robots.txt cannot enumerate Cloudflare’s live list of ad-serving pages.

For most existing Cloudflare customers, no action is required; current settings carry over automatically.

If a site wants mixed-use crawlers removed entirely, selecting Block will stop Applebot, Bingbot, and Googlebot, including search.

Applebot and Googlebot: Search Ranking Protected

Applebot honors a robots.txt Disallow for Applebot-Extended for training.

Apple has stated that the training opt-out does not affect search ranking.

The same separation exists for Googlebot through the Google-Extended user agent.

Google also provides a webmaster portal toggle to exclude a site’s content from generative search results.

Google has confirmed that disallowing Google-Extended does not alter search ranking.

Apple currently supports summary preference through the nosnippet directive and paywalled content labeling.

URL-level inspection is still in progress for next year.

Google has additional URL-level transparency tools for Google-Extended on its near-term roadmap.

Bingbot’s Time-Bound Gap

Microsoft’s Bingbot presents a more complex interim state.

Bing currently supports the NOARCHIVE meta tag for AI training preference, and Webmaster Tools offers granular controls.

Microsoft is building a robots.txt-level no-training mechanism at the domain level, targeted for early 2027.

Until that support launches, Cloudflare’s Disallow AI Training setting will not automatically convey a no-training preference to Bing through robots.txt.

In practical terms, that matches the previous Training Block behavior, which also did not apply to Bingbot as a mixed-use crawler.

Amazon, Anthropic, Meta, and OpenAI: An Uneven Split

Cloudflare also categorizes relevant training crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable.

For Amazon, Anthropic, and OpenAI, the separation between training and search crawlers is well documented.

Meta’s crawler documentation is less clean, with general and user-triggered crawler types rather than a widely published training/search split.

That means the control plane can still enforce block rules, but the public documentation does not give SEO teams the same clean user-agent mapping.

Because those organizations use distinct bots for training and search, Cloudflare can block the training crawler without affecting search discoverability.

That distinction matters for SEO teams because a robots.txt rule aimed at GPTBot does not control OAI-SearchBot.

ClaudeBot and Claude-SearchBot follow the same separation.

The Crawler Audit SEO Teams Must Run Now

The Cloudflare change is not a reason to automate a blanket AI block.

It is a reason to map every crawler decision by vendor and by intent.

For OpenAI, disallowing GPTBot opts out of training; allowing OAI-SearchBot preserves ChatGPT Search inclusion.

For Anthropic, ClaudeBot and Claude-SearchBot perform separate jobs.

For Amazon, Amazonbot is the training target, while Amzn-SearchBot supports search experiences.

Google’s model is different.

Google-Extended controls Gemini and Vertex AI training and grounding, but AI Overviews run on the same Googlebot search infrastructure.

There is no separate AI Overviews opt-out.

Bing similarly folds search and AI uses into Bingbot, which is why the NOARCHIVE tag and Webmaster Tools controls remain critical through early 2027.

Perplexity is another case to audit precisely: PerplexityBot is documented as search-only and not used for foundation-model training.

Blocking it removes Perplexity search visibility without improving training protection.

Perplexity-User generally ignores robots.txt.

Meta and ByteDance remain the least standardized.

Meta does not publish a clean training/search split, and independent crawler monitoring has repeatedly documented ByteDance’s Bytespider ignoring robots.txt directives.

That makes network-level enforcement the only reliable layer for aggressive crawlers.

Robots.txt Grouping Creates Hidden Exposure

A named crawler group containing only Allow:/ can override a wildcard Disallow rule.

Because robots.txt applies the single most specific matching group, site owners can accidentally expose content while believing a global rule protects it.

SEO teams should read robots.txt the way a crawler would: count the User-agent groups, compare named groups with wildcard rules, and eliminate over-broad Allow:/ declarations.

Raw HTML remains the source of truth because major AI crawlers have historically not rendered JavaScript at scale.

A 2024 crawl study tracked 569 million GPTBot fetches against 4.5 billion Googlebot fetches, with ChatGPT hitting 404 pages in more than a third of requests.

That does not mean AI crawlers are permanently incapable of rendering, but it does mean server-rendered content and clean static directives still carry outsize weight.

Timing and Validation Vary by Vendor

OpenAI says OAI-SearchBot can take up to 24 hours to reflect robots.txt changes.

Amazon caches robots.txt for up to 30 days.

Google Search Console’s generative AI control rolled out to all sites on August 31 and defaults to include, with exclusions taking effect within one to two days.

Excluded sites receive no traffic or impressions from those generative features.

Cloudflare’s Bot Preference Sync eliminates some delay at the network layer, but SEO teams should still validate against server logs and published crawler IP ranges.

Firewall and CDN rules can override or undermine robots.txt signals.

Google, OpenAI, and Perplexity recommend allowing their published IP ranges; Anthropic warns that IP blocking can prevent its crawler from reading robots.txt without guaranteeing opt-out.

A crawl audit that stops at robots.txt is not an audit.

AI Summaries Are the Next Zero-Click Battleground

Training opt-outs solve the ingestion question.

AI summaries solve a different question: whether a potential visitor needs to reach the site at all.

Cloudflare’s framework already requires Accountable operators to offer an AI summary opt-out.

The next phase will add granularity over how much content appears in a summary, set once in Cloudflare rather than operator by operator.

A site-wide yes or no is too blunt for this problem.

More than half of consumers read AI summaries in search, and those users are over 40% more likely to end their search after reading one.

That pressure can reduce visits.

Yet consumers referred by AI Search convert at between three times and more than five times the rate of traditional search referrals.

Ad-funded publishers may optimize for volume.

A retailer may prefer fewer visitors who arrive with stronger purchase intent.

Cloudflare’s position is not to pick a winner, but to expose the tradeoff and provide controls.

Standards such as ai-prefs will likely mature into the next layer of machine-readable preferences.

A Training-Resistant Web Without Losing Discovery

Cloudflare’s September 15 rollout separates search visibility from AI training consent for the first time at network scale, and teams that treat crawler preferences as a structured audit will control both traditional search and generative answers. For teams building technical AI-search governance that must survive crawler changes, programmatic SEO and AI automation is how Andres SEO Expert approaches it — contact Andres SEO Expert.