On 1 July 2026, Cloudflare replaced its blunt allow-or-block approach to AI crawlers with a three-way classification — Search, Agent and Training — and gave every customer, including those on the free tier, controls to manage each category separately. The change that will actually move traffic comes later: from 15 September 2026, Cloudflare will set new defaults that block Training and Agent crawlers on pages carrying advertising, while leaving Search allowed. Those defaults are scoped — they apply to new domains, new sites added by existing customers, and free-tier accounts that have not changed their settings — but for anyone running a site behind Cloudflare, or operating a crawler that touches one, it is a structural shift in how automated access to the web is negotiated, not just a new toggle.

From allow/block to a purpose-based taxonomy

The core idea is to classify a bot by what it does with content rather than by whether it is broadly AI. Cloudflare's three buckets are straightforward: Search crawlers collect or index content to answer questions about it later; Agent crawlers act in real time on a person's behalf, such as chat-fetch bots and browser-use agents completing a task with a human waiting on the result; and Training crawlers take content to train or fine-tune a model, permanently absorbing it into the model's weights. Cloudflare classifies other behaviours too — ad verification, feed fetching, agentic transactions — but these three are the ones it wants every site owner to be able to govern independently.

The important design decision is how multi-purpose bots are handled. Rather than tag a crawler with a single primary purpose, Cloudflare's updated taxonomy tracks a bot under every purpose it serves, and it is encouraging operators that run one crawler for search indexing, agent tasks and training collection to split that automation into separate, declared crawlers. The stated rationale is transparency: a site owner should be able to see why a given bot is visiting and decide accordingly, which is only possible if the bot's purpose is legible in the first place.

The mixed-purpose crawler question

The multi-purpose handling is where the policy becomes sensitive. Cloudflare says crawlers that combine Search and Training will be affected by Training-block configurations, because the most restrictive applicable policy wins. That means a crawler used for both search indexing and AI training can be blocked on ad-supported pages even when pure Search crawling remains allowed. Independent SEO coverage has highlighted Googlebot as the obvious example of why this matters: publishers want search visibility, but may not want the same crawler relationship to imply AI-training permission. Cloudflare's own data puts roughly 36% of crawler activity in this mixed-use, Search-and-Training bucket.

This is the pressure point, and Cloudflare has been fairly direct about the target. Its argument is that the largest search provider has access to roughly twice the content available to leading AI companies, because remaining discoverable in search effectively requires consenting to AI use as well. Google offers a Google-Extended opt-out in robots.txt that is meant to separate AI training from search, but it does not keep content out of AI Overviews, which draw on the search index itself. Cloudflare's stated goal is to push providers to separate their crawlers structurally, so that a publisher can be indexed for search without automatically feeding model training and inference. Whether that pressure works is an open question.

The new defaults, and what operators must do before 15 September

The default change is scoped, but broad in reach. From 15 September, the blocking applies automatically to new domains onboarding to Cloudflare, to new sites added by existing customers, and to free-tier customers who have not changed their settings. Existing paying customers keep their current configuration and can override the defaults at any time, and Cloudflare says it will send advance notifications ahead of the date. The practical takeaway for site operators is a calendar item: if you rely on Training or Agent crawlers reaching your ad-supported pages — for inclusion in an AI product, for an agent integration, or because a mixed-purpose search bot matters to your traffic — you need to review your zone Security settings and opt out before 15 September, or accept the new posture. Cloudflare is describing the next two months as an engagement and testing window, so the specifics could still shift before they land.

The monetisation layer underneath

Blocking is only half the direction of travel. Cloudflare is also expanding the commercial layer around machine access. Its earlier Pay Per Crawl feature created a way to charge crawlers for fetching content; the newer Monetization Gateway waitlist points toward charging for broader resources sitting behind Cloudflare — web pages, datasets, APIs and even MCP tools — using the x402 payment protocol, which builds on the long-dormant HTTP status code 402 "Payment Required." That code has sat unused in the web's specification since the 1990s because per-page payment was unworkable for human users; automated agents remove that friction, which is what makes a machine-payable web plausible now. In plain terms, Cloudflare is trying to make automated access not just blockable, but billable. Alongside this, it is proposing machine-readable content-use signals in robots.txt and identity tooling such as Web Bot Auth and a BotBase directory, so that verified bots can be checked against a site's declared preferences. Verification now confirms only a bot's identity; access depends on its classification and the site's policy, where previously all verified bots were allowed by default.

The context Cloudflare cites for all this is a web where, by its own network measurement, automated traffic has passed half of the total, and where more than half of AI crawler requests re-fetch pages that have not changed — bandwidth spent re-reading unchanged content, which the new content signals are also meant to reduce.

Key Takeaways

  • On 1 July 2026, Cloudflare began classifying AI bot traffic into three categories — Search, Agent and Training — for all customers including the free tier, tracking multi-purpose bots under every purpose they serve.

  • From 15 September 2026, new defaults block Training and Agent crawlers on ad-displaying pages (Search stays allowed) for new domains, new sites from existing customers, and unchanged free-tier accounts; existing paying customers keep their settings, and anyone can opt out in Security settings before then.

  • Because mixed Search-and-Training crawlers are judged under the most restrictive policy, a crawler used for both — Googlebot being the example independent SEO coverage highlights — can be blocked on ad pages even where Search is allowed; Cloudflare's lever to push providers to separate their crawlers, since ~36% of crawler activity is mixed-use.

  • Cloudflare is pairing the blocks with a commercial layer — the Monetization Gateway waitlist uses the x402 protocol (built on HTTP 402) to let owners charge for pages, datasets, APIs and MCP tools — plus machine-readable content-use signals and Web Bot Auth. But those signals remain voluntary, so adoption by AI operators, not the policy itself, will decide the outcome.