AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare announced the change on July 1, and most of the coverage since then has focused on Google. The more useful part is what it asks of everyone building agents, and what it offers them in return.
Cloudflare has replaced its single block-AI-bots switch with three categories. Search covers bots that index a page to answer questions about it later. Agent covers automated systems acting in real time for a user, including ChatGPT’s fetch bot and browser-driving agents. Training covers crawlers that pull content into a model’s weights. The controls went live on July 1 for every customer, including the free tier.
From September 15, the defaults change. Training and Agent will be blocked on pages that display ads, while Search will stay allowed. The new defaults apply to domains newly onboarding to Cloudflare, new sites set up by existing customers, and all existing free-tier customers.
Anyone who does not want them can opt out through their security settings before the date.
Cloudflare’s logic is that an advertisement is evidence that a page was built for a human to land on. A search crawler that sends a reader back is a referral. A bot that reads the page and hands the answer to someone else is not.
What AI agent crawlers run into now
Agentic deployments have been built on the assumption that the open web stays open. A research agent fetches a competitor’s pricing page. A monitoring tool checks a supplier’s announcements. A customer-service agent pulls a manufacturer’s specification sheet. None of this involves a licence, and until now, none of it needed one.
Cloudflare sits in front of a large share of the world’s web traffic, and its blocks operate at the network level rather than as a robots.txt suggestion a crawler can ignore. Ad-supported pages are exactly the pages agents want, because that is where news, reviews, pricing and product coverage live. The failure mode for an enterprise agent is not a lawsuit. It is silence, or an answer built from whatever it could still reach.
There is a Google-shaped complication. Googlebot crawls for both search and training in a single bot, so under the most restrictive rule, a site that blocks Training also blocks Googlebot. Cloudflare CEO Matthew Prince said the company hopes the changes will “encourage mixed-use crawlers to separate search from agent use and training”, which is a polite way of saying the pressure is the point.
Getting permission for AI agent crawlers
Anyone running agents should start by working out which of their Cloudflare accounts will read as Agent-class. The classification is behavioural rather than something you opt into, so a research agent that browses in real time is caught whether or not its operator thinks of it as a crawler.
Expect degraded coverage rather than a clean failure, because the block lands on ad-supported pages and leaves the rest reachable. Negotiated access, not a rewritten user-agent string, is the way through.
Publishers have a different homework list. Check your tier first, since existing free-tier customers are moved to the new defaults automatically on September 15, a detail most coverage skipped. Then decide whether blocking Training is worth what it costs, because it takes Googlebot with it and your search visibility along with it.
The mechanism worth watching is the money. Pay Per Crawl is becoming Pay Per Use, with Ceramic.ai paying publishers when their content appears in AI search results, and You.com paying when an agent reaches premium content. Cloudflare says more than half of AI crawler traffic is spent re-fetching pages that have not changed, so there is waste on both sides worth pricing out.
This is the first round of the content fight, where the answer on offer is a rate rather than a wall.
One weakness lies in the taxonomy itself. Search, Agent and Training are behaviours the AI companies declare about their own bots, and a firm that would rather not have its training run classified as training has an obvious incentive. The announcement does not explain what stops it.
Access to the open web has been free and unlimited for thirty years, and the bill is now itemised. Agent builders who sort out their access before September have a workable problem. The ones who find out from a 403 will be rebuilding on the fly.
See also: Visa ChatGPT integration enables AI agent retail purchasing
Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.
AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
