Can AI search find your website? What Cloudflare and OpenAI are changing about crawler visits

What are Cloudflare and OpenAI changing?

Signals from Cloudflare’s network help OpenAI’s search crawler find new pages, skip dead ones and come back only when a page is worth another visit.

The pilot started in July 2026¹. It drew on a participating group of about 1% of Cloudflare’s network, and Cloudflare says it helped OpenAI find and index “hundreds of millions of new web pages”¹. OpenAI still decides on its own how those pages rank¹.

Cloudflare found a lot of wasted visits: bots asking for files that are not there, sites that send almost any address to their homepage, and addresses that lead to copies of the same content¹. Filtering now keeps crawlers away from pages that do not exist or that redirect elsewhere, and “change signals” tell them when a page is worth another visit¹. 1 catch: finding new pages “can initially increase crawling” while they are evaluated¹.

Cloudflare also stresses that “a signal about a page doesn’t grant permission to access it”, and that the research uses data about pages, not page content for model training¹.

What changes for the crawlers that visit your site?

Cloudflare now checks whether Verified bots (its list of identified bots) respect your published preferences, and whether they slow down when asked.

The main way a site states its preferences is a robots.txt file, and Cloudflare’s system checks whether a bot’s traffic matches it over time². A robots.txt sets minimum rules: anything it does not mention is not automatically allowed². A site can also ask a busy bot to back off with an HTTP 429 “Too Many Requests” reply, and the system checks whether the bot slows down².

For now Cloudflare uses this internally². It plans to make it available to all its customers, who will choose how strict to be with Verified bot traffic, and to publish the scores on its Radar site².

Does this apply to every website?

Not yet. It is a pilot with 1 AI company, on a small part of Cloudflare’s network.

Most sites, on Cloudflare or not, are not part of it. The next phase will test better page selection and faster change signals, and measure what happens after a page is found¹. Our view: other AI crawlers may behave differently, so the checks below are worth doing whichever ones visit you.

What the pilot does show is the kind of page AI search looks for: the manual for an exact model, current requirements and fees, a restaurant’s latest menu and opening hours¹. Cloudflare notes such pages “can be valuable even if few other sites link to them”¹.

What should you check this week?

1. Look at AI Crawl Control. If your site runs on Cloudflare, its AI Crawl Control (opens in a new tab) shows how AI systems access your site and lets you manage it¹. Decide which crawlers you welcome.
2. Tidy dead links and stray redirects. Our view: if every wrong address lands on your homepage, crawlers waste their visits there, and so do customers.
3. Make the useful pages easy to reach. Prices, fees, opening hours, manuals and answers to common questions should each have a clear page, linked from your own site.
4. Read your robots.txt once. Check that it says what you mean today, for search engines and AI crawlers alike.