Cloudflare and OpenAI pilot: network signals from ~1% of Cloudflare's traffic helped OpenAI's search crawler find and index hundreds of millions of new pages

Why it matters

【Key points】

On October 15 Cloudflare shared early results from a pilot with OpenAI that began in July: signals from Cloudflare's network help OpenAI's search crawler find new pages, and OpenAI independently decides what to index and how to rank it. Using a cohort of about 1% of Cloudflare's network, the pilot let OpenAI discover and index hundreds of millions of previously unknown pages for ChatGPT search.

【Verifiable】

Cloudflare blog: https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/

Methods described: filter out nonexistent URLs, catch-all redirects and duplicates before a crawler fetches them; use change signals to decide when a page is worth revisiting; test flags for single compromised pages (such as a planted fake login form) without penalising the whole domain. Cloudflare says the work uses web metadata, not page content, and is not for model training; a signal does not grant permission to access a page, and site owners' choices still apply. Next: better page selection, faster change signals, and measuring outcomes after discovery.

【Our read】

1. AI search is bottlenecked on discovery as much as on models. A CDN knows which pages really exist and when they change, which works like a live sitemap for crawlers.

2. Cloudflare has pushed default blocking of AI crawlers and pay-per-crawl; now it is also a matchmaker for OpenAI. That widens its role from gatekeeper to broker, with leverage over both site owners and AI companies.

3. The post doesn't say whether these signals are exclusive to OpenAI, or whether sites in the cohort opted in one by one. If exclusive, it is a structural index-coverage edge for ChatGPT search that rival AI search firms and publishers should watch.