Cloudflare and OpenAI pilot, ChatGPT indexes hundreds of millions of new web pages

In July 2026, the internet infrastructure company Cloudflare launched a research pilot with OpenAI to test whether Cloudflare's network signals could help ChatGPT users discover new content more efficiently. Both parties reported positive early results, with the pilot incorporating previously unknown pages into OpenAI's search index, providing ChatGPT with more sources to reference.

The pilot utilized about 1% of Cloudflare's global participation queue, allowing OpenAI to discover and index hundreds of millions of new web pages. OpenAI independently evaluated and ranked these pages based on relevance and usefulness, while ChatGPT Search enabled users to view original content and publishers through backlinks. Observations indicated that automated traffic might make non-existent pages appear noteworthy, with some sites redirecting arbitrary addresses to their homepage and others pointing to duplicate content. Both parties are also testing signals to identify potentially harmful pages, such as fake login pages embedded on legitimate sites, advocating for the marking of single pages rather than entire domains.

Related signals are also used to filter out non-existent or redirecting pages and to assess whether unchanged pages are worth revisiting. Page signals do not constitute access permission; the study utilized web metadata for search and discovery rather than using page content for model training. Site owners can understand and manage how AI systems access their websites through Cloudflare's AI Crawl Control. The next phase will improve page selection and test faster, more reliable change signals.