Not everything on the web ever makes it into a search engine. Because traditional crawlers typically find new pages by following links from other sites, valuable content — a washing machine manual, a local restaurant's menu — can remain invisible in search results simply because no other site ever linked to it.

A joint project from Cloudflare and OpenAI, launched in July, tackles exactly that gap. Using traffic signals from Cloudflare's network, hundreds of millions of pages that slip past conventional crawlers are being identified and added to ChatGPT's search index.

Cloudflare-and-OpenAI

The classic discovery method works by following links from one page to another. The new approach instead relies on data about which pages real users are actually visiting. That means content that never received a single inbound link but still serves a concrete need can now be discovered.

One bad page shouldn't sink an entire domain

A single page on a site carrying a security issue doesn't mean everything else on that domain is harmful. Evaluating problematic pages separately from the rest of a site's content could preserve the site's visibility in search results.

That said, the technical details of how this approach was implemented in the pilot still need to be independently verified.

Crawling resources could be used more efficiently

Automated crawlers waste resources requesting pages that don't exist or content that is duplicated. Analyzing traffic patterns could make it easier to identify such pages, streamlining the crawling process.

What does it mean for publishers?

As the number of discovered pages grows, publishers may gain an entirely new channel through which readers reach their content. Still, being indexed doesn't automatically translate into more visitors.

Measuring the project's real impact will require closely monitoring the traffic and click-through rates that ChatGPT sends to sites. Site owners' ability to control AI crawlers' access will also be an important part of the process.