Shoal Research: Robots account for approximately 60% of HTML requests

Research institution Shoal Research released a report stating that bots account for about 60% of HTML content HTTP requests on the Cloudflare network, while humans account for 38.2%. Cloudflare CEO Matthew Prince previously predicted that bot traffic would surpass human traffic by 2027, and later announced that the growth of agent traffic has already led to bots exceeding human traffic.

The report states that AI applications keep users in chat interfaces and summarize content, rather than redirecting them back to the original site. AI crawlers have a much higher crawl-to-return ratio than Googlebot, with ClaudeBot averaging about 4,200 pages crawled per return visit over the past year. Pew research shows that AI summaries have reduced click-through rates from 15% to 8%.

The robots.txt and content exchange traffic model are difficult to maintain. Litigation and private authorization are stopgap measures. Pay-per-request is divided into a closed market with keys and open protocols using HTTP 402 and stablecoins like x402 and MPP. Most websites still offer free access or block it.