The UK CMA requires clearer opt-out options to prevent AI usage of content while preserving search rankings, with broader global rollout anticipated after testing.

A proposed robust remedy is a full separation of Google’s crawler into two systems: one for search indexing and one for AI scraping, allowing publishers to block AI use independently.

Internal 2024 slides suggest revenue pressures influence policy, reinforcing the case for separating crawlers to empower publishers against AI training.

Cloudflare’s CEO highlights that Googlebot captures a much larger swath of the web than rivals, fueling AI-driven traffic that could sideline human readers and original publishers.

Google has merged its web crawling for search with its AI data needs, using the same crawler to feed Gemini and other AI models.

Critics warn this bundling makes it harder for publishers to block AI training without losing search traffic, potentially eroding incentives to create original content.

Technological and regulatory developments could push Google to reform its data-use practices, though incentives to monetize data often influence corporate behavior.

There is a pending shift in policy and practice that could change how publishers monetize and control their content in the AI era.

Even with opt-out options, Google’s crawlers still combine data for search and AI training, meaning publishers must trust Google to honor opt-outs unless crawling is structurally split.

A long-standing issue remains: Google’s crawler bundles data collection for both search and AI, placing trust in Google unless a structural separation of search indexing and AI scraping is implemented.

Regulatory and industry pushback includes testing an opt-out for AI data use without affecting search rankings, with the UK CMA ordering clearer opt-outs for content owners.

Cloudflare will block mixed-purpose crawlers for ad-supported sites starting Sept. 15, potentially cutting Google’s access to many sites.