Monday, 3 August 2026 Sign in

Cloudflare announces new AI crawler classification system with default blocking policy for training and agent bots

Feature update Provisional 85% confidence first seen 3 July 2026, 10:27

Cloudflare introduced a new framework categorizing AI crawlers into three types—Search, Agent, and Training—to give website owners granular control over different automated traffic. Beginning September 15, 2026, new domains will have Training and Agent crawlers blocked by default on ad-supported pages, while Search crawlers remain permitted.

Source coverage

3 Jul, 10:27 Cloudflare sets deadline to block AI crawlers that bundle search with AI training TLDR newsletter 6 Jul, 08:30 Cloudflare Drew A New Line For AI Crawlers The Neuron newsletter

The daily briefing

Every AI story that matters,
in your inbox by 8am.

TLDRocket reads 60+ sources, removes duplicate coverage, and summarises the day in two minutes. Free, no spam, unsubscribe anytime.

TLDRocket 3 August 2026

The story of AI in August 2026 is one of consolidation meeting enforcement. Alibaba's Qwen3.8-Max, a 2.4-trillion-parameter mixture-of-experts model with a million-token context window, arrives as a direct challenge to OpenAI and Anthropic's frontier offerings—priced at $2 per million input tokens, it's positioning itself as the capable-and-cheaper alternative that China's AI push has long promised. But the more consequential development is structural: enterprises are finally getting the plumbing to actually deploy these models. June, backed by Salesforce's Marc Benioff, raised $20 million to automate AI integration into legacy systems, scanning corporate infrastructure and generating deployment workflows without expensive consultant fees. This solves what's become clear as the real bottleneck—not model capability, but implementation friction in messy, heterogeneous corporate environments.

Read the full briefing →

Cloudflare announces new AI crawler classification system with default blocking policy for training and agent bots

Source coverage

Every AI story that matters, in your inbox by 8am.

Every AI story that matters,
in your inbox by 8am.