Cloudflare Drew A New Line For AI Crawlers
The Neuron ● Covered by 2 sources
Cloudflare introduced a new classification system for AI crawlers that divides them into three categories—Search, Agent, and Training—allowing website owners to manage different types of automated traffic separately rather than blocking all AI bots at once. Starting September 15, 2026, new domains will have Training and Agent crawlers blocked by default on ad-supported pages, while Search crawlers remain allowed. Website owners can now grant different permissions based on how bots use content, ranging from immediate interaction-only access to full reproduction rights, giving them finer control over their content's use in AI systems.
Why it matters
Cloudflare introduced new AI traffic controls that split bot behaviors into three categories: Search, Agent, and Training. Starting September 15, 2026, new domains will block Agent and Training bots by default on ad-supported pages while allowing Search bots to continue.
