Making a website AI friendly: what the bots actually fetched from mine
Amir Pournasserian
The author analyzed 18 days of Cloudflare logs, plus MCP telemetry, to see which “AI-friendly” website files were actually fetched by different bot types. ClaudeBot fetched the sitemap 122 times in that period (30 September to 17 October 2026), while no verified AI or search crawler fetched llms.txt in 17 days. The findings shift attention from extra AI-specific files toward making the main page content readable and correctly rendered, with robots.txt/sitemaps helping mainly with indexing and change detection rather than AI citation or training.
Why it matters
Analysis of 18 days of traffic to one personal site found verified crawlers mostly fetched ordinary pages, robots.txt, and the sitemap. Markdown copies attracted verified crawlers within a day, while llms.txt saw no verified training or search crawler requests.