AI crawler census: which of the top 5,000 sites block AI crawlers (robots.txt, September 2026)

<p dir="ltr">A crawler-level census of robots.txt access across a random sample of 4,999 sites drawn from the Tranco top 1,000,000 (tranco-list.eu), checking whether each site blocks or allows 20 named crawlers across four categories: AI-training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent, Bytespider, Applebot-Extended, Amazonbot, cohere-ai, Diffbot, omgili, TimpiBot), AI-search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot), AI-search user-action fetchers that only run during a real person's live question (ChatGPT-User, Perplexity-User, Claude-User), and classic search crawlers (Googlebot, Bingbot) as a baseline.</p><p><br></p><p dir="ltr">Collected 2026-09-07.

Of the 4,999 sites checked, 2,771 (55.5%) serve a robots.txt file at all; percentages in this dataset are computed against that base. A crawler counts as blocked when the most specific robots.txt group that applies to it disallows the site root.</p><p><br></p><p dir="ltr">This is the raw JSON output behind the free, public checker and write-up at ai-visibility.lastminutedealshq.com/data, which also documents the full methodology and lets anyone check a single site's own robots.txt against the same crawler list.