GPTBot, ClaudeBot, PerplexityBot: the complete AI crawler list (2026)

Who actually blocks AI crawlers? We measured 52 sites

Almost everything written about this question is opinion. We had data instead: every site run through our free checker gets its robots.txt fetched and parsed. In October 2026 we re-fetched robots.txt for all 52 third-party sites audited to date (our own domains excluded) and counted which AI user-agents each one names and which it disallows from the site root.

Exactly 4 of 52 disallow an answer engine — and the way all three do it is the interesting part. Every one of them blocks GPTBot and ClaudeBot, the agents that collect content for training. Not one of them blocks ChatGPT-User, OAI-SearchBot or PerplexityBot — the agents that fetch a page because a user asked a question right now. Across all 52 sites those three user-agents are disallowed by nobody at all.

4 sites is an anecdote; 4 sites drawing the same line is a position. 3 of them reach it by naming only the training crawlers and staying silent on the rest, and 1 names the live-answer agents explicitly in order to allow them — so at least one author sat down with the user-agent list and made the distinction on purpose. That distinction is the one most of the public argument collapses: "blocking AI" is treated as a single switch when in practice it is two. Every other AI user-agent anyone in the sample disallows is a training or scraping crawler:

• Bytespider — blocked by 7 of 52 sites
• Applebot-Extended — blocked by 5 of 52 sites
• CCBot — blocked by 5 of 52 sites
• GPTBot — blocked by 4 of 52 sites
• ClaudeBot — blocked by 4 of 52 sites
• Google-Extended — blocked by 4 of 52 sites
• meta-externalagent — blocked by 4 of 52 sites
• Amazonbot — blocked by 4 of 52 sites
• Claude-Web — blocked by 2 of 52 sites
• anthropic-ai — blocked by 2 of 52 sites
• cohere-ai — blocked by 2 of 52 sites

The rest of the picture, same sample:

• 22 of 52 have a robots.txt that never mentions an AI crawler at all — the default is not "blocked", it is "never considered".
• 27 of 52 name at least one AI user-agent, and 27 of those name GPTBot, ClaudeBot and PerplexityBot — all but 4 of them to allow, not to block.
• 3 of 52 serve no robots.txt at all, which allows everything by default.

Read the sample honestly: 52 sites is small, and they are self-selected — people who run an AI-visibility checker skew toward small SaaS and indie tools that already care about being cited. That is exactly why the result is worth stating. Among the people most likely to block answer engines, almost nobody does — and the ones who do still let the live-answer agents through. The "should I block GPTBot" debate is loud in public and near-absent in production robots.txt files. The real failure mode in these audits is not blocking — it is the 22 sites that never publish a signal either way.

Every reading we have taken, oldest first: 0 sites blocking an answer engine at 39 sites (16 September), 1 at 48 (20 September), 3 at 50 (28 September), 4 at 50 (2 October), 4 at 50 (4 October), 4 at 52 (5 October), 4 at 52 (7 October). We re-run the survey before any release that quotes it and this page moves with it, including when that makes a tidier earlier claim untrue. Treat the direction as unsettled and the sample as small; what has held for every blocker we have actually found is the shape — whoever blocks, blocks the training crawlers and leaves the live-answer agents alone.

Method: ops/ai-crawler-survey.mjs fetches /robots.txt per host, groups User-agent blocks, and counts a crawler as blocked only on a whole-site Disallow: /. Last measured October 2026.

Why do AI crawlers matter for your site?

Traditional web crawlers (Googlebot, Bingbot) index your pages for search rankings. AI crawlers do something different: they read your content to train LLMs or to power real-time AI answers on platforms like Perplexity, ChatGPT, and Claude.

If these crawlers are blocked or can't access your site, your content is invisible to AI answer engines — the same way a noindex tag makes a page invisible to Google. As AI answers capture more of the "zero-click" question market, this matters increasingly for traffic.

Use our free AI crawler checker to see which crawlers can currently access your site.

Which AI crawlers are active in 2026?

Each entry includes the user-agent string (what robots.txt uses to identify it), the owning company, and copy-paste robots.txt snippets for allowing or blocking.

What does each AI crawler actually do?

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: GoogleOther-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Bytespider
Disallow: /

Should I block AI crawlers?

This is the biggest question in AI visibility right now, and the 52-site sample above already shows what people do when they actually edit a robots.txt: they allow the answer engines and block the scrapers. That maps onto three positions:

• For most sites: allow all. Your content being cited by Perplexity or ChatGPT is traffic and brand exposure. Blocking reduces your AI visibility for minimal gain unless you have a specific content-protection reason.
• If you sell subscriptions or paywalled content: block selectively. Block crawlers from your premium content paths (e.g. Disallow: /premium/) while allowing them on your free content.
• If you have strong IP concerns: block training crawlers, allow search crawlers. Block GPTBot, ClaudeBot etc. (training) while ensuring PerplexityBot (search engine) can still access you. The two use-cases have different implications.

Note: blocking AI crawlers does NOT affect your Google Search rankings in any way. GPTBot, ClaudeBot and PerplexityBot are completely separate from Googlebot.

How do you allow all AI crawlers in robots.txt?

Add this to your robots.txt to explicitly allow all major AI crawlers:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: GoogleOther-Extended
Allow: /

How do you check which crawlers reach your site?

Our free AI crawler checker tests whether your robots.txt is allowing or blocking each major AI crawler — and flags if you're accidentally blocking crawlers you want to allow, or allowing ones you want to block.