AI visibility · Reference
GPTBot, ClaudeBot, PerplexityBot: the complete AI crawler list (2026)
TL;DR: There are 8+ named AI crawlers from major AI companies. Most respect robots.txt. Blocking them cuts your content out of AI-generated answers. You can allow all, allow selectively, or block all — each has different consequences for your AI visibility.
Why AI crawlers matter for your site
Traditional web crawlers (Googlebot, Bingbot) index your pages for search rankings. AI crawlers do something different: they read your content to train LLMs or to power real-time AI answers on platforms like Perplexity, ChatGPT, and Claude.
If these crawlers are blocked or can't access your site, your content is invisible to AI answer engines — the same way a noindex tag makes a page invisible to Google. As AI answers capture more of the "zero-click" question market, this matters increasingly for traffic.
Use our free AI crawler checker to see which crawlers can currently access your site.
The complete AI crawler list
Each entry includes the user-agent string (what robots.txt uses to identify it), the owning company, and copy-paste robots.txt snippets for allowing or blocking.
| Crawler | Company | User-agent | Purpose |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Training data collection and ChatGPT browsing / real-time answers |
| ClaudeBot | Anthropic | ClaudeBot | Training data and real-time information retrieval for Claude |
| PerplexityBot | Perplexity AI | PerplexityBot | Real-time web search for Perplexity answer engine |
| Applebot-Extended | Apple | Applebot-Extended | Training data for Apple Intelligence (generative AI features) |
| GoogleOther-Extended | GoogleOther-Extended | Training data for Google AI products (Gemini, AI Overviews) | |
| Amazonbot | Amazon | Amazonbot | Training data for Amazon Alexa and AI products |
| Meta-ExternalAgent | Meta (Facebook) | Meta-ExternalAgent | Training data for Meta AI (Llama models) |
| Bytespider | ByteDance (TikTok) | Bytespider | Search indexing and AI training for ByteDance products |
Detailed breakdown
GPTBot (OpenAI)
OpenAI was the first major AI company to publish a named crawler. Blocking GPTBot prevents your content from appearing in ChatGPT answers and from being used in future training. Most sites with public content should allow it.
Introduced: August 2023
Official docs: https://platform.openai.com/docs/gptbot
To block in robots.txt:
User-agent: GPTBot
Disallow: /ClaudeBot (Anthropic)
Anthropic's web crawler for Claude's training corpus and browsing capabilities. Respects robots.txt. The main AI-search user-agent to allow if you want your content cited by Claude.
Introduced: Early 2024
Official docs: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web
To block in robots.txt:
User-agent: ClaudeBot
Disallow: /PerplexityBot (Perplexity AI)
Perplexity is the search engine most directly powered by web crawling for real-time answers. Blocking PerplexityBot means your content will not appear in Perplexity citations — the most direct consequence of any AI crawler block.
Introduced: 2023
Official docs: https://docs.perplexity.ai/docs/perplexity-bot
To block in robots.txt:
User-agent: PerplexityBot
Disallow: /Applebot-Extended (Apple)
Apple's extended crawler specifically for AI training (separate from the standard Applebot used for Siri and Spotlight). Block Applebot-Extended if you don't want your content used for Apple Intelligence training — blocking Applebot does NOT block the extended variant.
Introduced: Late 2023
Official docs: https://support.apple.com/en-gb/111900
To block in robots.txt:
User-agent: Applebot-Extended
Disallow: /GoogleOther-Extended (Google)
Google introduced separate user-agents for AI training. Blocking GoogleOther-Extended does NOT block Googlebot (your search rankings are unaffected). Blocking both GoogleOther and GoogleOther-Extended opts you out of Google AI training.
Introduced: 2024
Official docs: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
To block in robots.txt:
User-agent: GoogleOther-Extended
Disallow: /Amazonbot (Amazon)
Amazon's AI training crawler for Alexa and Amazon's LLM products. Generally lower priority for most website owners than the major search-AI crawlers.
Introduced: 2022
Official docs: https://developer.amazon.com/amazonbot
To block in robots.txt:
User-agent: Amazonbot
Disallow: /Meta-ExternalAgent (Meta (Facebook))
Meta's crawler for Llama model training. Respects robots.txt. Lower direct user-facing impact than the answer-engine crawlers (Perplexity, ChatGPT), but relevant if you have concerns about Meta AI training.
Introduced: 2024
Official docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
To block in robots.txt:
User-agent: Meta-ExternalAgent
Disallow: /Bytespider (ByteDance (TikTok))
ByteDance operates one of the largest web crawlers. Often blocked by default on many sites due to its aggressive crawling behavior. Blocking it has no impact on any major Western search engine rankings.
Introduced: 2022
To block in robots.txt:
User-agent: Bytespider
Disallow: /Should I block AI crawlers?
This is the biggest question in AI visibility right now. Here's our honest take:
- For most sites: allow all. Your content being cited by Perplexity or ChatGPT is traffic and brand exposure. Blocking reduces your AI visibility for minimal gain unless you have a specific content-protection reason.
- If you sell subscriptions or paywalled content: block selectively. Block crawlers from your premium content paths (e.g.
Disallow: /premium/) while allowing them on your free content. - If you have strong IP concerns: block training crawlers, allow search crawlers. Block GPTBot, ClaudeBot etc. (training) while ensuring PerplexityBot (search engine) can still access you. The two use-cases have different implications.
Note: blocking AI crawlers does NOT affect your Google Search rankings in any way. GPTBot, ClaudeBot and PerplexityBot are completely separate from Googlebot.
Allow all AI crawlers — copy-paste robots.txt
Add this to your robots.txt to explicitly allow all major AI crawlers:
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: GoogleOther-Extended Allow: /
Check which crawlers can access your site
Our free AI crawler checker tests whether your robots.txt is allowing or blocking each major AI crawler — and flags if you're accidentally blocking crawlers you want to allow, or allowing ones you want to block.
Check your site's AI crawler access
Paste your URL and see which AI crawlers can access your site right now — free, no signup.
Check AI crawler access →Keep reading