SEO Analysis Checker.com

AI crawler checker

Is your site open to AI answer engines?

Search is splitting in two. Alongside the classic blue links, a growing share of people now get answers directly from ChatGPT, Claude, Perplexity and Google’s AI Overviews — and those systems can only cite what their crawlers are allowed to read. A single line in your robots.txt can quietly make your entire site invisible to an AI answer engine, and most site owners have no idea whether they are open or shut.

This checker fetches your live robots.txt and tests it against 14 major AI crawlers — GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and Meta-ExternalAgent — labelling each with its operator and whether it trains models, retrieves pages live, or builds an AI search index. It also checks whether you publish an llms.txt, the emerging standard for pointing AI systems at your best content, and gives you a plain-English verdict on whether your site is open to AI answer engines. This is the core of AI-visibility positioning: you cannot be cited by an AI you have accidentally locked out.

← All free toolsRun the full audit →
Enter a domain to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 10 more AI crawlers are allowed to read it — and whether it has an llms.txt.

How it works

Fast, transparent, no signup

  1. Enter your domain — the tool fetches your live robots.txt server-side.
  2. It tests 14 AI crawlers against it and checks for an llms.txt file.
  3. Read the per-bot allow/block table and the plain-English AI-visibility verdict.

FAQ

Questions people ask about this tool

Why does it matter if AI crawlers can access my site?

AI answer engines like ChatGPT Search, Perplexity, Claude and Google AI Overviews increasingly sit between users and websites. If their crawlers are blocked in your robots.txt, your content cannot be read, summarized or cited — so you become invisible in the fastest-growing discovery channel. This tool shows exactly which AI crawlers can reach you.

Should I allow or block AI crawlers?

It depends on your goals. If you want AI visibility and citations, allow the retrieval and search bots (ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-User). Some publishers block the training-only bots (GPTBot, CCBot, Google-Extended, Applebot-Extended) to keep content out of model training while staying visible in live answers. This tool distinguishes each bot and its purpose.

What is the difference between the bots you check?

Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, anthropic-ai) gather data to train models. Live-retrieval agents (ChatGPT-User, Claude-User, Perplexity-User) fetch pages in real time to answer a specific question. Search indexers (OAI-SearchBot, PerplexityBot, Amazonbot) build AI search indexes. Blocking one type does not block the others.

What does llms.txt have to do with this?

robots.txt controls whether AI crawlers may access your site; llms.txt (a newer, optional file) tells the ones that can access it which pages are most worth reading and citing. Allowing crawlers and publishing a good llms.txt together give you the best shot at accurate AI citations.