How it works
Fast, transparent, no signup
- Paste your robots.txt into the editor.
- Enter a path (or full URL) and choose a user-agent to test.
- Read the verdict, the deciding rule, the AI-crawler summary and the declared sitemaps.
Robots.txt tester
Free robots.txt tester
A single misplaced Disallow line in robots.txt can hide an entire section of your site from Google — or leave a private area wide open. Because the matching rules are surprisingly subtle (longest-match wins, Allow beats Disallow on ties, wildcards behave in specific ways), it is easy to misread what a file actually does. This tester removes the guesswork: paste your robots.txt, enter a path, pick a crawler, and it tells you exactly whether that URL is allowed or blocked and which line decided it.
It goes further than a standard robots checker in two ways. First, it evaluates a panel of AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more — against the same path, so you can see whether answer engines can read your content, which increasingly determines whether you get cited in AI results. Second, it lists every Sitemap: directive it finds, so you can confirm your sitemap is actually declared. Everything is parsed locally in your browser using Google’s documented precedence rules, so it is safe to test staging or draft rules before you ship them.
Tester
Paste robots.txt and test a path
Rules are evaluated by the most specific matching user-agent group and the longest Allow/Disallow pattern — the way Google does it.
Sitemaps declared
- https://example.com/sitemap.xml
FAQ
Questions people ask about this tool
How does Google decide allowed vs blocked?
Google selects the group whose user-agent token most specifically matches the crawler, then applies the most specific rule — the longest matching Allow or Disallow path. When an Allow and a Disallow are equally specific, Allow wins. This tester implements that same precedence, including * and $ wildcards.
Does robots.txt block indexing?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in search results without a snippet if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Why does the tool test AI crawlers separately?
Blocking Googlebot and blocking GPTBot are independent decisions. Many sites unintentionally block AI crawlers (or fail to) because those user-agents were added later. The AI-bot summary shows, at a glance, which answer engines can read the path you are testing.
Is my robots.txt uploaded anywhere?
No. This tester parses everything in your browser. Nothing you paste is sent to a server, so you can safely test staging or unpublished rules.