Robots.txt explained: what we check & why it matters

What is robots.txt?

Robots.txt is a plain text file that uses the Robots Exclusion Protocol — a standard since 1994 — to communicate with web crawlers. It's always located at the root of your domain:

# robots.txt for example.com
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /staging/

Sitemap: https://example.com/sitemap.xml

The file consists of rules: each User-agent line specifies which crawler the rules apply to (* means all), and Allow/Disallow lines specify which paths are permitted or blocked. The Sitemap directive points crawlers to your sitemap.

When a search engine bot visits your site, the very first thing it does is fetch /robots.txt. If the file exists and contains rules, the bot follows them. If the file is missing (404), the bot assumes everything is allowed.

Why robots.txt matters for SEO

Robots.txt directly controls what search engines can crawl:

• Crawl budget optimization: Large sites benefit from blocking paths that shouldn't be crawled — admin panels, internal search results, faceted navigation, staging paths. This focuses Googlebot's limited crawl budget on your most important pages.
• Preventing duplicate content crawling: If your CMS generates duplicate content at certain paths (e.g., tag archives, sorted views), robots.txt can prevent crawlers from discovering them.
• Sitemap discovery: The Sitemap: directive in robots.txt is one of three ways Google discovers your sitemap (along with GSC submission and sitemap links from other sitemaps).
• Preventing accidental de-indexing: A misconfigured robots.txt can block your entire site from being crawled. This is one of the most devastating technical SEO errors — and it's surprisingly common.

How our engine checks robots.txt

We run two robots.txt checks on every site:

Check 1: Robots.txt exists

ID: technical-robots-txt-exists
Severity: Warning
Scope: Site-wide
Logic: We fetch /robots.txt at the site's root domain. If the response is a 404 (or any non-200 status), the check fails. This check only runs for live URL crawls — it's skipped for pasted HTML analysis.

Why we flag it: While a missing robots.txt isn't catastrophic (crawlers assume everything is allowed), it means you're missing the opportunity to optimize crawl budget, point crawlers to your sitemap, and control AI crawler access. Every production site should have one.

Check 2: Robots.txt blocks entire site

ID: technical-robots-blocks-all
Severity: Critical
Scope: Site-wide
Logic: If a robots.txt exists, we parse it and check whether the root URL (/) is disallowed for our crawler's user agent. If the root is blocked, the check fails — meaning robots.txt is preventing crawlers from accessing the entire site.

Why it's critical: A robots.txt that blocks the root URL (Disallow: / for all user agents) prevents all search engine crawling. This is correct for staging or development sites, but devastating for a production site — Google will stop crawling all your pages, and they'll gradually disappear from search results. Common cause: a developer sets Disallow: / during staging and forgets to remove it before launch.

How to read your robots.txt result

When you run a URL through our free checker, each robots.txt check shows:

• Pass (green): Robots.txt exists and doesn't block the entire site. Crawlers can access your content.
• Warning (yellow) — missing: No robots.txt file found. Create one at your site root.
• Critical (red) — blocks all: Robots.txt exists but blocks the entire site. If this is a production site, fix immediately.

Robots.txt best practices

# Good robots.txt template
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /staging/
Disallow: /search?

# AI crawler control (optional)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Sitemap location
Sitemap: https://example.com/sitemap.xml

• Always include Sitemap: — it helps search engines discover your sitemap even if it's not submitted in GSC.
• Block admin paths: No reason for crawlers to access/admin/, /wp-admin/, or similar.
• Block internal search: /search? pages generate infinite URL combinations that waste crawl budget.
• Don't block CSS/JS: Google needs to render your page — blocking CSS and JavaScript files can prevent proper rendering and hurt rankings.
• Test before deploying: Use Google Search Console's robots.txt Tester to verify your rules before pushing to production.

How to fix robots.txt issues (by platform)

We have step-by-step fix guides for robots.txt on every major CMS:

• Fix robots.txt in WordPress →
• Fix robots.txt in Shopify →
• Fix robots.txt in Webflow →

Robots.txt and AI visibility

Robots.txt is the primary mechanism for controlling AI crawler access. Major AI crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and others — all respect robots.txt directives. You can selectively allow or block specific AI crawlers:

# Allow AI retrieval (for citations) but block training
User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Disallow: /

For a complete list of AI crawlers and how to control them, see our Complete list of AI crawlers guide. For testing your current robots.txt against AI crawlers, try our Robots.txt tester tool.

FAQ

What is robots.txt?

Robots.txt is a plain text file at the root of your website that tells web crawlers which URLs they can access. It's the first file search engine bots check when visiting your site.

Does robots.txt block pages from being indexed?

No — robots.txt blocks crawling, not indexing. If other sites link to a blocked page, Google may still index the URL with a "No information available" description. Use a noindex meta tag to prevent indexing.

What happens if my site has no robots.txt?

Crawlers assume they can access everything. This is fine for small sites, but you lose the ability to optimize crawl budget, point to your sitemap, and control AI crawler access.

Can robots.txt control AI crawlers?

Yes. AI crawlers like GPTBot, ClaudeBot, and PerplexityBot respect robots.txt directives. You can selectively allow or block specific AI crawlers.