SEO Analysis Checker.com

Fix guide · Webflow

How to Fix robots.txt Issues in Webflow (2026)

Webflow automatically generates a robots.txt file for every published site. If our checker flagged "missing robots.txt", the most likely cause is that your site isn't published yet (only staged), your custom domain isn't resolving correctly, or a CDN proxy is intercepting the request. If you need to customize the robots.txt content, Webflow provides a field in Project Settings.

Free · no email · no signup · results stream in ~10 seconds

Run a free check to confirm this is affecting your site

Why it matters

What this issue costs you

robots.txt tells search engines and AI crawlers which pages they can access. If it's missing or misconfigured, crawlers may index pages you don't want indexed, or — worse — they may not crawl your site at all. Webflow's default allows all crawlers, which is correct for most sites.

Step-by-step

How to fix: Missing robots.txt

Webflow

  1. 1

    Verify your robots.txt exists

    Visit yourdomain.com/robots.txt in your browser. If you see a robots.txt file with "User-agent: *" and "Sitemap:" lines, it's working. If you get a 404: check that your site is published (not just staged) and that your custom domain is properly connected in Project Settings > Hosting.

  2. 2

    Check staging vs. production

    Webflow serves different robots.txt on staging (yoursite.webflow.io) vs. production (yourdomain.com). Staging sites include "Disallow: /" to prevent indexing of the webflow.io domain. If your custom domain still shows staging robots.txt, your domain isn't properly connected — verify DNS and SSL in Project Settings > Hosting.

  3. 3

    Customize robots.txt content

    Go to Project Settings > SEO > Robots.txt. Webflow provides a text field where you can add custom directives. Add rules to allow AI crawlers explicitly:

    Add to Project Settings > SEO > Robots.txt
    User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /
  4. 4

    Ensure sitemap is referenced

    Webflow automatically includes a Sitemap line in robots.txt pointing to your sitemap.xml. If it's missing, add "Sitemap: https://yourdomain.com/sitemap.xml" in the custom robots.txt field. This helps crawlers discover all your pages.

Verify the fix

Run the checker to confirm the issue is resolved

After applying the steps above, paste your URL below. Our checker will re-run every applicable check and show whether missing robots.txt still shows as a failing check.

Free · no email · no signup · results stream in ~10 seconds

FAQ

Common questions about missing robots.txt

Can I fully customize robots.txt in Webflow?

Partially. Webflow generates default rules (User-agent: * Allow: /) and you can add additional directives via Project Settings > SEO. You cannot override the default rules, only append to them. For most sites, the defaults plus AI crawler rules are sufficient.

Why does my Webflow staging site show "Disallow: /"?

This is intentional. Webflow blocks all crawlers on the .webflow.io staging subdomain to prevent Google from indexing your work-in-progress site. Once you connect a custom domain and publish, the production robots.txt allows all crawlers.

Does Webflow block AI crawlers by default?

No. Webflow's default robots.txt uses "User-agent: * Allow: /", which permits all crawlers including AI bots. However, explicitly naming AI crawlers with Allow rules makes your intent unambiguous and future-proofs against potential default changes.

While you're here

Related fix guides for Webflow

Missing robots.txtWordPress→Missing robots.txtShopify→Missing XML sitemapWebflow→

Need a full audit?

This checker spots individual issues instantly. deepseoanalysis.com crawls your whole site and tracks changes weekly, from $24/mo.

Deep audit →

See all WordPress fix guides, or run a free check now.