Fix guide · WordPress

How to Fix a Missing robots.txt in WordPress (2026)

robots.txt is a plain-text file that tells crawlers which parts of your site to access or avoid. A missing robots.txt means bots have no guidance file — they'll still crawl your site, but you lose the ability to direct their behavior. For WordPress sites, a missing robots.txt often means the file exists but is blocked by file permissions, or the virtual robots.txt isn't being generated.

Run a free check to confirm this is affecting your site

Why it matters

What this issue costs you

robots.txt lets you protect non-public areas of WordPress (wp-admin, certain wp-content directories) and point bots to your sitemap. While Google will crawl without it, a well-configured robots.txt is part of baseline technical SEO hygiene.

Step-by-step

How to fix: Missing robots.txt

WordPress

1
Option A: Use Yoast SEO (recommended)
Install Yoast SEO if you haven't. Go to Yoast SEO → Tools → File editor. If a robots.txt file doesn't exist in your root directory, Yoast will offer to create one. Add or edit your robots.txt content here. Yoast automatically adds your sitemap URL to the robots.txt file.

Option B: WordPress virtual robots.txt

WordPress generates a virtual robots.txt at /robots.txt if no physical file exists in the root. You can customise it by filtering the output in functions.php:

functions.php

add_filter( 'robots_txt', function( $output, $public ) {
  $sitemap = get_sitemap_url( 'index' );
  if ( $sitemap ) {
    $output .= "\nSitemap: " . esc_url( $sitemap ) . "\n";
  }
  return $output;
}, 10, 2 );

3
Option C: Create a static robots.txt file
Upload a robots.txt file to the root of your WordPress installation (same level as wp-config.php). A minimal, sensible robots.txt for most WordPress sites:
robots.txt
```
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://yoursite.com/sitemap_index.xml
```
4
Verify the file is accessible
Visit https://yoursite.com/robots.txt in your browser. You should see plain text. If you see a 404 or are redirected, check that no .htaccess rule or server config is blocking /robots.txt.

Verify the fix

Run the checker to confirm the issue is resolved

After applying the steps above, paste your URL below. Our checker will re-run every applicable check and show whether missing robots.txt still shows as a failing check.

FAQ

Common questions about missing robots.txt

Does a missing robots.txt harm my SEO?

It's a risk rather than a direct penalty. Without robots.txt, search engines may crawl admin or staging areas, waste crawl budget on low-value pages, or not find your sitemap URL. It's a quick fix with clear upside.

Should I disallow /wp-content/ in robots.txt?

No — that's a common mistake. /wp-content/ contains your theme files, plugins and media. Blocking it hides your images and static assets from Google, which can hurt image search rankings and prevent Googlebot from loading your page correctly.

My WordPress site has a robots.txt but the checker says it's missing — why?

Check for a trailing slash redirect loop, a password-protected staging environment, or a caching plugin serving a 404 for /robots.txt. Also verify the file is in the true root of your domain, not in a subdirectory.

While you're here

Related fix guides for WordPress

Missing XML sitemapWordPress→www / non-www inconsistencyWordPress→Missing canonical tagWordPress→

Need a full audit?

This checker spots individual issues instantly. deepseoanalysis.com crawls your whole site and tracks changes weekly, from $24/mo.

Deep audit →

See all WordPress fix guides, or run a free check now.