What is llms.txt? The complete guide for website owners (2026)

What is llms.txt?

llms.txt is a proposed open standard — a plain-text file placed at the root of a website (e.g. https://yoursite.com/llms.txt) that provides a structured, human-readable summary of your site's content for Large Language Models (LLMs).

Think of it as robots.txt for the AI era. robots.txt tells traditional web crawlers what they can and cannot access. llms.txt tells AI models what your site is, what it does, and which pages are most important to read when generating answers about your topic area.

The specification was proposed by Jeremy Howard (fast.ai) in September 2024 and has since been adopted by thousands of sites including major developer documentation platforms, SaaS products, and publishers. Anthropic's Claude, Perplexity, and various AI search tools have begun parsing llms.txt as part of their content discovery pipelines.

What does an llms.txt file look like?

The format is Markdown — human-readable, simple, and parseable by any LLM. A minimal llms.txt has three parts:

1. A # Site Name heading with a brief description of what the site does
2. An optional ## About section with more context
3. One or more ## Section headers with bullet-point links to key pages

Here's a real example for an SEO tool:

# SEO Analysis Checker

> Free, instant, ungated SEO and AI-visibility checker. Paste a URL, get
> a full report in seconds — no signup required.

## Tools
- [llms.txt Generator](/tools/llms-txt-generator): Create your llms.txt file
- [AI Crawler Checker](/tools/ai-crawler-checker): See which AI bots can access your site
- [Meta Tag Analyzer](/tools/meta-tag-analyzer): Inspect title, description, Open Graph tags

## Learn
- [AI Crawler List](/learn/ai-seo/ai-crawler-list): GPTBot, ClaudeBot, PerplexityBot and more
- [What is GEO?](/learn/ai-seo/aeo-vs-geo-vs-seo): AI SEO explained

Why does llms.txt matter for your website?

AI answer engines (Perplexity, ChatGPT with browsing, Claude, Google AI Overviews) are increasingly the first place people go to answer questions that used to send traffic to your site. These systems don't rank pages like Google does — they read content and synthesize answers, citing sources they find credible and relevant.

llms.txt gives you a way to influence that process:

• Content inventory: LLMs reading your llms.txt understand which pages exist and what they cover, even if they haven't crawled every URL yet.
• Topic authority: A clear, structured description of what your site covers reinforces your topical expertise in the model's context window.
• Citation preference: When a model has indexed your llms.txt, it knows which of your pages to cite for which queries — improving the chance your site is the cited source rather than a competitor.

Does Google use llms.txt?

As of July 2026, Google has not announced support for llms.txt in its traditional web indexing pipeline. Google's AI Overviews system uses its existing crawl infrastructure and does not specifically parse llms.txt.

However, Google is testing various structured-content signals for its AI systems, and the ecosystem is evolving quickly. The current adopters of llms.txt are primarily non-Google AI answer engines: Perplexity (which has stated support), Anthropic's Claude for reading company documentation, and several AI search startups.

Setting up llms.txt now costs a few minutes and positions you for broader adoption as the standard matures — similar to how early schema.org adopters benefited from rich results before they were widely understood.

How to create an llms.txt file

There are three ways to create your llms.txt:

Option 1: Use our free generator (fastest)

Our free llms.txt generator takes your site URL and key pages, then produces a correctly formatted llms.txt you can download and upload to your root directory. Takes under a minute.

Option 2: Write it manually

Create a file called llms.txt in a text editor, following the format shown above. Upload it to the root of your server (same level asrobots.txt). Verify it's accessible at https://yoursite.com/llms.txt.

Option 3: WordPress plugin

Several WordPress plugins (including some SEO plugins) now support auto-generating llms.txt. Search the WordPress plugin directory for "llms.txt". Alternatively, you can place a static llms.txt in your theme's root folder or WordPress install root.

What should I include in llms.txt?

Keep it concise and focused on pages that matter:

• Your most important product or service pages
• Your key documentation or how-to guides
• Your about/team page (establishes brand identity for LLMs)
• Your blog or learn section with a description of topics covered

Do not include: admin pages, checkout flows, search result pages, low-quality or thin content. Think of it as a curated table of contents — the pages you'd want an AI to read to understand your site.

How do I verify my llms.txt is working?

After uploading, run our free SEO checker on your homepage. One of the checks specifically tests for the presence and accessibility of your llms.txt file. If it fails, the report will explain why (wrong path, server error, blocked by robots.txt, etc.).

You can also check directly: visit https://yoursite.com/llms.txt in your browser. You should see plain Markdown text. If you get a 404 or see HTML, the file isn't in the right place.

llms.txt vs robots.txt — what's the difference?

• robots.txt: Tells web crawlers (Googlebot, Bingbot, etc.) which URLs they are allowed or disallowed from crawling. A technical access-control document.
• llms.txt: Tells AI models what your site is about and which pages are most relevant. A content-description document. It does not control crawler access — use robots.txt User-agent: GPTBot Disallow: / to block AI crawlers if that's your goal.

You need both. robots.txt for crawler access control; llms.txt for AI content discovery.

Frequently asked questions