AI visibility · GEO
What is llms.txt? The complete guide for website owners (2026)
TL;DR: llms.txt is a plain-text file at the root of your domain that tells AI models what your site is about and which pages to prioritize. It's not a ranking signal for Google — it's a content-discovery signal for AI answer engines. We parsed every real llms.txt among the sites run through our checker (39 files, October 2026): almost all of them get the shape right. The job, describing each link so a model knows why to read it, is done by 22 of 39, but 17 of those files are one shared template. Among the other 22, only 5 describe every link. Outside that template, the bar to beat is low.
What is llms.txt?
llms.txt is a proposed open standard — a plain-text file placed at the root of a website (e.g. https://yoursite.com/llms.txt) that provides a structured, human-readable summary of your site's content for Large Language Models (LLMs).
Think of it as robots.txt for the AI era. robots.txt tells traditional web crawlers what they can and cannot access. llms.txt tells AI models what your site is, what it does, and which pages are most important to read when generating answers about your topic area.
The specification was proposed by Jeremy Howard (fast.ai) in September 2024. It is a proposal, not a ratified standard, and no major AI vendor has published a commitment to read it — treat anyone who tells you ChatGPT or Claude definitely parses your llms.txt as guessing. What is verifiable is that sites are shipping them: in our own audit sample, 39 of 52 sites serve a real one. The measured section below is about what those files contain.
What does an llms.txt file look like?
The format is Markdown — human-readable, simple, and parseable by any LLM. A minimal llms.txt has three parts:
- A
# Site Nameheading with a brief description of what the site does - An optional
## Aboutsection with more context - One or more
## Sectionheaders with bullet-point links to key pages
Here's a real example for an SEO tool:
# SEO Analysis Checker > Free, instant, ungated SEO and AI-visibility checker. Paste a URL, get > a full report in seconds — no signup required. ## Tools - [llms.txt Generator](https://seoanalysischecker.com/tools/llms-txt-generator): Build a valid llms.txt from a URL, no signup - [AI Crawler Checker](https://seoanalysischecker.com/tools/ai-crawler-checker): Test whether GPTBot, ClaudeBot and PerplexityBot are allowed by your robots.txt - [Meta Tag Analyzer](https://seoanalysischecker.com/tools/meta-tag-analyzer): Inspect title, description and Open Graph tags on any page ## Learn - [AI Crawler List](https://seoanalysischecker.com/learn/ai-seo/ai-crawler-list): Every AI user-agent, what it does, and how to allow or block it ## Optional - [Privacy policy](https://seoanalysischecker.com/privacy): How the checker handles the URLs you submit
We parsed 39 real llms.txt files. Here's what's actually in them
Every site run through our free checker gets /llms.txt probed. In October 2026 we went back and fetched the file itself for all 52 third-party sites audited to date (our own domains excluded), kept the 39 that serve a real one, and tested each file against the clauses of the spec — mechanically, so the numbers are reproducible rather than impressionistic. It is the same pipeline behind our robots.txt survey of AI crawlers, pointed at a different file.
The shape is easy and almost everyone gets it. 39 of 39 open with an # heading, 38 use at least one ## section, and 18 include the > one-paragraph summary. 18 of 39 have all three.
The job is harder, and outside one template few files do it. The point of the format is not the headings — it is the line after each link that tells a model what it will find there. 22 of 39 files describe every link they list, which looks healthy until you see where they come from. 17 of the 39 share one section template, the same ## headings in the same order on different domains, and all 17 of them describe their links. Among the 22 files that don't use it, only 5 of 22 do. The other 17 files in the sample leave a model guessing from a URL slug, which is exactly the situation llms.txt exists to remove:
- 14 of 39 have at least one bullet that is a naked URL — no link text, no description.
- 1 of 39 contains no links at all: a prose pitch for the company at a URL where a table of contents should be.
- 4 of 39 use relative paths (
/pricing). Nothing in the spec forbids it, but the file gets read out of context — absolute URLs remove the ambiguity for free. - 3 of 39 use the spec's
## Optionalsection, the one explicit way to say "skip this if you're short on context". It is the most-ignored part of the format.
Size is the other failure mode. The median file is 2.9 KB with 12 links — about right for a table of contents. But 1 of 39 is over 20 KB and the largest is 44 KB, which is a site dumping its content into the file instead of pointing at it. A file that large competes for the same context window as the page the model actually wanted, and gets truncated with no say over which half survives.
One thing that is not broken: the links mostly work. We resolved up to 4 links per file — 143 URLs — and only 1 was dead. Link rot is the problem people expect here and it is not the one they have.
What to copy, in one line: an H1, a > summary, ## sections, and then — the part almost everyone skips — - [Page name](https://absolute/url): what this page answers for every single link. Keep it under a few KB. That puts you ahead of 17 of the 39 files we measured.
Read the sample honestly: 52 sites is small and self-selected — people who run an AI-visibility checker skew toward small SaaS and AI-adjacent tools. That bias is why we publish nothing here about how many sites on the web have an llms.txt; that number would describe our traffic, not the web. It distorts the format findings too. Between 2 and 4 October 2026, on the same sites, the files that describe every link went from 10 of 37 to 28 of 38, and 23 of those 28 used the shared template. On 5 October it was 21 of 38: the template had shrunk to 16 files, and six sites from the same family of AI-tool microsites now share a second template that leaves its links undescribed. On 7 October it was 22 of 39, with the first template at 17 files. A count that swings this much in three days, driven by one family of sites, says little about how people write the file. That is why we report it with and without the template.
Method: ops/llms-txt-structure.mjs fetches /llms.txt per host, keeps only responses with a non-HTML text/* content type, and tests each spec clause with a regex — no judgement calls. Numbers are imported into this page from a committed snapshot, not retyped. Last measured October 2026.
Why does llms.txt matter for your website?
AI answer engines (Perplexity, ChatGPT with browsing, Claude, Google AI Overviews) are increasingly the first place people go to answer questions that used to send traffic to your site. These systems don't rank pages like Google does — they read content and synthesize answers, citing sources they find credible and relevant.
llms.txt gives you a way to influence that process:
- Content inventory: LLMs reading your llms.txt understand which pages exist and what they cover, even if they haven't crawled every URL yet.
- Topic authority: A clear, structured description of what your site covers reinforces your topical expertise in the model's context window.
- Citation preference: When a model has indexed your llms.txt, it knows which of your pages to cite for which queries — improving the chance your site is the cited source rather than a competitor.
Does Google use llms.txt?
As of July 2026, Google has not announced support for llms.txt in its traditional web indexing pipeline. Google's AI Overviews system uses its existing crawl infrastructure and does not specifically parse llms.txt.
Nor has any other vendor committed publicly to using it. The honest position: the file is cheap to publish, it is trivially readable by any model that is handed it, and nobody can currently show you a log line proving a named answer engine fetched it and acted on it. Publish one because it costs ten minutes, not because someone promised you traffic for it.
Setting up llms.txt now costs a few minutes and positions you for broader adoption as the standard matures — similar to how early schema.org adopters benefited from rich results before they were widely understood.
How to create an llms.txt file
There are three ways to create your llms.txt:
Option 1: Use our free generator (fastest)
Our free llms.txt generator takes your site URL and key pages, then produces a correctly formatted llms.txt you can download and upload to your root directory. Takes under a minute.
Generate your llms.txt now
Paste your URL and we'll create a ready-to-upload llms.txt in seconds — free, no signup required.
Generate llms.txt →Option 2: Write it manually
Create a file called llms.txt in a text editor, following the format shown above. Upload it to the root of your server (same level asrobots.txt). Verify it's accessible at https://yoursite.com/llms.txt.
Option 3: WordPress plugin
Several WordPress plugins (including some SEO plugins) now support auto-generating llms.txt. Search the WordPress plugin directory for "llms.txt". Alternatively, you can place a static llms.txt in your theme's root folder or WordPress install root.
What should I include in llms.txt?
Keep it concise and focused on pages that matter:
- Your most important product or service pages
- Your key documentation or how-to guides
- Your about/team page (establishes brand identity for LLMs)
- Your blog or learn section with a description of topics covered
Do not include: admin pages, checkout flows, search result pages, low-quality or thin content. Think of it as a curated table of contents — the pages you'd want an AI to read to understand your site.
How do I verify my llms.txt is working?
Check the content type, not just the status code — this is the one check most people and most tools get wrong. Of the 52 sites we probed, 1 answered /llms.txt with HTTP 200 and an HTML page: a framework catch-all route serving the app shell for a path that doesn't exist. Status code 200, nothing usable behind it. Anything that tests only for "did it 200" calls that a pass — including our own checker's llms.txt check today, which is written up with the rest of our known limitations.
Thirty seconds in a terminal settles it:
curl -sI https://yoursite.com/llms.txt | head -n 3 # You want: # HTTP/2 200 # content-type: text/plain; charset=utf-8 (or text/markdown) # # You do NOT want: # content-type: text/html <- a catch-all route, not your file
For reference, of the 39 real files in our sample 35 are served as text/plain and 4 as text/markdown. Either is fine.text/html means you are looking at a page, not a file.
Then run our free SEO checker on your homepage for the rest of the picture — llms.txt sits alongside AI crawler access, schema and answer structure in the AI-visibility score, and a file on its own moves less than people expect.
llms.txt vs robots.txt — what's the difference?
- robots.txt: Tells web crawlers (Googlebot, Bingbot, etc.) which URLs they are allowed or disallowed from crawling. A technical access-control document.
- llms.txt: Tells AI models what your site is about and which pages are most relevant. A content-description document. It does not control crawler access — use robots.txt
User-agent: GPTBot Disallow: /to block AI crawlers if that's your goal.
You need both. robots.txt for crawler access control; llms.txt for AI content discovery.
Frequently asked questions
Does llms.txt hurt my Google SEO?
No — Google does not currently use llms.txt in its ranking systems. It's a separate file from robots.txt and sitemap.xml. Having one has no negative effect on your search rankings.
What happens if I don't have an llms.txt?
AI models will still crawl and index your site — they don't require llms.txt. The difference is that without it, they have to infer what your site is about from your content alone, which may be less accurate, especially for new or small sites with limited external references.
What do real llms.txt files look like in practice?
We fetched and parsed the llms.txt of every third-party site audited through this checker — 39 real files as of October 2026. All 39 open with an H1 and 38 use at least one ## section, but only 22 of 39 give every link the one-line description the format exists for, 1 contain no links at all, and 1 exceed 20 KB (largest 44 KB) against a median of 2.9 KB.
How do I check my llms.txt is really being served?
Look at the content type, not just the status code. Of 52 sites we probed, 1 answered /llms.txt with HTTP 200 and an HTML page from a catch-all route — present to any status-code-only checker, useless to any model. Run curl -sI https://yoursite.com/llms.txt and require a 200 with a text/plain or text/markdown content type.
How often should I update my llms.txt?
Update it whenever your site's key page structure changes significantly — new product, major new content section, rebrand. You don't need to update it for every blog post; think of it as a quarterly maintenance task.
Keep reading