Is llms.txt a standard?

A proposal, from Answer.AI's Jeremy Howard in late 2024. No standards body has adopted it, and unlike robots.txt there is no agreement on what a crawler should do with it. That is why the honest answer to 'does it work' is 'it depends on which crawler'.

Who actually reads it?

Anthropic has said its crawlers use it and publishes its own; Perplexity has said it retrieves llms.txt to help choose pages. Google has said, on the record, that it does not and will not, comparing it to the keywords meta tag. OpenAI has not stated a position. Crawler-log studies in 2025–26 found requests for /llms.txt to be a negligible fraction of AI crawler traffic.

If you have ten minutes, yes, because it costs nothing and two of the crawlers that matter say they use it. But do it after the things the check flags as blockers — a firewall block or JavaScript-only content makes llms.txt irrelevant, because the crawler never gets that far.

What about llms-full.txt?

The same proposal suggests a companion file containing the full text of your key pages in one Markdown document. It is popular with documentation sites (Mintlify generates it) and pointless for most others — if your pages are readable HTML, a crawler can read them directly.

llms.txt — what it is, what it isn't, and a template

Anthropic and Perplexity say they read it, Google says it won't, OpenAI hasn't said. What that means for whether to bother, and a template that takes ten minutes.

This diagram is free to reuse with credit — all eighteen, as SVG or PNG.

What it is

A plain Markdown file at /llms.txt: a one-line description of the site, a short paragraph of context, and a list of the pages most worth reading, each with a one-line summary. The idea is that a language model with a limited context window gets a curated map instead of having to crawl and guess. The AI visibility check reports whether one exists, under “llms.txt”, and lists it as a refinement, never as a blocker.

What the evidence says

Three facts, each from the party in question. Anthropic supports it and publishes one for its own documentation. Perplexity has said it fetches the file to help prioritise pages. Google has said it does not use it and has no plans to. OpenAI has said nothing either way. Independent crawler logs put requests for the file at a rounding error of AI crawler traffic, and an analysis of the fifty most-cited domains in AI answers found one with the file.

Put together: it is cheap, two relevant crawlers say they read it, and nobody has shown it changes outcomes. That is the definition of a cheap bet. Make it, and spend your real effort on the blockers.

The template

# What To Fix First

> Free Core Web Vitals checker that leads with the one fix that recovers
> the most load time, plus an AI visibility check and mobile usability test.

Built for site owners who are not performance engineers. Every result
explains the free fix before mentioning any product.

## Tools

- [Core Web Vitals check](https://whattofixfirst.com/): one fix first, then the numbers
- [AI visibility check](https://whattofixfirst.com/ai-visibility): can AI crawlers read a page?
- [Mobile usability test](https://whattofixfirst.com/mobile-usability-test): the retired Search Console checks

## Guides

- [How to fix a slow LCP](https://whattofixfirst.com/guide/largest-contentful-paint): find the phase the time sits in
- [Which AI crawlers to allow](https://whattofixfirst.com/guide/ai-crawlers-robots-txt): fetchers vs training bots

## Optional

- [About](https://whattofixfirst.com/about): who runs it and how it makes money

The structure is the proposal’s: an H1 with the site name, a blockquote summary, free paragraphs, then H2 sections of links with one-line descriptions. An ## Optional section marks pages a model may skip when short on context. Serve it as text/plain or text/markdown at the site root.

Where to put it

Static sites and frameworks — drop the file in public/.
WordPress — Yoast and Rank Math both generate one from your content; or upload the file to the web root by FTP.
Shopify — no root file access; the nearest equivalent is a page at /pages/llms-txt, which defeats the purpose. Skip it.
Documentation on Mintlify, GitBook or Docusaurus — built in or a plugin away; this is where the file is most used.

Check it worked

curl -s https://example.com/llms.txt | head should print the Markdown, not an HTML 404 page. Re-run the check and the llms.txt row reads Present.

Common questions

Is llms.txt a standard?: A proposal, from Answer.AI's Jeremy Howard in late 2024. No standards body has adopted it, and unlike robots.txt there is no agreement on what a crawler should do with it. That is why the honest answer to 'does it work' is 'it depends on which crawler'.
Who actually reads it?: Anthropic has said its crawlers use it and publishes its own; Perplexity has said it retrieves llms.txt to help choose pages. Google has said, on the record, that it does not and will not, comparing it to the keywords meta tag. OpenAI has not stated a position. Crawler-log studies in 2025–26 found requests for /llms.txt to be a negligible fraction of AI crawler traffic.
So should I bother?: If you have ten minutes, yes, because it costs nothing and two of the crawlers that matter say they use it. But do it after the things the check flags as blockers — a firewall block or JavaScript-only content makes llms.txt irrelevant, because the crawler never gets that far.
What about llms-full.txt?: The same proposal suggests a companion file containing the full text of your key pages in one Markdown document. It is popular with documentation sites (Mintlify generates it) and pointless for most others — if your pages are readable HTML, a crawler can read them directly.

Check a page against this

Enter any public URL. You’ll get the one thing stopping AI crawlers from reading the page (if anything is), then a crawler-by-crawler table and what the HTML actually tells a machine.

Other guides

How to fix a slow LCP — Your main image or headline takes too long to appear.
How to fix layout shift — Content jumps around while the page loads.
How to fix slow INP — Taps and clicks take a moment to do anything.
How to fix a slow server response — Your server is slow to start replying at all.
How to eliminate render-blocking CSS and JS — CSS and JS files stop anything appearing on screen.
Why you're seeing "no field data" — Google has no real-visitor data for your site yet.
How to fix oversized and outdated images — Your images weigh far more than they need to.
How to cut unused JavaScript — You ship JavaScript this page never runs.
How to fix your cache headers — Returning visitors re-download files they already have.
Why your score is different on every run — The score is different every time you test.
Fast on desktop, slow on mobile — Desktop looks fine; the phone test says otherwise.
Why Search Console still shows the old numbers — You fixed it weeks ago and Search Console hasn't noticed.
How to stop plugins loading on every page — A form plugin's CSS on pages with no form — times every plugin.
Which AI crawlers to allow in robots.txt — You block training bots and accidentally block the ones that cite you.
Your firewall is blocking AI crawlers — robots.txt allows everyone, yet AI crawlers get a 403 before they read a byte.
Your content only exists after JavaScript runs — The server sends an empty shell and a script fills it in — AI crawlers see nothing.
Structured data that AI crawlers actually use — The page is readable but nothing tells a machine what it is.