llms.txt Explained: What It Does and Who Actually Reads It

Jeremy Howard of Answer.AI published the proposal at llmstxt.org on September 3, 2024. The problem it set out to solve is narrow and worth quoting, because it's the sentence everyone skips: the file exists to "provide information to help LLMs use a website at inference time," since "context windows are too small to handle most websites in their entirety."

Inference time. Not training time, not crawl time. The scenario is a model already pointed at your site, in a session, with a few thousand tokens of room and no idea which of your 400 URLs matter. llms.txt hands it a shortlist in clean markdown instead of making it guess from your navigation.

So the file is a table of contents for a reader who has already arrived. Most advice about it treats the file as a way to make readers arrive, which is where the topic goes sideways.

The format is deliberately small. Only one element is mandatory.

That "Optional" section is the one genuinely clever piece of design here. It gives a client a documented way to degrade gracefully: take everything if there's room, drop the Optional block if there isn't. Nothing in robots.txt or a sitemap expresses priority that way.

A minimal, spec-shaped file looks like this:

# SparkCliks

> Search CTR and website traffic services. This file lists the pages that
> describe each product and how its results are measured.

## Products

- [SERP Clicks](https://www.sparkcliks.com/): what the SERP Clicks product delivers
- [Sparky Traffic Bot](https://www.sparkcliks.com/sparky-traffic-bot/): browser automation priced per concurrent session

## Optional

- [Blog](https://www.sparkcliks.com/blog/): background reading on click behavior and AI search

The file lives at /llms.txt. Note where it does not live: /.well-known/, the registry-backed location real web standards use. That placement is a fair summary of the proposal's status.

Five things get attributed to this file that it has no mechanism to perform.

• It does not block AI training. There is no allow or disallow grammar in the spec. Every line is a link or a description. A file made entirely of suggestions cannot forbid anything.
• It does not control crawling. That's robots.txt, and the distinction between refusing a fetch and refusing storage is its own trap, covered in robots.txt vs noindex.
• It does not submit you anywhere. No provider operates an llms.txt endpoint, queue or registry. Publishing one notifies nobody.
• It does not replace a sitemap. A sitemap enumerates every URL for machine discovery. llms.txt curates a handful for a model that is reading right now. Different consumer, different job.
• It does not influence rankings. Not in classic results, not in AI answers.

The three files get grouped together because they all sit at the root and all address machines. The similarity ends there.

The last row is the one that decides whether the file is worth your time. robots.txt and sitemap.xml are pulled on a schedule by infrastructure that has been fetching them for two decades. llms.txt is pulled only when something deliberately goes and gets it.

Google's own documentation on AI features in Search is unambiguous: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." That's from the AI features guidance in Search Central, and it covers AI Overviews and AI Mode.

John Mueller went further in an April 2025 Reddit comment, reported by Search Engine Journal: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." The same discussion drew a comparison to the keywords meta tag.

That comparison is memorable and slightly unfair. The keywords meta tag died because it was a self-declared ranking input, so it got stuffed into uselessness. llms.txt was never proposed as a ranking input, so it can't fail that way. Its problem isn't manipulation, it's that the clients it was written for mostly never shipped support. Different failures, different odds of reversing, and conflating them makes people dismiss the one real use case the file has.

The consumers are real but they're pull-based, not crawl-based:

• Coding agents and IDE assistants, when a developer points them at your documentation URL. This is the dominant genuine use.
• Documentation platforms that generate the file automatically. Hosted docs providers rolled llms.txt out across their customer bases in late 2024, which is why a long list of AI companies appear to have adopted it. Most of them adopted a default.
• Custom retrieval pipelines somebody built on purpose, pointed at a specific set of domains.
• Agents handed your domain as part of a task, which fetch the file as a cheap first look before deciding what else to load.

One distinction cuts through most of the confusion in this topic: publishing an llms.txt is not evidence of consuming one. Several well-known AI companies publish the file. That tells you their documentation is meant to be loaded into coding agents. It tells you nothing about whether their crawlers read yours.

So llms.txt is pull, not push. Nothing fetches it on a schedule, so it can't be a discovery channel. It improves an interaction with a visitor you already earned: a real benefit, a modest one, and a completely different purchase from the one most articles are selling. Which sources an assistant reaches for in the first place gets decided upstream, in the retrieval process covered in how AI assistants pick sources.

Two companion ideas travel with the proposal.

llms-full.txt is the whole documentation set concatenated into one markdown file. It isn't in the original spec, which describes llms-ctx.txt and llms-ctx-full.txt produced by a generator tool, but the llms-full.txt naming won by adoption. The trap is size: these files routinely run to megabytes, and a file too large for the context window fails the exact constraint that motivated the proposal. If yours can't fit alongside the user's actual question, it's a download, not a context aid.

The .md convention asks you to serve a clean markdown version of any page at the same URL with .md appended. This is the more useful half of the proposal and gets a fraction of the attention. It needs no curation, stays current automatically because it renders from the same source, and it's what an agent wants the moment it follows one of your links. Retrieval works on passages rather than whole documents anyway, which is why chunk-level retrieval rewards clean per-page markdown over a curated index.

Stop reading other people's adoption studies and measure your own domain. If you have a raw access log in combined format, one command answers it:

grep -E 'GET /llms(-full)?\.txt' access.log \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn

Field 6 under -F'"' is the user agent in combined log format. You get a frequency table of every client that has ever requested the file. On Cloudflare, the same answer comes from a Logs or Analytics filter on the request path, grouped by user agent.

How to read the output. Excluding your own uptime monitor and any generic HTTP library that shows up once, count distinct requesting agents over a 30 day window, measured only after the file has been live at least 30 days. Then, and this is the part that makes the number mean something, run the identical command against /robots.txt over the same window:

grep -E 'GET /robots\.txt' access.log \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn

robots.txt is fetched constantly by every crawler that touches your site, so it's a live, site-specific baseline for what real automated attention looks like. The ratio between the two counts is the honest measure of how much of the machine web reads your llms.txt. No benchmark figure applies here. Your own two numbers, same log, same window, settle it.

Reading the outcome. Only your own monitoring in the llms.txt column means nothing in the wild uses the file, and maintaining it is currently unpaid work. Recognizable coding agent or assistant user agents mean developers are loading your docs deliberately, which is the file working as designed. Re-run quarterly, since the answer changes if clients ship support.

Run this checklist against your own site.

The failure mode nobody mentions: a stale llms.txt is worse than none. A stale robots.txt costs you some crawl efficiency. A stale llms.txt means the one time an agent fetches it, it gets dead links and outdated descriptions carrying the implied authority of a file you chose to publish, and that lands in somebody's answer. Maintenance cost is the whole decision. Automated, the file is close to free. Manual, be honest about whether you'll update it in eight months.

If what you actually wanted was control over AI usage of your content, that's a different file. Use robots.txt for crawler access today, and watch the IETF AI Preferences working group, which is drafting a Content-Usage vocabulary delivered through an HTTP header or a robots.txt rule. Those drafts are still in progress and not yet a published standard, but that's the standards track for the job people keep assigning to llms.txt.