What llms.txt Is, and Whether It Actually Does Anything

The file itself is unglamorous. It sits at the root of your domain, at /llms.txt, and it's plain markdown. An H1 with the name of the site, a short summary in a blockquote, then a few H2 headings with lists of links underneath, each one optionally followed by a note explaining what's there. That's the whole specification. You could write one by hand in twenty minutes.

Jeremy Howard published the proposal in September 2024, and the idea behind it is easy to like. A language model working with a limited context window shouldn't have to crawl and parse your entire site to find the six pages that matter. Hand it a curated index instead. Every part of that reasoning is sound.

What's happened since is a different question, and it's the one worth spending time on.

What the file actually contains

The specification is deliberately small. Only the H1 is required. Everything else is optional, including the summary blockquote and the link lists, though a file with no links isn't doing much.

Each link list is ordinary markdown. A bullet, a hyperlink, and then optionally a colon and a short note about what the page covers. Those notes are the part people underuse. A bare list of URLs tells a model nothing it couldn't get from your sitemap. A list where each entry explains what the page is for is doing the actual work the format was designed for.

The spec also allows the file at any subpath, so /docs/llms.txt is valid. That flexibility turns out to matter more than it first appears, because the places the file genuinely earns its keep tend to be documentation subtrees rather than whole marketing sites.

How it differs from robots.txt and sitemap.xml

The name invites a comparison that doesn't hold. Robots.txt is a permission file. It tells crawlers where they may not go, and the major crawlers honour it. That compliance is the whole reason it works. Sitemap.xml is a completeness file. It tells a crawler every URL you have so nothing gets missed.

llms.txt is neither. It's a curation file, and curation only means something if the reader agrees to be curated. Robots.txt works because refusing to honour it carries a reputational cost. There's no equivalent pressure to read llms.txt, and no consequence for ignoring it.

Worth knowing if you run both: if you block a path in robots.txt and then list it in llms.txt, the crawler resolves that contradiction by obeying robots.txt. The curation file doesn't grant access it doesn't have.

What Google has said

Google's position has been consistent and unusually blunt. At the Search Central Deep Dive event in Asia Pacific in July 2025, Gary Illyes said Google doesn't support llms.txt and isn't planning to. The advice paired with it was that ordinary SEO practice is what gets content into AI Overviews, and that no separate discipline is required for it.

You can read that as a company protecting its existing crawl infrastructure, and there's something to that. Google already crawls your site thoroughly. A curated index of your best pages is a much smaller marginal gain for Google than it would be for a system starting from nothing.

Either way, if AI Overviews are the surface you care about, the file isn't the lever.

The adoption numbers

SE Ranking ran the largest public look at this, across roughly 300,000 domains, and found the file on about 10% of them. That's a real number and it's grown quickly from a much smaller base, so the format isn't dead in the sense of nobody bothering.

The more interesting half of that work was the citation test. They checked whether having an llms.txt correlated with how often a site appeared in AI answers, using both straightforward correlation and a gradient-boosted model that weighs many site factors at once. They found no meaningful relationship. Removing llms.txt from the model made its predictions slightly more accurate, which is what you'd expect from a variable contributing noise rather than signal.

That's one study, and correlation work on citations is genuinely hard because site authority, publishing cadence and structured data all move together. But it's the best evidence available, and it points at nothing.

Who publishes one, and what that proves

The strongest argument you'll hear in favour of the file is that the AI labs publish their own. That part is true. It's also the argument that gets misread most often.

We requested /llms.txt from a set of domains on 12 August 2026 and recorded the response.

All three major labs publish one, Google included. Its Gemini API documentation serves a file at ai.google.dev/gemini-api/docs/llms.txt, which sits oddly beside the same company telling you its search product won't read yours.

It gets odder. Chrome ships a Lighthouse audit that checks whether a site has an llms.txt, filed under agentic browsing checks. So one part of Google scores you on having the file while another part says it doesn't use it.

That isn't hypocrisy, and reading it as a hint that Google secretly does use the file would be a mistake. It's the distinction the whole debate keeps missing. Publishing an llms.txt and consuming one are separate acts with separate motives. The labs serve the file so that coding agents pointed at their docs can find the right page without crawling the whole tree, and Lighthouse checks for it because agents fetching your docs benefit from it. Neither says anything about whether a search crawler reads the file on your marketing site.

Notice too that the sites serving one skew heavily toward documentation. That's the strongest available clue about where the format earns its keep.

What version 2 changed

Howard published a second version of the proposal in August 2026, and it's worth reading because of what the revisions concede.

The largest addition is discoverability. v1 left agents to guess where a file might be, and v2 answers with standard link relations. A rel="alternate" link with type="text/markdown" points at a page's markdown version, and rel="describedby" points at the llms.txt file covering it. Both work as HTML link elements or as an HTTP Link header, and the header form is the interesting one, because a CDN can add it across a whole site without touching a single page.

Two smaller changes bless what people were already doing. v1 specified one URL form for markdown versions of pages, with .md appended to the full URL. Publishing tools had been replacing the extension instead, so v2 permits both. And subpath files now have defined semantics, where a file covers everything under its path and the most specific one wins. That last one is what lets a project on a shared host participate at all when it only controls a directory.

The removal is the most telling part. v1 described a tool for expanding an llms.txt into a single context blob, and v2 drops it, stating instead that agents should read or search the file and then follow the links they need. The Optional section, which existed to tell that tooling what to leave out, keeps its name as a convention and loses its mechanical meaning.

Read those together and the direction is clear. The proposal has moved away from being a feed that something ingests and toward being an index that something consults, and it has quietly narrowed itself to the use case that was already working.

Where it demonstrably works

Strip away the search angle and there's a real use case underneath, and it's the one the format was arguably built for.

When a developer points an agent at your documentation, the agent has a specific problem. It needs the authentication page and the endpoint reference, and it does not need your changelog, your marketing pages, or eleven blog posts about your Series B. An llms.txt that lists the ten pages that matter, each with a one-line note, solves that directly. The agent has been handed the file explicitly. There's no question of whether anything crawls it.

This is why documentation platforms generate them automatically now. It's a documentation navigation feature that happens to be shaped like an SEO file, and the confusion between those two things is most of why the format is argued about.

The case for adding one anyway

The honest case is cheapness. If you have documentation, write one, because the developer-agent use case is real and immediate. If you have a marketing site, an afternoon spent on a curated index is a defensible bet at that price, and there's no plausible downside beyond the time.

The line to hold is what you move to make room for it. If writing an llms.txt displaces work on the pages themselves, you've traded something that demonstrably affects citation for something that so far doesn't.

The case against bothering

The stronger objection is that maintaining it is where the cost actually lives. A curated index that stops matching the site is worse than no index. Links rot, pages get renamed, and the notes that made the file useful go stale first because nobody rereads them. A file that quietly describes your site as it was eighteen months ago is a liability, and it's the default outcome for anything without an owner.

The second objection is opportunity cost dressed up as diligence. There's a version of this work that's really about having something concrete to report, and a curated index is satisfying to produce in a way that improving eleven mediocre pages isn't. That's a bad reason to do it.

What to do instead, if you only have one afternoon

The things with actual evidence behind them are unglamorous and mostly predate AI search. Make the pages good and specific enough that a model quoting them produces something a reader can use. Keep your structured data accurate rather than extensive. Make sure the pages you care about are reachable, fast, and not hidden behind rendering that a crawler gives up on.

None of that is new advice, which is exactly Google's point about ordinary practice. The uncomfortable part of the current moment is that most of what works in AI search is the same work that always worked, and a new file at the site root is more satisfying to add than another pass over the content.

How to tell if this changes

Two things would move this from a lottery ticket to a real tactic. The first is a major provider stating that its retrieval reads llms.txt from third-party sites, in its own documentation rather than in a conference remark. The second is a replication of the citation study finding an effect that survives controlling for authority and content quality.

Until one of those lands, the file is a low-cost bet with a documented use in developer docs and no demonstrated use anywhere else. Treat it as exactly that, and don't let anyone sell it to you as an AI search strategy.

Frequently asked questions

Q: What is llms.txt?

A: It's a markdown file you put at the root of your site, at /llms.txt, that points language models to the pages you consider most useful. Jeremy Howard proposed it in September 2024. The only required part is an H1 with the project name, and most of the file is lists of markdown links under H2 headings, with an optional note after each link.

Q: Does Google use llms.txt?

A: No. Gary Illyes said at Google's Search Central Deep Dive in Asia Pacific in July 2025 that Google doesn't support llms.txt and isn't planning to, and Google's guidance has been that ordinary SEO is what gets content into AI Overviews. Nothing since has changed that position.

Q: Is there evidence that llms.txt increases AI citations?

A: Not so far. SE Ranking looked at roughly 300,000 domains and found no meaningful relationship between having the file and how often a site got cited in AI answers. Dropping llms.txt from their model actually made its predictions slightly better, which suggests the file was contributing noise rather than signal.

Q: Why do the AI labs publish llms.txt files if models don't read them?

A: Because publishing one and consuming one are different acts. OpenAI, Anthropic and Google all serve an llms.txt on their developer documentation so that coding agents pointed at those docs can find the right page quickly, and Chrome's Lighthouse audits sites for one on the same reasoning. Those are decisions about making documentation agent-readable, and none of them tells you that a search crawler fetches the file from your site.

Q: Should I add one anyway?

A: If you have developer documentation that agents get pointed at directly, yes, because that's the use case it demonstrably serves. For a marketing site hoping to lift citations, treat it as a cheap lottery ticket rather than a project. It takes an afternoon, it can't hurt, and you shouldn't move anything else down your list to make room for it.

Q: How is llms.txt different from robots.txt?

A: Robots.txt is a permission file that tells crawlers where they may not go, and the major crawlers honour it. llms.txt is a curation file that suggests where a model should look, and it carries no permissions at all. Blocking a crawler in robots.txt while listing the same pages in llms.txt is a contradiction the crawler resolves by obeying robots.txt.