What it is and why it was proposed
The proposal lives at llmstxt.org, signed by Jeremy Howard of Answer.AI on 3 September 2024. The problem it set out to solve is real: a web page is built for people, with menus, notices, scripts and design, and to a language model that is noise taking up room in its context. The idea is to give it, at a fixed address, a clean index in Markdown: what this site is and where the important parts are.
A second version of the proposal was published in August 2026, now aimed at AI agents. It allows files in subdirectories (the most specific one wins) and defines how to link Markdown versions of each page.
The format
It is Markdown with a fixed structure, in this order:
- An H1 title with the site's name. It is the only required part.
- A blockquote (
>) with a short summary. - Optional paragraphs or lists, without headings, with whatever context is needed.
- H2 sections with lists of links in the form
[name](url): notes.
By convention, a section called "Optional" holds secondary links an agent can skip when it is short of space. An extract from ours:
# Volqer
> Search and AI visibility. Free live SEO and GEO audits, a EUR 49 full report,
> and done-for-you management across organic and paid surfaces.
## Pages
- [SEO audit](https://volqer.com/seo-audit): Run a live SEO audit of any domain.
- [Services](https://volqer.com/services): Every service with its price and what it includes.You can see all of it at volqer.com/llms.txt.
Who publishes it, and who reads it
Two things that often get mixed up need separating here.
Who publishes it. A lot of people, starting with the AI companies themselves: the documentation of OpenAI, Anthropic and Perplexity has its llms.txt, and so do several of Google's developer sites, such as web.dev and the Chrome documentation. Many documentation platforms generate one automatically.
Who reads it. Here the evidence is thin:
- Google Search does not use it. Its guide to optimising for generative AI features has said so unambiguously since June 2026: it is not needed to appear in Search, it neither helps nor harms visibility or rankings, and there is no problem keeping one for other services.
- OpenAI, Anthropic, Perplexity, Apple and Meta do not mention llms.txt in their crawler documentation. Some site owners have seen requests for the file in their server logs, but none of these companies has said it uses it to decide what to cite.
- Chrome does look at it, though not for ranking: Lighthouse includes an llms.txt check among its agentic browsing audits. If you do not have one, it marks the check "not applicable"; it only fails if your server returns an error when asked for it.
Where it makes the most sense today is technical documentation, which is where it has spread furthest: a developer can hand it to a coding assistant so it understands an API without walking through a hundred pages.
llms.txt, robots.txt and sitemap.xml are not the same
| File | For whom | What it does | Do search engines use it? |
|---|---|---|---|
robots.txt | Crawlers | Says what they may and may not crawl | Yes, all the major ones |
sitemap.xml | Search engines | Lists the URLs you want indexed | Yes |
llms.txt | Models and agents | Summarises the site and links the important parts | Google says no; the others have not confirmed it |
A common confusion: llms.txt does not control access. If you want to allow or block AI bots, that is done in robots.txt. We explain it in AI crawlers in robots.txt.
How to create yours in ten minutes
- Create a text file called
llms.txtat the root of your domain, in UTF-8. - Start with an H1 carrying your brand and a blockquote with one sentence saying what you do, for whom and where. Use the same description as in your Organization structured data and your profiles.
- Add H2 sections with your key pages: services (with prices if they are public), the pages that explain how you work, and your guides. Absolute links, each with a short note.
- Keep it short and curated: it is an index, not a dump of the whole site.
- Check that the address answers 200. A server error is the only thing Lighthouse counts as a failure.
- Update it when you publish or remove important pages.
Is it worth it?
If you have ten minutes, yes: it costs next to nothing, it saves work for the agents that look for it, and Google confirms it does no harm. What is not worth it is treating it as a visibility lever. If ChatGPT or Google do not mention you, the reason will almost never be a missing llms.txt. Look first at crawler access, indexing, the clarity of your brand and what other sites say about you. That is where our free GEO audit starts, and we explain it in what is GEO.
Sources
- llmstxt.org: the proposal, by Jeremy Howard (September 2024; version 2, August 2026)
- Google Search Central: Optimizing for generative AI features
- Google Search Central: latest documentation updates (note on llms.txt, June 2026)
- Chrome for Developers: llms.txt audit in Lighthouse
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
Can AI assistants read your site?
The free GEO audit checks, live, whether AI crawlers can reach you, whether your brand is a clear entity and whether your pages are ready to be cited.