Blog
Not a standard, but a growing practice.
What llms.txt is, whether major AI providers already act on it, and whether your site needs it.
llms.txt is a proposal for a markdown file in your site's root that gives AI assistants an ordered overview of your content.
It is not an established standard and no major AI provider has publicly promised to read it.
Google even stated explicitly that Search does not use the file.
Below is what belongs in it and when it is worth the effort.
What is llms.txt?
llms.txt is a short markdown file that describes what is on your site and where the main pages can be found.
The proposal comes from Jeremy Howard and has been on llmstxt.org since 3 September 2024, where it has been updated since.
The reason behind it is technical, not commercial.
A language model that fetches a page gets back HTML with navigation, cookie notices and scripts, while its context window is limited.
A list of links with one line of explanation per link costs far fewer tokens.
What does such a file look like?
The format is strict and has exactly one mandatory part: the H1.
The rest is optional, but does follow a fixed order.
- H1: the name of the site or project, the only mandatory part.
- Blockquote: a summary of a few lines about what the site is and who it is for.
- Free paragraphs: optional background without headings.
- H2 sections: per section a list of markdown links, each optionally followed by a colon and a short note.
- Optional section: by convention the place for links an agent may skip when context is tight.
My own file lives at /llms.txt and largely follows that pattern: an H1 with my name, a blockquote about fifteen years of Drupal, Herk-de-Stad and my working languages, then H2 sections for services, case studies, articles, expertise, contact and the English version.
Behind the links under services and case studies sits one sentence saying what the page is about.
No sales language, because the file is meant to be read, not to persuade.
Where should the file live?
In the root of your domain, so at /llms.txt, as plain text with markdown in it.
A file on a subpath is also allowed, for example /docs/llms.txt, which then applies to the URLs under that path, with the most specific file taking precedence.
That is meant for documentation sites with several product lines.
The proposal actually goes further than that one file: you also offer clean markdown versions of your pages, at the same URL with .md added or with .md instead of the extension.
That part is more work, because it needs a separate rendering in your CMS.
llms.txt, robots.txt and sitemap.xml: three different things
Only robots.txt and sitemap.xml are agreements that crawlers actually follow. llms.txt looks similar on the surface, but the difference lies in its status and in who reads it.
| File | Status | What it is for | Who reads it |
|---|---|---|---|
| robots.txt | RFC 9309, standard since September 2022 | Allow or deny access per user agent | Search engines and most AI crawlers |
| sitemap.xml | Sitemap protocol 0.9, supported since 2006 | List all indexable URLs in XML | Search engines, mechanically |
| llms.txt | Proposal, not a standard | Summarise the site and point to the key pages | Unknown, no provider confirms it |
So an llms.txt replaces neither of them.
Whether AI crawlers may fetch your pages is governed by robots.txt, and making sure everything gets found remains the job of your sitemap.
That is why, in technical SEO, I look at those two first before adding a new file.
What belongs in it, and what doesn't?
Links to the pages you want cited, with one sentence of explanation per link.
That's the entire content.
A file with a few dozen links works better in practice than an exhaustive list, because the value lies precisely in the selection you make.
What doesn't belong in it matters just as much.
No keyword list, because the file is maintained by you and is therefore just as reliable as the old keywords meta tag.
No instructions to the model, because your own server is not the place to enforce an assistant's behaviour.
No URLs you want kept out of the index, because you are simply publishing them here.
And no content that diverges from the pages themselves.
What can you realistically expect from it?
At this point, virtually nothing measurable.
Gary Illyes said at Search Central Live Deep Dive Asia Pacific (July 2025) that llms.txt is not a Google initiative, that Google Search does not use it, and that no plans are known.
John Mueller compared the file to the keywords meta tag, which search engines have ignored for more than ten years because the site owner fills it in themselves.
At the same time, since Lighthouse 13.3 (May 2026), Google does check for the presence of an llms.txt in the domain root in the experimental Agentic Browsing category, not for Search but for agent readability, arguing that agents would otherwise have to crawl more.
Other providers have likewise not publicly documented that they fetch it.
What you do see are individual agents and developer tools that use the file when it's there, especially with technical documentation.
A real but narrow benefit, so don't count on visitor growth.
Structured data already does what llms.txt promises
If you want to do something for AI discoverability today, you start with schema.org rather than a separate file.
Structured data is not an extra URL but a layer within the page itself: a block of JSON-LD that repeats, in machine language, what the text already says.
Who the author is, when the article was last updated, which organisation is behind it, which question belongs to which answer.
Because it lives at the same URL as the content, it can't drift away from your pages the way a manually maintained link list can.
If you move your site, all of that still has to hold up: that is what migrating without losing your search visibility is about.
The difference with llms.txt again comes down to status. schema.org is a shared vocabulary that search engines effectively read, with documentation on which types are supported and validators to check that your markup is correct.
Types such as Article, Organization, Person, BreadcrumbList and FAQPage cover most of what a business site needs.
For an AI assistant that delivers the same benefit as for a crawler: less guesswork about what a piece of text means and who is saying it.
In Drupal I prefer to put that markup in the templates and in the fields you already fill in, not in a separate text field someone has to remember.
That way every new page gets it automatically and it stays correct as your content changes.
That is part of your Drupal architecture and development, and how you make your own content and data more usable for models is something I wrote up in your website as an AI source.
Do you need llms.txt?
For most websites, no, or at least not as a first step.
If you run a business site of thirty pages, a correct HTML structure, a logical heading hierarchy, accurate structured data and fast pages deliver more than a file that maybe nobody requests.
That's the honest version, even though it sells less well.
It does make sense in a few cases.
With extensive technical documentation, an API reference or a knowledge base that developers search with AI tools, the file earns its place quickly.
The same applies if your site is large and layered and you want to steer which twenty pages represent your organisation.
And it costs little: an hour to write, then an update whenever your structure changes.
That's why there is one on my own site, as a trial, not because I know it gets read.
If you want to know where your site technically stands for search engines and AI assistants, that's where you start.
I map that out as part of technical SEO and frontend performance, and I always tell you what you can safely leave alone.
Send your question via contact and you'll have an honest picture within 24 hours.
Frequently asked questions
What is llms.txt?
llms.txt is a markdown file at the root of a website that gives AI assistants a summary of that site, plus links to the most important pages.
The proposal comes from Jeremy Howard and dates from 3 September 2024.
It consists of a mandatory H1, a blockquote, and H2 sections with link lists.
It remains a proposal, not a standard, so nobody is required to read it.
Does Google use llms.txt?
Not for Search.
Google Search doesn't support llms.txt: Gary Illyes said at Search Central Live Deep Dive Asia Pacific (July 2025) that it's not a Google initiative and no plans are known, and John Mueller compared it to the keywords metatag, which has been ignored for over ten years because the site owner fills it in themselves.
Chrome's Lighthouse has been checking the file since May 2026 though, in the experimental Agentic Browsing category, so for agent readability and not for Search.
What is the difference between llms.txt and robots.txt?
robots.txt controls access, llms.txt describes content. robots.txt has been a real standard since September 2022, RFC 9309, and states per user agent which paths may be crawled. llms.txt says nothing about permission and only gives an overview of your pages.
If you want to block or allow AI crawlers, you do that in robots.txt.
Should I also create an llms-full.txt?
You don't have to, since it's not part of the proposal. llms-full.txt is a convention that grew out of the field, usually a single file with the complete text of a documentation site.
The proposal asks for something else: clean markdown versions of your existing pages.
A text file like that quickly becomes large and outdated, so maintenance weighs heavier here than completeness.
Does structured data help more than llms.txt for AI assistants?
Probably, since structured data sits in the page itself and gets effectively read.
You write a block of JSON-LD following schema.org's vocabulary that states, in machine language, who the author is, what the page is about, and which question belongs to which answer.
Types like Article, Organization, Person, BreadcrumbList and FAQPage cover most of what a business site needs.
Because it lives on the same URL as your text, it can't drift out of sync with your content, and that's exactly the weak point of a manually maintained link list like llms.txt.
Can I do SEO optimisation myself?
A large part, yes.
Titles, meta descriptions, heading structure, internal links and correctly sized images you manage yourself in Drupal, and Lighthouse gives you a first picture.
What remains manual work is caching on the right layer, render-blocking resources, structured data in code, and cleaning up duplicate URLs.
Tools for the basics, manual work for the rest.
A question about this topic?
Briefly describe your situation, and I'll let you know what's going on and what it would cost. No sales pitch.
