llms.txt is a proposed standard for giving AI systems a clean, curated index of a website’s most important content. The file lives at the domain root, the same place as robots.txt, and it is written in plain markdown: a site name, a short summary, and organized sections of links with one-line descriptions.
Jeremy Howard published the spec in September 2024, reasoning that AI systems work with limited context windows and that a typical website, full of navigation, scripts, and layout markup, is a difficult environment for a model trying to extract what actually matters. llms.txt hands the machine a shortcut instead: here is who this is, here is what matters, here are the direct links.
What does the llms.txt format look like?
The spec defines a simple structure, in order: an H1 with the site or project name, a blockquote summarizing what the site is and who it serves, optional context paragraphs, and H2-delimited sections containing link lists, each entry a markdown link followed by a colon and a one-line description. An optional final section can hold secondary links that a model can skip when it needs a shorter context. It reads like a curated table of contents written for a machine rather than a person. I walk through a full example, including how this site’s own file is structured, in what is llms.txt.
How is llms.txt different from llms-full.txt?
llms.txt is the short index: links and one-line descriptions, meant for orientation. llms-full.txt is the expanded companion that includes full page or article content in a single file, so a retrieval system can use the actual material without crawling each URL separately. Not every site needs the full version, but publishing both gives a model a quick map and, if it wants more, the complete text in one request.
Do AI crawlers actually read llms.txt?
Honestly, adoption in 2026 is unproven and debated. Google has said plainly that no special AI-specific files are needed to appear in AI Overviews or AI Mode, so llms.txt does nothing for Google AI visibility. No major chat assistant, including ChatGPT, has publicly committed to reading llms.txt as part of how it decides what to cite. The clearest real-world use today is narrower: AI coding assistants and developer tools that fetch llms.txt files to pull documentation into a working context, which is why documentation-heavy sites like Anthropic’s and Supabase’s publish them.
That gap between hype and confirmed use does not mean the file is worthless, only that it should be treated as low-cost insurance rather than a proven ranking lever.
Is it worth creating one?
Yes, if it costs an hour of work and nothing more. The exercise of writing a tight blockquote summary and curating a list of a site’s best pages is useful on its own, independent of whether any given AI engine ever fetches the file. For a fast starting point, arthurdosik.com’s free llms.txt generator builds the file from a site’s existing pages in the browser. Whether hand-written or generated, keep it updated: a curated index missing the last six months of work tells a machine the site went quiet.