llms.txt is having a moment. It shows up in GEO checklists, agency pitch decks, and “rank in ChatGPT” listicles, usually described as the new robots.txt for AI.
Here is the uncomfortable part: Google has said plainly that it does not need AI-specific files. No major AI engine has committed to reading llms.txt. And yet I publish one on this site, I keep it updated, and I think you probably should too.
Those positions are not contradictory. They just require knowing what the file actually is, who reads it today, and what it can and cannot do for you. That is this post.
TL;DR
- What it is: llms.txt is a proposed standard, published by Jeremy Howard in September 2024, that gives AI systems a clean, curated markdown index of a website’s most important content.
- Google ignores it: Google’s documentation states there is no need to create special machine-readable files for AI Overviews or AI Mode, so llms.txt does nothing for Google AI Overview visibility.
- ChatGPT does not commit to it: There is no published evidence that ChatGPT reads llms.txt, and the documented path to ChatGPT visibility is allowing OAI-SearchBot in robots.txt and being crawlable.
- Who actually uses it: AI developer tools and coding assistants fetch llms.txt files to load documentation context, which is why companies like Anthropic and Supabase publish them.
- My verdict: I publish one because it costs an hour, carries zero risk, and forces a clean entity definition of the business, but real AI citations still come from crawlable, structured, authoritative content.
What is llms.txt?
llms.txt is a proposed standard, published by Jeremy Howard in September 2024, for giving large language models a curated, LLM-friendly index of a website. The file lives at the root of your domain, the same way robots.txt does, and it is written in plain markdown.
The reasoning behind it is solid. AI systems work with limited context windows, and your actual website is a hostile environment for them: navigation, scripts, cookie banners, footers, and layout markup wrapped around the content that matters. llms.txt hands the machine a clean map instead: here is who we are, here is what matters, here are direct links with descriptions.
Note what it is not. It is not an access-control file like robots.txt, which tells crawlers what they may not touch. llms.txt is the opposite: an invitation and a guide to your best content.
What does an llms.txt file look like?
The spec defines a simple markdown structure, in this order:
- An H1 with the site or project name. The only required element.
- A blockquote summarizing what the site is and who it serves.
- Optional context paragraphs with anything an AI should know upfront.
- H2-delimited sections containing link lists. Each entry: a markdown link, a colon, a one-line description.
- An optional “Optional” section for secondary links that can be dropped when a shorter context is needed.
A minimal example, abbreviated from the one on this site:
# Arthur Dosik
> Arthur Dosik is an AI Consultant, SEO and GEO Strategist with 20+
> years of experience in search and digital marketing.
## Pages
- [SEO and GEO Strategy](https://arthurdosik.com/seo-geo-strategy/):
Integrated SEO and GEO including AI Overviews strategy and citation share.
## Articles
- [GEO is Just SEO With a Rebrand](https://arthurdosik.com/blog/geo-is-just-seo-with-a-rebrand/):
Why GEO is not a new discipline, and where to start.That is the whole format. If you can write a README, you can write an llms.txt.
llms.txt examples worth studying
- Anthropic publishes one for the Claude developer docs: a long, sectioned link index that AI coding tools can pull into context.
- Supabase uses theirs as a hub pointing to per-product full-text files like llms/js.txt and llms/cli.txt, a nice pattern for large doc sites.
- Mine covers a consulting site: services, articles, prompt library, and contact, with the blockquote doing the entity-definition work I would want any AI to get right.
Notice who is on that list. Developer documentation and expertise sites, where the consumer is often an AI tool fetching context on demand. That is not an accident, and it leads to the honest part.
Does Google use llms.txt?
No. Google’s documentation on AI features says there is no need to create special machine-readable files for AI Overviews or AI Mode. Google’s AI surfaces are built on the regular search index, the one your existing pages are already in.
So let me say it without hedging: publishing llms.txt will do nothing for your Google AI Overview visibility. Any agency selling llms.txt as an AI Overviews tactic is selling you a file Google has explicitly said it does not need.
Does ChatGPT read llms.txt?
There is no published evidence that it does. OpenAI documents its crawlers clearly: GPTBot gathers training data, OAI-SearchBot powers ChatGPT search visibility, ChatGPT-User handles live user-initiated fetches. llms.txt appears nowhere in that documentation. If you want to be visible in ChatGPT, the documented mechanism is being crawlable by OAI-SearchBot, which is a robots.txt and content problem, not an llms.txt problem.
Where llms.txt demonstrably gets used today is narrower and more interesting: AI developer tools. Coding assistants and agent frameworks fetch llms.txt files to load documentation context on demand. If your site is documentation, an API, or a knowledge base that AI tools might pull into a working context, the file earns its keep right now. That is exactly why Anthropic and Supabase bother.
Should you create one anyway?
Yes, if it costs you an hour. Here is the honest cost-benefit:
| The case | |
|---|---|
| For | Costs an hour. Zero risk. Useful to AI dev tools today. Forces you to write a clean entity definition of your business. Cheap insurance if any engine adopts the standard later. |
| Against | No major search or answer engine commits to reading it. Zero measurable visibility lift today. One more file to keep updated, and a stale index is worse than none. |
The reason I land on “publish it” is the second-order benefit: writing the file forces you to answer, in one blockquote, what your business is, who it serves, and what your most important pages are. That clarity exercise is the same entity-definition work that actually does move AI visibility. The file is a byproduct of work worth doing anyway.
How to create an llms.txt file
- Write the H1 and blockquote first. Name, then two or three sentences: who you are, what you do, who you serve. This is the highest-value part of the file.
- List your core pages under an H2, each with a one-line description written for a machine: literal, specific, no marketing adjectives.
- Add sections that fit your site. Articles, documentation, products, contact. Keep each entry to one line.
- Consider an llms-full.txt if you publish substantial content. The index says what exists; the full file carries the actual text so a retrieval system can use it without crawling. This site generates llms-full.txt at build time from the same content that makes the pages.
- Save it as plain markdown at your domain root and add it to your deploy process so it cannot silently go stale.
- Update it when you publish. A curated index that is missing your last six months of work tells a machine your site stopped.
Skip the generators if your site is small. The file is short, the thinking is the value, and hand-writing it takes less time than evaluating tools to avoid hand-writing it.
How do you get indexed by LLMs through llms.txt?
You do not, and this misconception is common enough that it deserves its own answer.
llms.txt is not a submission form. There is no queue on the other end. No engine treats the file as a request for inclusion. LLM visibility comes from the unglamorous stack: being crawlable by the AI bots in your robots.txt, existing in the sources engines retrieve from, publishing content structured for passage-level extraction, and being the kind of source other sites cite.
llms.txt rides along as a courtesy map. The map is worth drawing. Just do not confuse drawing the map with being on anyone’s route.
The takeaway
llms.txt is a reasonable idea, a trivial implementation, and a wildly oversold tactic. Google does not read it. ChatGPT has not committed to it. AI developer tools genuinely use it. Publish one because it costs an hour, forces useful clarity about your entity, and might matter more later. Then put your real effort where the citations actually come from: crawlable, structured, authoritative content.
The file is the easy part. It was always going to be the easy part.
Frequently asked questions
What is llms.txt?
llms.txt is a proposed web standard, published by Jeremy Howard in September 2024, for giving large language models a curated, markdown-formatted index of a website’s most important content. The file lives at yourdomain.com/llms.txt and contains the site name, a short summary, and organized lists of links with one-line descriptions. The goal is to help AI systems find and use your best content without parsing your full HTML, navigation, and scripts.
Does Google use llms.txt?
No. Google’s official documentation on AI features states there is no need to create special machine-readable files or AI-specific text files to appear in AI Overviews or AI Mode. Google’s AI surfaces work from the regular search index. Publishing an llms.txt file will not improve your Google AI Overview visibility, and anyone claiming otherwise is ahead of the evidence.
Does ChatGPT read llms.txt?
OpenAI has not announced that ChatGPT’s crawlers consume llms.txt, and there is no published evidence that it influences ChatGPT search results. The documented way to be visible in ChatGPT search is to allow OAI-SearchBot in robots.txt and be crawlable. Where llms.txt files demonstrably get used today is in developer tools and AI coding assistants that fetch documentation context, which is why companies like Anthropic and Supabase publish them.
What does an llms.txt file look like?
An llms.txt file is plain markdown with a defined structure: an H1 with the site or project name, a blockquote summary, optional context paragraphs, and H2-delimited sections containing link lists where each entry is a markdown link followed by a one-line description. An optional final section lists secondary links that can be skipped when an AI needs a shorter context. It reads like a curated table of contents for machines.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the short curated index: links plus one-line descriptions. llms-full.txt is the expanded version that includes full page or article content in one file, so an AI system can load your actual material without crawling each URL. Many sites publish both: the index for orientation, the full file for retrieval. Mine are at arthurdosik.com/llms.txt and arthurdosik.com/llms-full.txt.
Should I create an llms.txt file for my website?
If it costs you an hour, yes. The realistic upside is modest: today the file mostly serves AI developer tools and any retrieval system that chooses to look for it. The downside is zero, it cannot hurt you. Treat it as cheap insurance on a possible standard, not as a GEO tactic that will move your visibility this quarter. Crawlability, content structure, and authority remain the work that actually moves AI citations.
How do I get indexed by LLMs through an llms.txt file?
You do not, and that framing is the most common llms.txt misconception. The file is not a submission mechanism and no major AI engine treats it as an index request. LLM visibility comes from being crawlable by AI bots like GPTBot and OAI-SearchBot, being present in the sources AI engines retrieve from, and publishing content structured for extraction. llms.txt can ride along as a courtesy map of your site. It does not get you in the door.
If you want your site’s technical and content foundations checked against how AI engines actually retrieve and cite, that is the core of my AI Search Visibility and SEO Strategy service. Or book a free 30-minute call and ask me anything, including whether llms.txt is worth your hour. No pitch, no pressure.
