What is llms.txt and does your website need one?
An honest look at llms.txt: what the proposed standard is, how it is structured, what is and is not known about AI support, and when it is worth adding.
llms.txt is a proposed convention for a plain Markdown file at the root of a website that gives AI models a short summary of the site and links to its most useful pages. It is not an official standard, and support from major AI companies is unconfirmed, so it is a low-cost optional extra rather than a priority.
Key takeaways
- llms.txt was proposed in 2024 as a guide for AI models; it is not a formal web standard.
- It does not block or allow crawlers; robots.txt still controls access.
- No major AI assistant has clearly confirmed using it, and Google has said Search does not.
- It is quick to create, so adding one is harmless if it stays accurate.
- Clear pages, consistent facts and crawler access matter far more for AI visibility.
If you have spoken to a web developer or SEO consultant recently, there is a good chance llms.txt came up, sometimes described as "robots.txt for AI" or as something every site now needs. Some providers now sell llms.txt setup as a standalone service.
The reality is more modest. llms.txt is an interesting idea, it takes little effort to add, and it may become more useful over time. But its effect today is uncertain, and it is nowhere near as important as the basics of being understood by AI search. Here is a plain assessment.
What is llms.txt?
llms.txt is a text file written in Markdown and placed at the root of a website, for example yoursite.com/llms.txt. It was proposed in September 2024 by Jeremy Howard of Answer.AI as a way to give large language models a concise, curated guide to a site's content.
The reasoning is sensible. Web pages are full of navigation, scripts, pop-ups and layout code, and AI tools have limited context windows. A short file that says "here is who we are and here are our most important pages, in clean text" could help a model use the site more efficiently, particularly when it is fetching pages to answer a question.
What goes inside an llms.txt file?
The proposal describes a simple structure that both people and machines can read:
- An H1 heading with the name of the site or organisation. This is the only required part.
- A short summary in a blockquote explaining what the site or business is.
- Optional paragraphs with further context, such as the areas served or how the content is organised.
- H2 sections containing lists of links, each with a short note on what the page covers, for example services, guides or policies.
- An optional section for secondary links that can be skipped when space is short.
The proposal also suggests offering clean Markdown versions of important pages. Some sites additionally publish a longer companion file, often called llms-full.txt, containing full text, though this is a community practice rather than part of any formal specification.
How is llms.txt different from robots.txt and sitemaps?
They do three different jobs, and llms.txt replaces neither of the others.
| File | Purpose | Status |
|---|---|---|
| robots.txt | Tells crawlers which parts of a site they may or may not access | Long-established and formalised as an internet standard; respected by major search and AI crawlers |
| XML sitemap | Lists URLs so search engines can discover and recrawl them | Widely supported by search engines |
| llms.txt | Offers AI models a curated, readable summary of key content | Proposed convention; adoption and use by AI companies unclear |
The most common misunderstanding is that llms.txt can stop AI companies using your content. It cannot. If you want to allow or block crawlers such as GPTBot, OAI-SearchBot or PerplexityBot, that still happens in robots.txt and, where needed, at your firewall or CDN.
Do ChatGPT, Gemini and other AI tools actually use llms.txt?
As of 2026, there is no clear public confirmation that the major AI assistants use llms.txt when choosing sources or generating answers. Google representatives have said publicly that Google Search does not use it. Other large AI providers have not announced support in their crawler documentation. Some AI coding tools and documentation platforms do read or generate llms.txt files, which is where most real-world use has been.
That could change. Conventions sometimes gain adoption gradually, and a cheap, readable summary file is the kind of thing AI tools might start to use. But anyone promising that llms.txt will get your business recommended by ChatGPT is claiming more than the evidence supports.
Does your website need an llms.txt file?
Most business websites do not need one, but many can add one at little cost. A simple decision framework:
- Worth adding now: software companies, SaaS products and businesses with technical documentation or APIs, where developers use AI coding tools that may read the file.
- Optional, low priority: service businesses, clinics, retailers and agencies that have already sorted out crawler access, clear service pages and consistent business information.
- Not yet: any site with basic problems such as blocked pages, slow mobile performance, thin content or conflicting contact details. Fix those first.
If you do add one, keep it short, accurate and maintained. An llms.txt that lists old services or broken links does more harm than having none.
How do you create an llms.txt file, step by step?
- Write a one or two sentence summary of your business, including what you do and where, for example "a Dubai clinic offering general and paediatric dentistry in English and Arabic".
- Choose 10 to 30 of your most useful pages: core services, locations, pricing explainers, key guides, and contact and policy pages.
- Group them under clear H2 headings with a one-line note for each link.
- Save the file as plain text named llms.txt and upload it to the root of your domain.
- Check it loads at yoursite.com/llms.txt and is not blocked by security rules.
- Add a reminder to review it whenever you launch, rename or remove services.
On bilingual sites, you can include links to Arabic pages in their own section so the file reflects both audiences.
Keep the tone factual. The file is not a place for sales copy or keyword lists; if llms.txt does become widely used, AI tools will value it for the same reason people would, because it describes the site accurately and briefly. Write it as you would brief a new colleague on where to find things.
What should you focus on instead?
The factors known to influence AI visibility are the same ones that help search: pages AI crawlers can access, direct answers to real questions, consistent business facts across the web and credible third-party mentions. Our practical guide on getting your business mentioned by ChatGPT and other AI assistants walks through them in order. Treat llms.txt as a small finishing touch on top of that work, and revisit the decision once AI providers publish clearer guidance on whether they read it.
How Upscaleads can help
As part of our GEO services, we review AI crawler access, robots.txt and content clarity first, and add an llms.txt file where it makes sense for your site, without overselling it. If your wider technical setup needs attention, our SEO team can help. Get in touch for a straight answer on what your site actually needs.
Frequently asked questions
Is llms.txt an official web standard?
No. llms.txt is a proposal published in 2024 and adopted voluntarily by some websites and tools. It has not been standardised by a body such as the IETF or W3C, and there is no formal validation or enforcement. Robots.txt, by comparison, is long established and was formalised as an internet standard in 2022. Treat llms.txt as an emerging convention whose future is still uncertain.
Can llms.txt stop AI companies from training on my content?
No. llms.txt is a guide to useful content, not an access control. To tell crawlers such as GPTBot, Google-Extended or other AI bots not to use your site, add rules to your robots.txt file, and consider firewall or CDN rules for crawlers that ignore robots.txt. Blocking some AI crawlers may also reduce your visibility in AI search tools, so decide deliberately.
Will adding llms.txt hurt my Google rankings?
There is no reason to expect it to. Google has indicated that it does not use llms.txt for Search, so the file should neither help nor harm rankings. The only practical risk is an inaccurate or outdated file that describes services you no longer offer. Keep it short, correct and reviewed whenever your site changes.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index: a summary of the site and a curated list of links with brief descriptions. llms-full.txt is a community convention for a much longer file that includes the full text of key pages in one document, popular on software documentation sites. Neither is widely confirmed as used by major AI assistants, and most business sites only need the short version, if any.
