What is llms.txt, and do I need one?
By Purlock · Updated 2 min read
The short answer
llms.txt is a plain text file at the root of a website, like yourdomain.com/llms.txt, that summarises what the site is and links to its most useful pages, written for AI tools to read. It was proposed in 2024 and is optional. Some AI tools and developer platforms use it, but major search engines haven't said they rely on it, so it's a quick extra rather than a way to rank. Purlock's AI crawler checker drafts one from your homepage if you don't have one.
Key points
- It's a summary for AI tools, not a set of rules like robots.txt.
- It takes minutes to write, and does no harm.
- Google has said it doesn't use it, so don't expect ranking changes.
- Allowing AI crawlers in robots.txt matters far more.
What goes in an llms.txt file?
The proposed format is Markdown: a title with the site's name, a one paragraph summary in a quote, then sections of links with a short description of each. A longer companion file, often called llms-full.txt, can hold more detail in one place.
A simple example: # Your Shop, then > Your Shop sells handmade ceramics and ships across the US, then a ## Pages section listing your shop, shipping, returns and contact pages with one line each.
AI crawler checker
Which AI bots can read your site, and a draft llms.txt.
How is llms.txt different from robots.txt and a sitemap?
| File | Who reads it | What it does |
|---|---|---|
| robots.txt | Every well behaved crawler | Says which pages crawlers may visit |
| sitemap.xml | Search engines | Lists every page you want found, with dates |
| llms.txt | Some AI tools | Explains the site and points to the best pages |
Do AI tools actually read llms.txt?
Some do, especially coding assistants and documentation tools that fetch a site's docs. Major AI search products haven't confirmed they use it, and Google has said it doesn't. That may change, and since the file is cheap to make, many sites add one anyway.
What reliably matters for AI answers is that their crawlers can reach your pages and that the pages answer questions clearly.
If robots.txt or your host blocks AI crawlers, an llms.txt file won't be read either.
How do I create an llms.txt file?
- Write a one paragraph summary of what you do and who it's for.
- List your 5 to 20 most useful pages, each with one line saying what's there.
- Save it as llms.txt in your site's root folder, so it loads at yourdomain.com/llms.txt.
- Or run the AI crawler checker, which drafts one from your homepage that you can edit.
What should I put in llms.txt for a small business?
Keep it short and true. It's read by tools that summarise you to people, so anything vague or exaggerated may be repeated back to customers.
- One sentence on what you sell or do, and who it's for.
- Where you operate, and any important limits, like regions you ship to.
- Your pricing page, and the starting price in words.
- Your most useful pages: product, pricing, guides, contact.
- Anything assistants often get wrong about you, stated plainly.
More questions
- Will llms.txt help my Google rankings?
- No. Google has said it doesn't use it.
- Can llms.txt block AI training?
- No. Use robots.txt to block training crawlers like GPTBot or Google-Extended.