What is llms.txt?
llms.txt — the index AI readers start from.
llms.txt is one Markdown file at the root of your site. It lists the pages that matter, each with a one-line description, so an AI assistant gets a curated map instead of a guess. Proposed by Answer.AI in 2024 — and one of the things Seogence serves for your domain.
The index is the front door. The pages behind it are served as clean Markdown too — see Markdown for AI agents.
What it is
One file, readable by anything.
The format is deliberately plain: a heading with the site’s name, a one-paragraph summary, then sections of links with short descriptions. No XML, no schema, no build step. A person can read it in a text editor; so can a model.
# Yoursite Living
> Handmade luxury decor from a small Swedish workshop. Product pages,
> care guides, and shipping details — key pages linked below.
## Products
- [Luxury Decor Bowl](https://yoursite.com/product): Handmade bowl, food-safe, one of a kind.
- [Oak Serving Board](https://yoursite.com/serving-board): Solid oak, oiled finish.
## Guides
- [Care instructions](https://yoursite.com/care): Cleaning and storage for wooden pieces.
## Optional
- [Press kit](https://yoursite.com/press): Logos and product photography.The format
Four elements, in order.
# Site nameA level-one heading with the site’s name. The only required element.> SummaryA blockquote right after the title: what the site is, in a sentence or two.## SectionGroups of links, one per line — [title](url): description. The description is the point: it tells a reader what the page means before fetching it.## OptionalThe one reserved section name. Links here are secondary — a reader short on context can skip them.
That is the whole format. The specification lives at llmstxt.org.
Not a replacement
Three files, three different jobs.
robots.txt controls access. sitemap.xml lists URLs. llms.txt explains what matters. They are complements, not rivals — a site that takes machine readers seriously keeps all three.
| File | Who reads it | What it answers |
|---|---|---|
robots.txt | Crawlers, before they fetch anything | Which parts of the site may be accessed at all. |
sitemap.xml | Search crawlers | Every URL you want indexed, with last-modified dates. |
llms.txt | AI assistants and agents | The pages that matter, curated, with a one-line meaning attached to each. |
Do it yourself
You can write one by hand.
Create a text file. Add the heading, the summary, and your key pages with one-line descriptions. Serve it at /llms.txt. Done — your site has an index.
Then keep it true. Every page you add, rename, retitle, or delete is another hand edit, and a stale index misdirects the exact readers it was written for. The file is easy; the maintenance is the job. That is the part worth automating.
Adoption
Where adoption stands, honestly.
Publishing is real: documentation-heavy platforms — Stripe, Cloudflare, and Anthropic among the visible ones — serve llms.txt today, and public surveys put adoption near one domain in ten. Consumption is a different story: no major AI vendor has committed to reading llms.txt in production, and Google states the file has no effect on Search.
To be plain about it: llms.txt is a bet on where AI reading is heading, not a switch that changes anything today. It costs one file — and the Markdown copies it links are served at direct URLs an agent can fetch either way, index or no index.
With Seogence
Generated from your pages. Kept current. Served on any host.
Add the Seogence tag once and the Knowledge Layer builds a per-domain llms.txt from your pages’ stored titles and descriptions — every entry linking that page’s clean Markdown copy. It is served over plain HTTP, works on WordPress, Shopify, Wix, Next.js or plain HTML with no server changes, and it re-reads itself when your content changes, by content hash, so the index does not go stale.
Ours is live — generated by the same pipeline, listing our own pages the way it will list yours.
# Read our own index — generated and served by the same pipeline
curl -L https://seogence.com/llms.txtRoot-domain recipes for every host and the full contract are in the Knowledge Layer reference. Curious who fetches it? Meet the AI crawlers. And the index is check 3 of the AI readability checklist.
Questions
Common questions.
What is llms.txt?
A plain-Markdown file served at /llms.txt that lists a site’s important pages, each with a one-line description, so AI assistants get a curated index instead of guessing from raw HTML. It was proposed by Answer.AI in September 2024 and the format is specified at llmstxt.org.
Do AI systems actually read llms.txt today?
Publishing is real; consumption is not confirmed. No major AI vendor has committed to reading llms.txt in production, and Google states the file has no effect on Search. It is a low-cost, forward-compatible index — and the Markdown pages it links are served at direct URLs an agent can fetch either way.
Does llms.txt replace robots.txt or sitemap.xml?
No. robots.txt controls which crawlers may access your site, sitemap.xml lists every URL for search engines, and llms.txt curates what matters for AI readers. The three do different jobs and work side by side.
How do I add llms.txt to my site?
By hand: create the file, list your key pages with one-line descriptions, serve it at /llms.txt, and edit it every time content changes. Or add the Seogence tag once — the Knowledge Layer generates your index from your pages and re-reads itself when they change.
Try it
See your page the way an AI reader gets it.
Enter any live page. Seogence analyzes it and shows the clean Markdown an AI reader would get — the same content your llms.txt would link.