Most major sites now publish an index for AI. Almost none serve anything to read.

The findings

Four signals, 27 reachable sites.

Three of the 30 selected sites did not return a readable homepage to our scanner and are excluded from every denominator, which is why n = 27.

The headline

The index-without-content gap.

The distance between those first and third rows is the whole story. Fifteen sites have published a table of contents for AI readers. Two of them have chapters those readers can actually open — the rest still serve only HTML built for browsers, so the index points at pages a no-browser reader has to reconstruct for itself.

That is not carelessness. Publishing an llms.txt is one file you write once; serving a readable copy of every page is a pipeline that has to stay current as content changes. The easy half has been adopted. The half that takes infrastructure has not.

By category

Publishers write for readers. Just not these ones.

The sharpest split in the data: every media and publishing site we reached had structured data, and not one served an llms.txt. Newsrooms have two decades of SEO practice and have adopted the machine-readable signal that search rewards — the new one has no such payoff yet, so it has not travelled. Developer-tooling companies show the mirror image: the highest llms.txt adoption in the sample, and the lowest structured data.

Access

Most sites have not decided about AI crawlers at all.

Of the 27 sites, 19 had no GPTBot-specific rule in robots.txt — the default, which is access. 4 blocked it outright and 4 named it to allow it. So roughly seven in ten major brands have not made an explicit decision about the crawler that has been the most discussed in the industry for two years. Whatever a site decides, the roster is worth reading first.

Method

What we did, and what it cannot tell you.

On 24 August 2026 we requested three URLs per site — the homepage, /llms.txt and /robots.txt — with an identified user agent, and recorded only what the responses directly show. An llms.txt counts only if it is served as text, opens with a heading, and contains link lines, so a soft-404 HTML page does not score.

The sample is not the web. These are 30 well-known brands chosen by hand across five categories, deliberately weighted toward tech-forward companies. That weighting is why we see 56% llms.txt adoption while published large-sample surveys of the web report roughly one domain in ten. Both are true: adoption is concentrated among technical teams. Read this as a snapshot of what leading brands do, never as a projection of the web.

Two further limits worth stating. A homepage is one page, and a site can be readable there and opaque three clicks deeper. And a low text count on an application shell is expected rather than damning — which is one reason we name no site for a negative result.

Check our work

Run it yourself, on any site.

The whole benchmark is four commands. Point them at any domain — ours included — and you will get the same signals we recorded.

# The llms.txt check, for any site
curl -sL https://example.com/llms.txt | head -5

# The structured-data check
curl -s https://example.com/ | grep -c 'application/ld+json'

# The machine-copy discovery check
curl -s https://example.com/ | grep -i 'rel="alternate".*text/markdown'

# What the AI crawlers are told
curl -s https://example.com/robots.txt | grep -A3 -i 'gptbot'

Credit where due

The sites already publishing an index.

These served a valid llms.txt when we looked. We name the sites that pass and leave the rest aggregate — a single measurement is a snapshot, and a snapshot is a fair basis for credit but a poor basis for blame.

• stripe.com
• vercel.com
• cloudflare.com
• github.com
• notion.com
• linear.app
• supabase.com
• shopify.com
• allbirds.com
• warbyparker.com
• hubspot.com
• squarespace.com
• wix.com
• mailchimp.com
• replicate.com

Two of them — cloudflare.com and supabase.com — also declared a machine-readable copy of the page. They are the 7%.

Where we stand

We run the same check on ourselves.

It would be a poor benchmark that exempted its author. Seogence serves its own Knowledge Layer: an llms.txt index at our root, a clean Markdown copy of each page at a direct URL, and the discovery link that points readers at it — the same three signals measured above, generated by the pipeline we sell rather than written by hand. Run the commands against seogence.com and you will see them.

That is also the honest limit of this research: we can measure what a site serves, and we do. What any AI system then does with what it reads is its own business, and no benchmark here claims otherwise.

References

Sources.

1. The llms.txt specification, llmstxt.orgProposed by Answer.AI in September 2024. The format this benchmark tests against.
2. Google: llms.txt has no effect on SearchReported from Google’s own statements — the reason we call publishing a bet, not a lever.
3. Vercel and MERJ, joint analysis of AI crawler behaviourFound no JavaScript execution by the major AI crawlers across hundreds of millions of fetches. Cited by name; see their published write-ups.
4. Seogence AI Readability Benchmark — raw data and scannerThe 30-site scan behind every number on this page, kept with the code so the results can be re-run and checked.

Questions

Common questions.