AI crawlers

The web has new readers. These are their names.

ChatGPT, Claude and Perplexity each send their own crawlers — one set to train models, one to build search indexes, one to fetch a page the moment a user asks about it. Gemini reads through Googlebot. Each announces itself with a user-agent string. Each decides, from what your server returns, what it can read.

Get started See what AI can read on your site

The three jobs

Most AI visits do one of three jobs.

Model trainingBulk crawls that collect text to train future models. GPTBot, ClaudeBot, CCBot. Slow-burn: what they read shapes what models know later.
Search indexCrawls that build the retrieval index behind AI search. OAI-SearchBot, Claude-SearchBot, PerplexityBot. This is where being readable can put your pages in front of people asking questions.
Live user fetchA single page, fetched right now, because a person asked an assistant about it. ChatGPT-User, Claude-User, Perplexity-User. Block these and the assistant reads nothing when someone asks.

The jobs deserve separate decisions. A robots.txt line that treats them as one thing is deciding all three at once. A few crawlers, like Meta-ExternalAgent, do more than one job. Those you allow or block whole.

The roster

Who is fetching, and why.

User agent	Operator	Job	What it does
`GPTBot`	OpenAI	Model training	Collects text for model training.
`OAI-SearchBot`	OpenAI	Search index	Builds the index behind ChatGPT search.
`ChatGPT-User`	OpenAI	Live user fetch	Fetches a page when a ChatGPT user asks about it.
`ClaudeBot`	Anthropic	Model training	Collects text for model training.
`Claude-SearchBot`	Anthropic	Search index	Builds the index behind Claude’s search.
`Claude-User`	Anthropic	Live user fetch	Fetches a page when a Claude user asks about it.
`Google-Extended`	Google	More than one	A robots.txt token, not a visiting bot: it tells Google whether crawled content may train Gemini or ground its answers.
`PerplexityBot`	Perplexity	Search index	Builds Perplexity’s answer index.
`Perplexity-User`	Perplexity	Live user fetch	Fetches a page when a Perplexity user asks about it.
`Applebot-Extended`	Apple	Model training	A robots.txt token controlling whether Applebot’s crawl may train Apple’s models.
`Meta-ExternalAgent`	Meta	More than one	Crawls to train Meta’s AI models and to index content.
`Amazonbot`	Amazon	More than one	Improves Amazon’s products and may train its AI models.
`CCBot`	Common Crawl	Model training	Builds the open corpus many training sets start from.
`Bytespider`	ByteDance	Search index	Crawls for ByteDance’s Toutiao search.

Two entries are controls, not visitors: Google-Extended and Applebot-Extended never fetch a page themselves — they are robots.txt tokens the regular Googlebot and Applebot crawls check for permission to use what they collect.

The divide

None of them runs your JavaScript.

These readers do not open a browser. Joint public analyses by Vercel and MERJ of hundreds of millions of fetches found no JavaScript execution by any major AI crawler — GPTBot, ClaudeBot, PerplexityBot included. They fetch the raw HTML, read what is in it, and move on. No rendering, no waiting, no second pass. Google is the exception: Gemini leans on Googlebot’s rendering infrastructure.

The consequence is blunt. Content that exists only after JavaScript runs — a client-rendered product list, a script-injected description — is not there for them. Your page can be perfect in a browser and near-empty to the readers deciding what to say about it. To find what is invisible on your own site and fix it, see JavaScript rendering & AI crawlers.

Access

robots.txt: make it a decision, not a default.

Every crawler above announces itself, and the major operators document that their crawlers honour robots.txt. That puts the choice on your side: allow the search and live-fetch agents so assistants can read your pages when people ask, and treat model training as the separate policy question it is.

One rule to know first: a named group replaces User-agent: * entirely. The moment you give OAI-SearchBot its own group, it stops reading the wildcard — so any Disallow you had there no longer applies to it. If you skip the repeat, you hand that crawler your login page.

robots.txt

# Default rules — every crawler without its own group uses these.
User-agent: *
Allow: /
Disallow: /admin
Disallow: /login

# AI search and live-fetch crawlers.
# Each group REPLACES the wildcard above for that agent,
# so the Disallow lines must be repeated here.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin
Disallow: /login

User-agent: ChatGPT-User
Allow: /
Disallow: /admin
Disallow: /login

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /login

User-agent: Claude-User
Allow: /
Disallow: /admin
Disallow: /login

User-agent: PerplexityBot
Allow: /
Disallow: /admin
Disallow: /login

# Training crawlers — your call. To opt out of model training
# (Google-Extended also opts your pages out of Gemini grounding):
# User-agent: GPTBot
# Disallow: /
#
# User-agent: Google-Extended
# Disallow: /

Seogence-served Markdown also carries a Content-Signal response header stating what the content may be used for — see Markdown for AI agents.

With Seogence

Being allowed is not being readable.

robots.txt opens the door. The page still has to be legible to a reader that will not run your scripts. That is the job Seogence does: one tag gives each page a clean Markdown copy — served over plain HTTP at a direct URL, no JavaScript required to read it, on any host — plus a per-domain llms.txt index so the whole site is discoverable from one file.

Be readable, not just crawlable — start with the six-point AI readability checklist.

Questions

Common questions.

Which AI crawlers matter in 2026?

OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic’s ClaudeBot, Claude-SearchBot and Claude-User; Perplexity’s PerplexityBot and Perplexity-User; Google’s Google-Extended control; plus Meta-ExternalAgent, Amazonbot, Applebot-Extended, CCBot and Bytespider. Most do one of three jobs: model training, search indexing, or fetching a page live because a user asked.

Do AI crawlers execute JavaScript?

Public analyses of hundreds of millions of fetches have found no JavaScript execution by the major AI crawlers — they read the raw HTML, once, and move on. Google’s infrastructure is the notable exception. Content that only appears after scripts run is simply not there for the rest.

Should I block AI crawlers in robots.txt?

Decide per job, not per vendor. Blocking training crawlers like GPTBot or ClaudeBot is a policy choice about model training. Blocking search and live-fetch agents also stops assistants from reading your pages at the moment a user asks about them — a different trade entirely.

How do I see what an AI crawler can read on my page?

Run the page through the Seogence analyzer. It fetches your page and shows the clean Markdown copy and structured data an AI reader could work with — no account needed.

Try it

See what these readers get from your page.

Enter any live page. Seogence analyzes it and shows the clean Markdown copy — with the structured data it would add — so you can see what a reader without a browser has to work with.