GPTBot, ClaudeBot, PerplexityBot: the crawlers actually reading your site
AI crawler user agents are the names systems like ChatGPT, Claude and Perplexity give themselves when they fetch a webpage, the string a server sees in its logs and a robots.txt file can target directly. Some train models, others power live search and citations, and knowing which is which is the difference between blocking training and accidentally blocking citations too.
Definition
An AI crawler user agent is the identifying string a bot sends with every request it makes, for example GPTBot or ClaudeBot, that lets server logs and a robots.txt file recognise which system is making the request. Major AI labs each operate more than one named agent: typically one for training their models, and a separate one for the live retrieval that powers citations in chat answers.
What a user agent string actually does
Every HTTP request carries a user agent header, a short piece of text identifying the software making the request. Browsers send one, and so does every bot: a search engine crawler, an uptime monitor, or an AI system fetching a page to train on or answer from.
A robots.txt file works by matching that string. A rule written under User-agent: GPTBot applies only to requests identifying themselves as GPTBot, leaving every other crawler, AI or otherwise, unaffected. That specificity is exactly why the individual names matter.
The major AI crawlers worth knowing
The established, major crawlers cluster around a handful of labs, and most of those labs run more than one agent for different purposes.
| Crawler | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Fetches content to help train OpenAI's models. |
| OAI-SearchBot | OpenAI | Fetches pages live to power ChatGPT's search and citation features. |
| ChatGPT-User | OpenAI | Fetches a specific page on demand when a user asks ChatGPT to browse it. |
| ClaudeBot | Anthropic | Fetches content to help train Claude's models. |
| Claude-SearchBot | Anthropic | Fetches pages to support Claude's search and citation features. |
| PerplexityBot | Perplexity | Crawls and indexes pages that feed Perplexity's answers and citations. |
| Google-Extended | Controls whether content can be used for Gemini and AI features, separate from classic Search indexing. | |
| Applebot-Extended | Apple | Controls whether content can be used to help train Apple Intelligence. |
| CCBot | Common Crawl | Builds the open web archive several AI labs train their models on. |
Why the same company runs more than one crawler
The split is deliberate. GPTBot exists to gather content for training; OAI-SearchBot and ChatGPT-User exist to fetch a page in real time so ChatGPT can answer a question or cite a source right now. Blocking one does not automatically block the other, so a robots.txt file that disallows only GPTBot can still leave a site fully eligible for citation in ChatGPT's live answers.
The practical consequence: a site deciding to keep its content out of AI training data while staying visible to AI search needs two separate rules, not one, written under the two different user agent names.
Do these crawlers execute JavaScript
Generally, no. Vercel instrumented AI crawler traffic across its network and found that the crawlers behind ChatGPT and Claude do request a page's JavaScript files, ChatGPT on 11.50% of requests and Claude on 23.84%, but never actually run them. Anything that only appears after a page hydrates in the browser is invisible to them.
That has a direct bearing on how these crawlers should be treated: a page that reads fine to a person can still hand most of these bots an empty document if its content depends on client-side rendering. See do AI crawlers read JavaScript for the fuller answer.
23.84%
ClaudeBot share of AI crawler requests measured across Vercel-hosted sites, against 11.50% for ChatGPT. None of these crawlers execute JavaScript.
Vercel, AI crawler study
Every AI company runs more than one crawler, and they do not all carry the same purpose or the same permissions.
Common questions
What is an AI crawler user agent?
The identifying string a bot sends with every request it makes, for example GPTBot or ClaudeBot. Server logs record it, and a robots.txt file can target it directly, which is what lets a site allow or block one AI system without affecting the others.
Does blocking GPTBot in robots.txt stop a business from appearing in ChatGPT answers?
No. GPTBot is OpenAI's training crawler. ChatGPT's live search and citation features run through separate crawlers, OAI-SearchBot and ChatGPT-User, so blocking GPTBot alone stops training use without affecting citation eligibility.
Do AI crawlers actually respect robots.txt?
The major labs state that their crawlers do. There is no technical mechanism forcing compliance, robots.txt is a published request a crawler is trusted to honour rather than an access control a server enforces.
Will this exact list of crawlers stay accurate?
The underlying pattern is stable: every major AI system identifies itself with a distinct named crawler, often more than one. The specific names can change as labs launch new agents or retire old ones, so treat this as the established, major crawlers rather than a real-time registry.
How can a business see which of these crawlers are actually visiting its site?
Server access logs record the user agent string on every request. Filtering those logs for the names above shows exactly which AI crawlers have fetched the site, and how often, without needing any special tooling.
AI-readable websites
The structural discipline these crawlers actually reward.
Do AI crawlers read JavaScript
What these same crawlers do, and do not, execute on a page.
Crawlability vs indexability
The two separate stages a page has to clear before it can be cited.
What is llms.txt
A separate, voluntary file some of these crawlers choose to read.
Check your own server logs against this list, or let the audit do it for you.