A mixed-use crawler fetches a page for more than one reason, under one identity
A mixed-use crawler is a bot that visits a website to do more than one job at once, typically indexing it for search and pulling it into AI model training, without the site ever knowing which use is happening on that particular crawl. That single identity is what makes it hard to allow one use and refuse the other.
Definition
A mixed-use crawler is a single bot identity that a company uses for more than one purpose, most commonly crawling for a search index and crawling to train or fine-tune an AI model. Because both uses share one user agent, a site owner cannot tell, from the crawl alone, which job the bot is doing on any given request, and blocking the bot blocks every use it carries.
Why crawlers started blending purposes
Bots used to be simple to reason about: a search crawler built an index, a scraper copied content, and the two rarely shared an identity. That changed once the same companies started operating both a search product and an AI product. Google, Microsoft and Apple each run search engines and AI systems side by side, and their crawlers now do both jobs.
The result is a bot that a site cannot cleanly categorize. Cloudflare, which classifies bot behavior across a large share of the web's traffic, treats Googlebot, Bingbot and Applebot as crawlers that carry both a search behavior and a training behavior under one name.
The three behaviors a crawler can carry
Cloudflare's classification splits crawler behavior into three: Search, crawling to build a search index; Training, crawling to train or fine-tune a model; and Agent, a bot fetching a page on behalf of a person using an AI assistant. A single crawler can carry more than one of these at once, and when it carries two or more under one identity, it is mixed-use.
That is not a small share of traffic. On Cloudflare's own network, mixed-use crawlers made up more than a third of verified crawler activity, which is why the distinction has moved from a footnote to something site owners are being asked to actively manage.
36.6%
of verified crawler traffic on Cloudflare's network was classified as mixed-use, a single crawler identity carrying more than one behavior such as search and training at once.
Cloudflare, Accountable mixed-use AI crawlers
A crawler that indexes a page for search and feeds it into model training can look identical from the server log.
What the split means for a site owner
The practical problem a mixed-use crawler creates is a forced choice. Blocking it to stop one use, usually AI training, also stops the use most businesses want, being found in search and read for AI citation. Allowing it to keep search visibility means accepting the training crawl too, with no way to separate them through a plain robots.txt disallow rule.
The direction the industry is moving is toward controls that let a site declare each behavior separately rather than allowing or blocking a crawler wholesale. Cloudflare introduced a setting specifically to let a site keep search visibility while declining training, and it matters to check whether a hosting or CDN provider offers an equivalent before reaching for a blanket block. Read the list of AI crawler user agents to see which bots this actually affects, and whether those crawlers can even read the page it is fetching in the first place.
The underlying shift is not about one company's policy, it is a structural fact of the current web: crawler identity and crawler purpose are no longer the same thing, and a site that wants to be read by AI systems while controlling how its content is used needs to think in terms of behavior, not just bot name.
Common questions
What is a mixed-use crawler?
A bot that fetches a page for more than one purpose under a single identity, most often search indexing and AI model training at once, without telling the site which one it is doing on a given visit.
Is Googlebot a mixed-use crawler?
Cloudflare classifies Googlebot, Bingbot and Applebot as both search and training crawlers. Their owners run consumer search products and AI products from the same company, so the two purposes ride the same crawl.
Why does the distinction matter to a site owner?
Under a single mixed identity, blocking the use you do not want, usually training, also blocks the use you do want, usually being found in search. Separating the two lets a site keep one and refuse the other.
Does blocking a mixed-use crawler hurt AI visibility?
It can. A crawler blocked entirely stops indexing the page for search and stops it being read for citation in an AI answer. The fix is not to block indiscriminately, it is to use a control that separates search from training where one exists.
How is this different from an AI crawler user agent list?
A user agent list identifies which bots exist. Mixed-use is about what a single bot is doing on a given crawl, which a static list of names cannot show on its own.
AI-readable websites
The hub this crawler distinction sits inside.
AI crawler user agents
The named bots behind GPTBot, ClaudeBot and Perplexity.
Do AI crawlers read JavaScript
What a crawler actually sees once it fetches a page.
Crawlability vs indexability
The two separate questions a mixed-use crawler answers at once.
Find out which crawlers can actually read your website, before deciding which to block.