Most AI crawlers check robots.txt. A few say the rule does not apply to them
Mostly, but with a documented gap. Anthropic states all three of its bots honor robots.txt across every use. OpenAI and Perplexity both say a real-time fetch, one triggered the instant a person asks their assistant a direct question, may not be bound by the file at all, because they classify it as something other than ordinary crawling.
Short answer
Robots.txt is a voluntary standard, not an enforceable rule, so compliance depends on what each company chooses to build. Bots that crawl on their own schedule, to index a page for search or to gather training data, are generally the ones most likely to check and follow it. Bots that fetch a single page in real time because a person is waiting on an answer are the documented exception, and two of the largest AI companies say so in their own crawler documentation.
Robots.txt was built as a request, not a lock
The robots exclusion protocol dates to 1994, years before any AI crawler existed. It works by convention: a bot fetches the file, reads which paths it is asked to avoid, and decides for itself whether to comply. Nothing in the standard can force that decision, which is why it has always depended on the reputation and self-interest of whoever operates the bot.
That worked reasonably well when the bots in question were run by a small number of search engines with a shared interest in staying welcome on the open web. AI companies inherited the same voluntary system, but arrived with a wider mix of bot types and a new use case, real-time fetches, that does not map cleanly onto the crawl-and-index model the protocol was designed around.
What the companies themselves say about compliance
Anthropic's documentation is the most direct: it states that all three of its bots respect robots.txt, without carving out an exception for live use. OpenAI takes a narrower position. Its own crawler documentation says ChatGPT-User visits a page when a person asks a live question that needs that specific page, and that robots.txt rules may not apply to that fetch, because it is triggered by an individual user in the moment rather than run as a background crawl. Perplexity states a similar position for Perplexity-User, for the same reason.
The distinction the companies are drawing is between a crawl, which happens on the bot's own schedule and can wait, and a real-time fetch, which happens because a specific person is waiting on a specific answer right now. Their position is that the second case is not really crawling at all, so the rule built for the first case does not automatically extend to it.
15%
of identified AI page fetchers reached URLs that European sites had explicitly marked disallowed in robots.txt, though the figure was uneven: ChatGPT-User, Bytespider and Youbot each reached disallowed pages on nearly half of the sites that had listed them.
TollBit, State of the Bots research
Robots.txt asks a bot to stay out. Whether it listens is still each company's own choice.
What this means for a website's own robots.txt file
Maintaining robots.txt is still worth doing. Most named AI crawlers, including the training-focused and search-indexing bots that generate the bulk of AI-relevant traffic, check and follow it, and Anthropic's bots follow it without the real-time exception the other two companies carve out. Removing the file, or leaving it misconfigured, gives up control over the majority of bots that do respect it in exchange for no real gain against the minority that might not.
What robots.txt cannot do is guarantee a page never reaches an AI answer. A disallow rule stops a well-behaved crawler from indexing a page in advance. It does not stop a real-time fetch bot that classifies itself outside the rule, and it cannot stop a person pasting a page's own text into a chat window themselves. Anyone deciding what to disallow should treat the file as the correct first layer of control, not the only one, and check the actual named bots against a current list rather than assuming a single blanket rule covers all of them.
Common questions
Do AI crawlers respect robots.txt?
Mostly, but not uniformly, and not always. Anthropic states that all three of its bots honor robots.txt. OpenAI and Perplexity both say the file may not apply to a real-time fetch triggered by a person asking their assistant a direct question, because that fetch is not a crawl in the traditional sense.
Is robots.txt legally binding on AI companies?
No. It is a voluntary standard that predates AI crawlers by decades. A well-behaved bot checks the file and follows it; nothing in the protocol itself forces compliance, and a company can state its own exceptions, which is exactly what has happened with real-time fetch bots.
What is the difference between a crawl and a real-time fetch?
A crawl happens on the bot's own schedule, discovering and indexing pages without a specific person waiting on the result. A real-time fetch happens because a person typed a question into a chat product right now and the assistant needs a specific page to answer it. OpenAI and Perplexity treat the second case differently from the first.
Does disallowing a page in robots.txt guarantee an AI system will never read it?
No. It stops well-behaved crawling bots that check and follow the file. It does not stop a real-time fetch bot that treats the rule as inapplicable, and it cannot stop a person pasting the page's own content into a chat window themselves.
Should a business bother maintaining robots.txt for AI bots at all?
Yes. Most named AI crawlers, including the training and search-indexing bots, do check and follow it, and Anthropic's bots follow it across all use cases. Skipping it because a small number of real-time fetchers may not honor it gives up control over the majority that do.
AI-readable websites
The full standard: what Reidify builds into every site by default.
AI crawler user agents
The named bots behind GPTBot, ClaudeBot and Perplexity, listed individually.
What is a mixed-use crawler
Why blocking one bot identity can block more uses than intended.
Do AI crawlers read JavaScript
What a crawler actually sees once robots.txt lets it through.
See which AI crawlers can actually reach your site, and which pages they can read once they do.