Clients ask us two opposite questions. One says “don't let AI train on my content, block everything”, the other asks “why don't we show up in ChatGPT?”. Often both come from the same file: the robots.txt at the root of the site. In this post we explain what each bot does, based on vendor documentation and without making up numbers.
Short Answer
You can block training bots (GPTBot, ClaudeBot, Google-Extended), and doing so does not by itself reduce your search visibility. If you block search and answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot), it becomes harder to appear in the corresponding AI search products. Because each bot has a different purpose, a single “block everything” line can break two jobs at once.
There are three kinds of bots
The crawlers of AI companies don't all do the same job. The distinction that comes out of the documentation is this:
- Training bots: may collect content to train generative AI models.
- Search bots: index pages so your site can be shown in that company's search feature.
- User bots: when someone asks an AI a question, they open and read the page at that moment.
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
According to OpenAI's bot page, GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot signals that your content should not be used for training. OAI-SearchBot works to show sites in ChatGPT's search features, and OpenAI recommends allowing it in robots.txt. Sites that block this bot stay out of ChatGPT search answers and can at most appear as a navigation link.
The third bot is ChatGPT-User. It can visit a page when a user asks ChatGPT a question. OpenAI writes that because these visits are initiated by the user, robots.txt rules may not apply. In other words, robots.txt alone is not a reliable brake for this bot.
A note on timing: OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt update. Don't change the file and expect results the same day.
Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic's support page uses the same three-part structure. ClaudeBot collects web content that may be used for training. Claude-SearchBot analyzes content to improve the quality of search results; if it is blocked, you are not indexed for search and your visibility may drop. Claude-User visits a site when someone asks Claude a question; if it is blocked, your content cannot be retrieved for user questions.
Anthropic also does not recommend blocking by IP address. The reason: an IP block also stops the bot from reading your robots.txt, so your preference may not be seen correctly.
Perplexity and Google-Extended
In Perplexity's documentation, PerplexityBot is defined as the bot that surfaces sites in Perplexity search results, and it is stated that it does not crawl content to train foundation AI models. Perplexity recommends allowing this bot. Perplexity-User visits a page when a user asks a question and, because it acts on the user's request, generally does not follow robots.txt rules.
On the Google side, the confusing name is Google-Extended. According to Google's documentation, it is not a separate user agent but only a product token used in robots.txt. It controls whether your content may be used to train future Gemini models and for grounding. The same page states clearly that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. So blocking it does not hurt your classic Google rankings. I could not find in the relevant documentation how AI Overviews relate to this token, so I won't claim anything definite on that.
A practical robots.txt example
If the goal is “don't use me for training, but let me appear in AI search”, the file looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://alanadiniz.com/sitemap.xmlThis is a template; look at your own situation before copying it verbatim. A site that publishes news or original research may want to shut training off. A business that sells services mostly wants its name to appear in AI answers, and training use is secondary for it. This is not legal or commercial advice, it is a choice, and both options are legitimate.
The limits of robots.txt
robots.txt is not a firewall, it is a note left for bots that comply. Big companies say their own bots comply, but there are third-party scrapers that ignore the rule. For bots that act on a user's request the rule may not apply either. Writing something in the file does not guarantee your content cannot be used in other ways. Content that has already entered training data is not taken back.
One infrastructure addition: in July 2025 Cloudflare announced that it had started blocking AI bots by default on the sites it hosts, and said it planned to ask new domain owners whether to allow AI crawlers (reported by MIT Technology Review). If your site is behind Cloudflare, bots may not even reach your site, regardless of what robots.txt says. If you want to appear in AI, check this setting in the dashboard.
Checklist
- Open your domain's /robots.txt and check for a leftover “Disallow: /” line. It is often left behind by a plugin or a test environment.
- Check whether AI bot blocking is turned on in your CDN or security layer (such as Cloudflare).
- Decide: training off and search on, or everything on?
- Update the file, wait a day, and watch bot requests in your server logs.
- Track whether your brand appears in answers with an AI visibility measurement.
Once bots can read your site, the next step is making what they read well organized. llms.txt is a good second step for that.
FAQ
If I block GPTBot, will I disappear from ChatGPT search?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search features, and the two are managed separately in robots.txt. You can block GPTBot and allow OAI-SearchBot.
If I block Google-Extended, will my Google rankings drop?
According to Google's documentation, no. Google-Extended does not affect inclusion in Search and is not a ranking signal. It only controls whether your content is used to train Gemini models and for grounding.
Does robots.txt definitely stop AI bots?
No. It gives instructions to bots that comply. OpenAI, Anthropic and Perplexity say their bots comply, but some crawlers ignore the rule, and for bots that act on a user's request the rule may not apply.
If you aren't sure whether your site is open to AI bots
Let's review robots.txt, CDN settings and your content structure together and make clear which bots can see you.