How to Block AI Crawlers Without Vanishing From AI Search
To block AI crawlers without vanishing from AI search, write robots.txt rules per product token: disallow training crawlers such as GPTBot and ClaudeBot, and allow search crawlers such as OAI-SearchBot and Claude-SearchBot. A blanket block is the mistake to avoid.
Key takeaways
- AI crawlers are not one thing: training, search and user-initiated fetches are separate product tokens and separate decisions.
- GPTBot and ClaudeBot are the training crawlers; OAI-SearchBot and Claude-SearchBot are the search crawlers; ChatGPT-User and Claude-User fetch pages for a user.
- Google-Extended is a robots.txt control with no separate HTTP user agent, and it also governs grounding in Gemini Apps.
- Disallowing a crawler does not automatically stop it, because individual rules only bind the providers that honour them.
- A shared AI training token is coming, but per-bot groups still do the work today.
On this page 10
- What is the difference between a training crawler and a search crawler?
- Which OpenAI bot does what?
- Which Anthropic bot does what?
- What does Google-Extended control?
- What is the new shared training token?
- What robots.txt should a founder use?
- How do you check that the rules work?
- A practical checklist
- Where ZeroRaised fits
- Sources
AI crawlers are not one product. Training, search and user-initiated fetches are separate robots.txt tokens. To block training without vanishing from AI search, disallow GPTBot and ClaudeBot, allow OAI-SearchBot and Claude-SearchBot, and decide separately about Google-Extended. A blanket Disallow: / is the mistake that costs you answers.
TL;DR
- Split the problem: training crawler, search crawler and user-triggered fetch are different tokens.
- Disallow GPTBot and ClaudeBot if you do not want your content used for training.
- Allow OAI-SearchBot and Claude-SearchBot if you want to appear in ChatGPT search and Claude search results.
- Google-Extended controls Gemini training and grounding, not Google Search inclusion.
- Keep a group per bot; wildcards express less than you think.
Disclosure: ZeroRaised publishes this guide and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 31 Oct 2026; ZeroRaised has no commercial relationship with the companies named.
AI crawler segmentation is the practice of writing separate robots.txt user-agent groups for each AI product token, so training access, search inclusion and user-initiated fetches can be controlled independently.
What is the difference between a training crawler and a search crawler?#
A training crawler collects content for model training. A search crawler collects content so an AI product can answer a question and cite a page (OpenAI crawlers). Blocking one does not block the other unless the provider says so. OpenAI states each setting is independent and gives the example of allowing OAI-SearchBot while disallowing GPTBot (OpenAI crawlers).
Which OpenAI bot does what?#
The OpenAI crawler docs list four product tokens worth knowing (OpenAI crawlers).
| Bot | What it does | What blocking it costs |
|---|---|---|
| GPTBot | Crawls content that may be used to train generative AI foundation models; disallowing it indicates the content should not be used for training | Nothing for ChatGPT search |
| OAI-SearchBot | Surfaces websites in ChatGPT search; opted-out sites will not be shown in ChatGPT search answers, though they may still appear as navigational links | ChatGPT search visibility |
| ChatGPT-User | Used for user-initiated actions; not used for automatic crawling, and OpenAI says robots.txt rules may not apply because the actions are user-initiated | Little, but the rule may be ignored |
| OAI-AdsBot | Only visits pages submitted as ChatGPT ads; that data is not used for training | Ads review, if you buy ChatGPT ads |
OpenAI says robots.txt changes can take about 24 hours to take effect for search, and it publishes its IP ranges, such as the GPTBot list (OpenAI crawlers).
Which Anthropic bot does what?#
Anthropic lists three bots (Anthropic Privacy Center). ClaudeBot collects content that could contribute to training, and restricting it signals that future material should be excluded from training datasets. Claude-User fetches pages when a user asks Claude something. Claude-SearchBot crawls to improve search result quality.
Anthropic supports the non-standard Crawl-delay extension if you want to slow a bot rather than block it (Anthropic Privacy Center). To block across the site you use a group for ClaudeBot with Disallow: /, repeated for every subdomain, because each host serves its own robots.txt (Anthropic Privacy Center). Blocking IP addresses may not work reliably, because it stops Anthropic from reading your robots.txt at all (Anthropic Privacy Center).
What does Google-Extended control?#
Google-Extended is unusual: it has no separate HTTP user agent, so crawling still uses Google's existing user agents and the robots.txt token acts as the control (Google crawler docs). It manages whether your content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google crawler docs).
The good part is that Google-Extended does not affect inclusion in Google Search and is not a ranking signal (Google crawler docs). The awkward part is that it bundles training and grounding into one switch. If Gemini grounding sends you traffic, disallowing Google-Extended removes that eligibility too.
What is the new shared training token?#
Google intends to support the AIPREF shared training signal, written as user-agent: __usage_ai-training, in early 2027 (Google crawler docs). OpenAI aims to implement the shared signal "in the coming months" and will not replace existing controls (OpenAI). Google has described the same shared policy token, based on IETF AIPREF work, as a control that builds on Google-Extended (Google Search Central).
The token is defined in an IETF Internet-Draft by Anthropic's Kevin Kelley and introduces no new directives: it is used in the User-Agent line (IETF draft). It is not a crawling directive, crawler rules are independent, and wildcard groups must not be read as matching it (IETF draft). The analysis of the shared opt-out token covers the rollout in more detail.
What robots.txt should a founder use?#
For the common case, AI search visibility with no training, the shape is simple. One group per bot, no wildcard pretending to be a policy.
User-Agent: GPTBot
Disallow: /
User-Agent: ClaudeBot
Disallow: /
User-Agent: OAI-SearchBot
Allow: /
User-Agent: Claude-SearchBot
Allow: /
User-Agent: Google-Extended
Disallow: /
User-Agent: *
Allow: /
Swap the Google-Extended block for Allow: / if Gemini grounding matters more to you than Gemini training. To express the training preference as well, add a group naming the token, which mirrors the draft's example (IETF draft):
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Keep the file plain: one group per agent. The llms.txt guide covers the companion file, and the AI search optimization guide covers the content side.
How do you check that the rules work?#
Read the provider docs you rely on. OpenAI publishes IP ranges and warns that robots.txt changes can take about 24 hours for search (OpenAI crawlers). Anthropic warns that IP blocking may stop it from reading your robots.txt at all (Anthropic Privacy Center).
Fetch robots.txt from every subdomain, confirm it returns a successful response, and check that the group you expect is actually served. A robots.txt that only exists on the apex domain does nothing for docs.example.com.
A practical checklist#
- Decide the goal for each surface: training, ChatGPT search, Gemini grounding, Bing search.
- Map each bot to a preference before you touch the file.
- Write one user-agent group per bot and keep the names exact.
- Do not use a wildcard to express a training preference.
- Keep the file in version control and re-check after a week.
Where ZeroRaised fits#
ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have a product people can try, you can submit your product. The pricing page explains the current options, and there are no promises of traffic or results. The free marketing channels guide covers distribution once your pages are crawlable.
If you want more analysis like this, join the ZeroRaised newsletter.
Sources#
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot is the crawler for content that may be used to train generative AI foundation models. OAI-SearchBot is the crawler used to surface websites in ChatGPT search, and OpenAI treats each setting as independent, so you can allow OAI-SearchBot while disallowing GPTBot.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot collects content that could contribute to training, and restricting it signals that future material should be excluded from training datasets. Claude-User fetches a page when a user asks Claude something, so it is closer to a user visit than a bulk crawl.
Can I block AI training but stay in Google Search?
Yes. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It does control whether your content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI, so blocking it costs you that grounding eligibility.
Does a robots.txt wildcard disallow block all AI crawlers?
It depends on the crawler and the provider. Wildcards are not a reliable way to express a training preference: the IETF draft for the shared token says wildcard groups must not be read as matching the policy token. Write a group per product token instead.
How long do robots.txt changes take to work?
There is no single answer. OpenAI says robots.txt changes can take about 24 hours to take effect for search, and other providers publish their own guidance. Keep the file stable and check the docs rather than assuming an instant change.