One robots.txt Line Now Opts You Out of AI Training (Almost)

On 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they will adopt a shared robots.txt policy token, __usage_ai-training, that lets a site express one AI training preference to several providers at once (The Verge). The token is defined in an IETF Internet-Draft by Anthropic's Kevin Kelley and is meant to replace the work of maintaining a separate rule for every crawler (IETF draft). It is voluntary, it is not a crawling block, and the four providers are rolling it out on different schedules.

TL;DR

• OpenAI, Google, Microsoft and Anthropic will honour the shared __usage_ai-training robots.txt token.
• Anthropic has adopted it, OpenAI aims to implement it in the coming months, and Google and Microsoft plan to support it next year.
• The token states a training preference; it does not block crawling.
• Wildcards do not match it, so a User-Agent: * rule expresses nothing about training.
• Blocking the wrong crawler can cost you ChatGPT search inclusion or Gemini grounding.

Disclosure: ZeroRaised publishes this analysis and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 31 Oct 2026; ZeroRaised has no commercial relationship with the companies named.

A shared AI training preference signal is a robots.txt policy token that lets a site state one preference about use of its content for generative AI training to every provider that has agreed to honour it.

What problem does the shared token solve?#

Until now, operators had to identify individual crawlers and maintain separate rules for each provider (OpenAI). That maintenance tax scales badly as more companies launch more crawlers. The shared token is voluntary and lets operators communicate one preference to multiple participating providers at once (OpenAI).

It is not a private standard invented by one vendor. OpenAI says it draws on the IETF AI Preferences working group and encourages other AI providers to honour the signal (OpenAI). Google says the proposal is consistent with the EU Code of Practice for general purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary method for machine-readable preferences (Google Search Central).

How does the token actually work?#

The draft defines the token __usage_ai-training, with two leading underscores, for use in the robots.txt User-Agent line. It introduces no new directives and no new grammar (IETF draft). The draft's own example looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against AI training for all paths except the press kit (IETF draft). Three boundaries matter. Wildcard groups must not be read as matching the token, so if no group names it, no preference has been expressed (IETF draft). The token is not a crawling directive, and the crawler rules for each bot remain independent (IETF draft). And AI training is defined narrowly, as "Using an asset to modify the learned parameters of a generative AI model", which excludes training for purely non-generative tasks such as classification, ranking and scoring (IETF draft).

The draft also says specific expressions should take precedence over general ones, and that private agreements may override preferences (IETF draft). It is a work in progress that expires on 3 May 2027 (IETF draft).

When will each provider honour it?#

Anthropic has already adopted the token (The Verge). OpenAI aims to implement it "in the coming months" and will share implementation details later (OpenAI). Google and Microsoft plan to support it next year, and Google's crawler documentation points to early 2027 (The Verge, Google crawler docs).

OpenAI says the signal will not replace existing controls, and existing crawler controls continue to apply (OpenAI). That matters if you already disallow GPTBot. Microsoft will offer granular opt-outs that separate training from grounding, and sites stay in Bing search results (The Verge). Meta's Andy Stone signalled support: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step" (The Verge).

What is the tradeoff between training and AI search visibility?#

This is the part founders get wrong. Google says that if a site uses the policy token to opt out of training and Google-Extended is not explicitly specified, Google plans to treat that as if the publisher had disallowed Google-Extended (Google Search Central). Company-specific controls keep working alongside it (Google Search Central). Sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps (Google Search Central).

On the OpenAI side, OAI-SearchBot is the crawler used to surface websites in ChatGPT search. Sites that opt out will not be shown in ChatGPT search answers, though they may still appear as navigational links (OpenAI crawlers). So a training opt-out and an AI search opt-out are different decisions, even when they sit in the same file. The guide to blocking AI crawlers walks through the per-bot rules, and the Google AI Mode ad analysis covers the paid side of the same visibility shift.

What should you put in robots.txt today?#

If you want AI search visibility but no training, do two things: keep explicit per-bot rules for the crawlers you care about now, and add the shared token as the longer-term preference. Do not rely on a wildcard, because the draft is explicit that a User-Agent: * group cannot express the preference (IETF draft).

For your own site, that means a group naming the token, plus groups for each product token whose crawling you want to limit, because crawler rules are independent (IETF draft). The __usage_ai-training example above mirrors the draft and is a safe starting point. The llms.txt guide covers the complementary file, and the AI search optimization guide covers the content side of the same problem.

What this means for bootstrapped founders#

• Treat it as a preference, not a lock. The token is voluntary. Providers opt in, and two of the four named in the Verge report plan to support it next year (The Verge).
• Decide per surface. If ChatGPT search sends you buyers, allow OAI-SearchBot while you disallow GPTBot (OpenAI crawlers). Google-Extended is a harder call, because one token covers both Gemini training and grounding in Gemini Apps, so allowing it for grounding also allows training (Google crawler docs).
• Write the rules down. Keep robots.txt in the repo and revisit it when a provider changes its docs.
• Watch the 2027 deadline. Google's docs point to early 2027 and the draft expires on 3 May 2027, so the text will move (Google crawler docs, IETF draft).
• Do not over-block. A wildcard disallow is easy to write and hard to debug when your pages stop appearing in answers.

Where ZeroRaised fits#

ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have a product people can try, you can submit your product. The pricing page explains the current options, and there are no promises of traffic or results.

If you want more analysis like this, join the ZeroRaised newsletter.

Sources#

• OpenAI: Making it easier for websites to express AI training preferences
• Google Search Central: AI training control
• IETF: AI training preference signal draft
• The Verge: OpenAI, Google, Microsoft and Anthropic adopt a shared robots.txt standard
• OpenAI: Overview of OpenAI crawlers
• Google: Google's common crawlers