On October 30, 2026, Google, Microsoft, OpenAI and Anthropic said they will honor the same robots.txt group, User-agent: __usage_ai-training, as a request not to use a site’s content to train AI models. The line is about training and nothing else. It does not remove the site from Google or Bing search, it is not an order for any bot to stop crawling, and for now none of the four companies fully applies it.

What the new line does, and what it leaves out

It replaces several rules with one. Until now, anyone who wanted their content kept out of training had to name each company: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google. With the shared token, you write the preference once.

The draft that defines the token calls training any use of content to change the parameters of a generative model. Models that only classify, rank or score are excluded. The text also makes clear that the token records a preference, not a crawl instruction: a bot may keep fetching pages as long as it does not train on them.

When does each company start honoring the token?

Each one has its own date. Going by what they published on October 30, 2026:

Anthropic: says it has already started implementing it and that covering all of its services will take some time.
Microsoft: Bing starts reading the new preferences on February 1, 2027, as it crawls pages.
Google: plans to support it in 2027, with no month set, and promised progress updates.
OpenAI: according to trade press, it intends to implement it in the coming months and share details when it does.

The document itself deserves caution. It is an individual IETF draft, written by Kevin Kelley of Anthropic and discussed in the AI preferences working group. It is not a standard. If nobody renews it, it expires on May 3, 2027.

Why the asterisk does not count as a refusal

A User-agent: * group does not stand in for the new token, and the draft says so explicitly. A robots.txt that already blocks everything for everyone still stops crawling, but it records no training preference through the shared channel. For that preference to exist, the name __usage_ai-training has to be written in the file.

The opposite mistake is just as easy. Adding the token does not stop GPTBot or ClaudeBot from crawling. Rules by bot name keep working exactly as before, next to the new one.

Training and answering are still different bots

Mixing up the bot that collects for training with the one that fetches for answers is the costliest error here, and the new line only covers the first. Blocking the trainer is a decision about how content is used. Shutting out the answerer pulls the site out of exactly the place where the client wants to be cited.

OpenAI: GPTBot collects for training; OAI-SearchBot decides what goes into ChatGPT search answers. OpenAI says a robots.txt change takes about 24 hours to show up in search.
Anthropic: ClaudeBot is the training bot, Claude-SearchBot indexes for search, and Claude-User opens a page when someone asks for it in chat.
Google: Google-Extended is not a bot but a name that only exists in robots.txt. Google’s documentation states it has no effect on inclusion or ranking in Search.

On Google, the shortcut can take Gemini with it

Google explained how it will read the token: if a site refuses training through the shared line and says nothing about Google-Extended, Google treats it as if Google-Extended were blocked. Under the current documentation, though, that same name also controls whether content grounds answers in the Gemini app and Vertex AI. Refusing training through the shortcut can quietly remove a site from Gemini answers that cite sources.

The announcement notes that sites explicitly allowing Google-Extended keep appearing in those answers, because a rule written by name beats the interpretation. The catch is that allowing that name also allows Gemini training: today the two go together. Our view is that this choice should be made in writing with the client, not inherited from a shortcut. Google promised updates through 2027, so this is a rule we will revisit.

On Bing, the split goes down to the page

Microsoft used the announcement to separate training from grounding, meaning the use of content to build an answer with a citation. Training accepts the token in robots.txt or a page-level setting in a meta tag or the X-Robots-Tag header. Grounding can only be set per page, with three options: allow, refuse, or allow only Microsoft’s own products, such as Copilot.

Two deadlines matter if you use NoArchive or NoCache to keep a client out of Microsoft’s AI. From December 15, 2026, Bing Webmaster Tools accepts a temporary grounding preference. After March 1, 2027, those two tags no longer count as a training or grounding refusal. According to Microsoft, none of this affects rankings in traditional Bing search.

How to measure this across clients

In Silent Puma, the AI bots section of each run shows what robots.txt says to GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others, with training bots kept apart from answering bots. It shows, it does not judge: blocking on purpose is the client’s call. The new token is not on that list yet, so for now you check it by opening the file.

What to do on Monday

Open each client’s /robots.txt and write down, by name, which AI bots are blocked and in which group.
Next to each one, note whether it reads to train or to answer. An OAI-SearchBot or Claude-SearchBot block inherited from a template is the first to investigate.
Ask the client two separate questions: do they accept training? Do they want to appear in answers with citations?
If they refuse training, add the __usage_ai-training group and keep the rules by name. Until each company confirms support, those are the ones that work.
Write the Google-Extended rule on purpose, based on what the client wants in Gemini.
For anyone using NoArchive or NoCache because of Bing, put March 1, 2027 on the calendar.
After changing a rule for an OpenAI search bot, wait a day before checking the effect.

One robots.txt line to refuse AI training