OpenAI, Google, Microsoft and Anthropic Back One robots.txt Opt-Out for AI Training

Updated November 1, 2026

On October 30, OpenAI, Google, Microsoft and Anthropic said they will support one shared signal in robots.txt that lets website owners opt their content out of AI training. Instead of blocking each company's crawler separately, a publisher could use a single entry. The details matter, though, because the signal is voluntary and rolls out on different dates.

What is changing?

Until now, a site that did not want its content used for AI training had to add a separate rule for every AI company's crawler, such as OpenAI's GPTBot, Google-Extended or Anthropic's ClaudeBot. The new signal uses one name, __usage_ai-training, that works across the companies that support it.

Based on reporting about the proposal, the rule is a robots.txt group that uses that name as the user-agent, followed by a Disallow line:

User-Agent: __usage_ai-training
Disallow: /

Publishers can also list specific paths instead of blocking the whole site.

Where it comes from

The signal comes from an individual Internet-Draft titled Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic. It is a proposal submitted to the IETF, not a finished, formal standard. Google, Microsoft and OpenAI have announced support for it.

When will each company honor it?

What it does not do

• It is voluntary. Like robots.txt itself, it works only if crawlers choose to respect it.
• It is about training. Training, AI-generated answers and normal search are separate uses. Microsoft, for example, is adding a separate page-level control for AI "grounding," so publishers must set these preferences independently.
• It is not retroactive. Updating robots.txt does not remove content that was already collected into existing datasets or used in trained models.
• No signal means no preference. A site without this group has not expressed an opt-out, and existing rules for other crawlers do not count as one.
• Existing controls stay. OpenAI, Google and Anthropic say their current controls keep working alongside the new token.

What website owners can do

1. Decide whether you want to opt out of AI training at all. This is separate from search visibility.
2. Add the __usage_ai-training group to your robots.txt, with the paths you want excluded.
3. Keep your existing rules for company-specific crawlers for now, since support is still rolling out.
4. Review other AI-related controls, such as Microsoft's grounding setting, if you care about answer engines.

Why it matters

For publishers, this could cut a long, constantly changing list of crawler rules down to one line. For AI companies, it is a public commitment to a common way of respecting publisher preferences. Whether it works in practice will depend on how widely it is adopted and enforced.

Disclosure: Anthropic, one of the companies named here, makes Claude, the AI assistant used to help research and draft this post.