🔍 Read the full analysis: How The New AI Training Opt-Out Works And Where It Falls Short on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, does not cover AI answers based on live web content, and will take effect on different schedules; Meta voiced support without committing to adopt it.
OpenAI, Google, Microsoft and Anthropic said on October 30 they will honor a shared instruction in websites’ robots.txt files asking them not to use the site’s content for generative AI training. The signal could spare publishers from maintaining separate training opt-outs for each company, but it is a voluntary request—not a technical lock—and adoption will happen on different timelines.
The proposed instruction uses a new user-agent label, User-Agent: __usage_ai-training, followed by Disallow: /. It is set out in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. The draft is associated with the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
The companies’ plans differ. Anthropic says it has started implementing support, while noting that adding it across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Google aims to support it in 2027, with its crawler documentation specifying “early 2027.” Microsoft says it will honor the training preference from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for “AI Grounding,” covering web content used to inform answers in Bing, Copilot and Microsoft 365. Its changes affect the role of the company’s existing NoArchive and NoCache tags: Microsoft says those tags will stop counting as training or grounding preferences after March 1, 2027. Existing controls such as GPTBot, Google-Extended and ClaudeBot remain relevant while companies roll out the shared signal.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, a common instruction may reduce the work of communicating a preference to participating AI companies. Instead of relying solely on separate crawler rules, a site could publish one shared training preference. That is a practical change, especially for organizations managing many pages or multiple AI-related policies.
But the signal does not compel compliance or prevent a crawler from accessing a website. It records a publisher’s preference, and its effect depends on companies choosing to honor it and implementing the draft. It also does not provide payment, traffic or guaranteed exclusion from AI systems. Microsoft explicitly says neither preference guarantees inclusion, traffic or payment.
The distinction between training and answers matters. The shared signal covers use of content to train generative AI models, not necessarily use of live web pages to generate or ground answers. Microsoft’s separate grounding preference addresses that use for its products. Publishers therefore need to decide separately how they want to handle training, AI answers and search, rather than assuming one instruction controls all three.
Popular right now
De'Longhi Rivelia Automatic Espresso Machine with Milk Frother
Shark Rotator Lift-Away Vacuum with DuoClean PowerFins (LA555) – Dark Grey
Bose QuietComfort Ultra Wireless Noise-Cancelling Headphones with Spatial Audio, Black
As an affiliate, we earn on qualifying purchases.
How Publisher Controls Are Changing
Until now, website operators have commonly used company-specific controls: GPTBot for OpenAI, Google-Extended for Google, and ClaudeBot for Anthropic. Microsoft has used page-level tags including NoArchive and NoCache for certain preferences. A site that wanted to communicate its position to several providers could need to maintain several rules, and a missed or misunderstood rule might not express its preference to a particular company.
The new proposal aims to provide a shared, machine-readable option. Google says that if a site uses the new token and has no Google-Extended rule, it plans to interpret the shared instruction as if the publisher had disallowed Google-Extended. The companies have not all committed to the same launch date, however, so the older company-specific controls may still matter during rollout.
The draft’s status is a key part of the story. It is an individual Internet-Draft, not a formally adopted IETF standard. At a working-group meeting on October 21, a chair said, “We haven’t declared consensus on anything yet.” The companies’ announcements show support for the proposed mechanism, but do not by themselves establish a final technical standard or universal industry practice.
“Anthropic “has already started implementing support for this draft.””
— Anthropic standards lead Kevin Kelley
Who Will Honor the Request
Meta has expressed support for publishers stating preferences, but its communications chief did not say whether the company will honor this specific signal. The source material reports no commitment from Apple, Amazon, xAI or Perplexity, and it remains unclear which other AI providers may adopt the draft.
The practical reach of the signal will depend on implementation and compliance. It is not yet an official internet standard, and the companies have not all provided firm dates. The source material also does not establish how publishers could verify whether a particular company followed the preference, or what recourse would be available if it did not.
The new instruction does not remove material from models that have already been trained, block crawling, or automatically govern answers generated from live web content. Those limits mean that a publisher’s choice should not be read as a guarantee that its material will be absent from existing models or every AI product.
Rollout Dates and Site Updates
Website operators can add the shared training rule if they want participating companies to receive that preference, while keeping their existing company-specific directives until each provider has implemented the new signal. Operators using Microsoft’s NoArchive or NoCache tags should review Microsoft’s stated March 1, 2027 change and decide whether its separate AI Grounding control is relevant to their pages.
The next developments to watch are whether the IETF working group reaches consensus on the draft and whether companies publish more detailed implementation guidance. Anthropic’s support is already underway, OpenAI’s target is “the coming months,” and Google and Microsoft have specified 2027 timelines. Those schedules, and any decision by Meta or other providers, will determine how broadly the shared preference works in practice.
Key Questions
What does the shared AI training opt-out look like?
The draft proposes a robots.txt group with User-Agent: __usage_ai-training and Disallow: /. It communicates that a publisher does not want participating companies to use the site’s content for generative AI training.
Does the signal prevent AI companies from crawling a website?
No. It expresses a training preference; it is not a technical lock that prevents crawling. Its effect depends on a company choosing to honor and implement it.
Does it stop websites from being used in AI-generated answers?
Not by itself. The shared signal is for training, while use of live web content in AI answers is a separate issue. Microsoft announced a separate AI Grounding preference for certain Microsoft products.
When will the companies support the signal?
Anthropic says implementation has started. OpenAI aims to implement it “in the coming months.” Google aims for 2027, with documentation saying “early 2027.” Microsoft’s stated start date is February 1, 2027.
Has Meta committed to honoring the signal?
No commitment is reported in the source material. Meta’s communications chief expressed support for publishers’ ability to state preferences but did not say whether Meta would honor this particular draft.
Source: ThorstenMeyerAI.com
