Four AI giants back one robots.txt training opt-out—but it is still voluntary

The story

OpenAI, Google, Microsoft and Anthropic are backing a shared signal that would let a website express its generative-AI training preference once, rather than maintain a separate robots.txt rule for every provider. The coordination is potentially significant for publishers and other web operators, but the announcement is not the arrival of an enforceable global standard. The mechanism is an individual Internet-Draft, participation is voluntary, implementation dates differ, and the signal covers a narrower activity than many descriptions of an AI scraping opt-out imply.

The October 30 proposal was written by Anthropic standards lead Kevin Kelley. It uses the existing robots.txt format and introduces a policy token named __usage_ai-training. A site can place that token in a User-Agent line, use Disallow for material it does not want used to train generative models, and create path-level exceptions with Allow. The draft's example blocks an entire site except its press kit. Because the design adds no new grammar, existing parsers that do not recognize the token should ignore it without disrupting ordinary crawler rules.

That compatibility is the proposal's technical advantage. Website operators currently identify training crawlers such as GPTBot, ClaudeBot or Google-Extended individually, and the list can change as companies rename services or separate crawling purposes. A common token moves the preference from the identity of a particular bot to the intended use of the content. The proposal defines AI training as using an asset to modify the learned parameters of a generative model. It does not include models used exclusively for classification, ranking or scoring, even if those systems can generate explanations.

The boundaries matter. The draft explicitly says the policy token is not a directive controlling whether a crawler may fetch a page. A publisher that wants to stop retrieval still needs ordinary rules addressed to the crawler's product token. The shared signal also does not, by itself, govern the use of a page to ground a live AI response, determine whether it appears in search, establish a price, or create a licence. Microsoft is adding separate page-level controls for training and grounding, but those extensions apply to Microsoft rather than automatically binding every participant.

Implementation is staggered. Kelley told the IETF mailing list that Anthropic has started building support, while warning that adding it across services will take time and expose edge cases. OpenAI says it intends to honor the signal in the coming months without replacing its existing crawler controls. Google plans support in 2027 and says a shared opt-out will be interpreted like a Google-Extended disallow when a site has not set a more specific Google rule. Microsoft will begin honoring its new preferences as it recrawls content on February 1, 2027.

No website should assume that adding the token today has the same effect everywhere. The IETF Datatracker labels the document an individual draft with no formal standing or endorsement; it expires on May 3, 2027 unless updated. The usual standards process can change its wording and semantics. The draft also says private agreements may override public preferences, while provider-specific tokens can take precedence over the shared one. Those clauses preserve commercial flexibility but make uniform behavior harder to infer from a site's file alone.

The proposal nevertheless answers a real coordination failure. Independent reporting by The Verge described growing publisher concern about AI crawlers and a patchwork of responses, including crawler blocks and emerging licensing systems. A single signal could lower the operational burden for small publishers that lack legal teams or continuously updated bot lists. Google also links the plan to the European Union's general-purpose AI code of practice, which recognizes robots.txt as a machine-readable way for rights holders to express reservations. Recognition, however, is not the same as proof that a provider complied.

INNOVOX analysis: the decisive question is whether a preference becomes verifiable. Robots.txt is public and simple, so a publisher can show what it declared. It cannot see which retrieved files entered a training corpus, whether older copies remain, or how a model developer interpreted exceptions. A credible implementation would therefore document collection dates, the treatment of previously acquired content, provider-specific overrides and the process for correcting mistakes. Shared syntax reduces friction; transparent records and remedies would turn it into meaningful governance.

What to watch next is standards adoption and market coverage. The IETF AI Preferences working group is already developing broader vocabulary and attachment methods, and it may absorb, modify or reject this proposal. The four supporters should publish exact launch dates, conformance tests and clear definitions for each service. Meta has voiced support for a common approach without publicly committing to honor this signal, according to The Verge. Until participation is broader and compliance can be checked, __usage_ai-training is a promising common language—not a universal switch for controlling how the web trains AI.

INNOVOX analysis

The important innovation is coordination rather than syntax. A common preference can reduce the cost of saying no across providers, but its value depends on observable compliance, stable definitions and adoption beyond four companies. Without enforcement or auditability, the proposal remains a promise expressed through infrastructure rather than a complete rights system.

What to watch

Watch whether the IETF working group adopts or revises the individual draft, whether the four providers publish testable implementation rules, and whether Meta and other major model developers commit. Publishers should also watch how training controls interact with grounding, licensing deals and regional law.