Skip to content

Guide7 min readPublished

Google’s shared AI training control: what the robots.txt token means

Google plans to read a shared __usage_ai-training token in robots.txt from 2027. What the token says, how Google maps it onto Google-Extended and Gemini grounding, and whether a store should change anything yet.

Google-Extended, the robots.txt token Google already offers for AI training, also covers grounding in Gemini Apps: in Google's words, "providing content from the Google Search index to the model at prompt time". Google says a site that opts out of training with the new shared token, and does not mention Google-Extended, will be treated "as if the publisher had disallowed Google-Extended." So a store that opts out of training this way could, as we read Google's words, also step out of Gemini's grounded answers.

The new control is a robots.txt group under the name __usage_ai-training, defined in an IETF Internet-Draft published on 30 October 2026. One group expresses a training preference to every AI company that adopts it, and Google says it aims to honour it in 2027. Bing gives a date: "Starting February 1, 2027". It is a request about how content is used, not a block on crawling, and it changes nothing about Google Search.

What did Google announce on 30 October 2026?

Google's post opens: "Today, we're announcing our plans to support a shared AI training control in robots.txt." Website owners "will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers", in place of one instruction per AI company.

The post does not print the token's name. Google's crawler documentation, updated the same day, does: Google "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027." The post itself says "We aim to support this new control along with other AI providers in 2027". Here is how the pieces line up:

The shared AI training control, from Google-Extended to 2027
When

Showing: Google-Extended

What they say

Google calls it “the AI training control we’ve offered since 2023”. Its crawler page: Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

Our reading

A Google-only token. It is no crawler of its own: Google says the token “is used in a control capacity”.

Sources: Google Search Central: Supporting a shared AI training control (30 Oct 2026) and Google: Google’s common crawlers (Google-Extended)

Google calls its own text a "draft proposal". The underlying Internet-Draft says it is intended for the Standards Track, and the IETF datatracker lists no stream for it, so for now it is a proposal by one author, who works at Anthropic.

What does __usage_ai-training mean in robots.txt?

It is a name in a User-Agent line that belongs to no crawler. The draft says it "introduces no new directive names and no new grammar", so a robots.txt parser that has never heard of it simply skips the group. Click through the draft's own example:

The draft’s example group, line by line
User-Agent: 

__usage_ai-training

In our words

A name in the User-Agent line that belongs to no crawler. It states a preference about use; it does not ask anyone to stay away.

What the source says

“A policy token need not appear in the User-Agent header field … of any outbound request.” “This policy token is not a directive for crawling behavior.” And: “wildcard groups (i.e., User-Agent: * ) MUST NOT be interpreted as matching the policy token.”

Source: IETF: draft-kelley-aipref-training-preference-signal-00

Three rules from the draft matter for a store. The * group never counts: "wildcard groups (i.e., User-Agent: * ) MUST NOT be interpreted as matching the policy token." A file with no __usage_ai-training group expresses no preference at all. And the token "is not a directive for crawling behavior": it asks AI companies not to train on a path, and it asks no crawler to stay away.

The name starts with two underscores. The draft explains that "the __ prefix makes accidental matches unlikely", since some parsers match agent names loosely. It also makes the line look like something you should not edit casually, which is fair advice for most robots.txt files.

How does the shared token map onto Google-Extended?

Google-Extended stays, and when you name it, it wins. Google's post says "Existing company-specific controls will continue to function alongside this new joint signal", and the translation applies only when "Google-Extended is not explicitly specified".

Google-Extended covers more than training. Google's crawler page says it manages whether content "may be used for training future generations of Gemini models" and "for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI." The post adds: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

Put those together and a store has three distinct choices with Google:

If your robots.txt hasGoogle, once it reads the tokenWhat that means, as we see it
No __usage_ai-training group and no Google-Extended groupNothing changesGemini may train on and ground with your pages, unless your * group blocks them
__usage_ai-training disallowed, no Google-Extended group"as if the publisher had disallowed Google-Extended"No training, and no grounding in Gemini Apps either
__usage_ai-training disallowed, plus User-agent: Google-Extended with Allow: /Google-Extended's rules decideAI companies that read the token get your opt-out; Google may still train and ground

The third row is the one a store that wants Gemini's shoppers but not everyone's training runs should look at. It is also the most awkward to explain to a colleague.

No, by Google's own words. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search", and the shared token only ever becomes Google-Extended. Googlebot's rules decide what Search crawls. If AI Overviews and AI Mode are what you are after, those are Search features, and our GEO vs SEO article covers what Google says they need.

Who else will read the token, and when?

Microsoft wrote on 30 October 2026 that "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls", and that "AI Training is the shared preference expressed in robots.txt for a site or path." Microsoft pairs it with a separate grounding preference of its own, set per page. The robots.txt guide has Bing's details and a generator for the crawler side.

The draft's author works at Anthropic, and its acknowledgements thank people at Google, Microsoft, OpenAI, Amazon Web Services, Meta and others, listed, as the draft says, "for identification only". A thank-you is not a commitment. Of the pages we read, as of November 2026, only Google's and Microsoft's say they will read it.

How would your robots.txt be read?

Paste your file, or start from one of the presets, and name a path. The first line follows the draft as written; the Google and Bing lines are our interpretation of their posts; the last runs our scanner's own robots.txt rule:

Read your robots.txt the way the draft, Google, Bing and our scan would
Start from
The shared token (the draft)
A preference against AI training for this path.
Google today (our reading)
No Google-Extended group, so the * group decides: allowed for this path. The shared token is not read yet.
Google, once it supports the token (our reading)
Treated “as if the publisher had disallowed Google-Extended”.
Bing, from 1 February 2027 (our reading)
Honours the preference against training for this path.
Google Search (our reading)
Decided by Googlebot’s rules, not by either training control: may crawl this path.
Our DISC-001
WARN: no agent is fully blocked, but each inherits some Disallow lines, such as cart or checkout. That is normal for a store and costs little.

The first line follows the draft as written. The Google and Bing lines are our reading of their 30 October posts. The last line runs our scanner’s own robots.txt rule, which reads named agents on our list and no policy tokens. Nothing you type leaves the page.

Sources: IETF: draft-kelley-aipref-training-preference-signal-00, Google Search Central: Supporting a shared AI training control (30 Oct 2026), Bing: clearer preferences for training and grounding (30 Oct 2026) and Storewalk: DISC-001

Try the draft's example with /press-kit/logo.png: the longest matching rule wins, so the press kit is welcome for training while the rest of the site is not.

Should a store change its robots.txt now?

Only if you already know your answer on training, and then with Google-Extended in mind. Neither Google nor Bing says it reads the token before 2027, so adding it now costs nothing and changes nothing yet. If you add it, decide at the same time whether you want a Google-Extended group, since without one Google plans to read your opt-out as covering Gemini's grounding too.

Our scan treats these lines the way it treats any robots.txt. DISC-001 reads the named agents on our list and no policy tokens, so a __usage_ai-training group alone changes nothing in it. Blocking Google-Extended is a training-only block, which DISC-001 grades as a warning: search and shopping agents can still read the store. VER-004 reads Content-Signal and Content-Usage lines, which are a different way of stating usage preferences, and not the token.

The token says nothing about who may fetch your pages, and that is the question that costs a store sales. DISC-001 answers it for each named agent on our list whenever you scan your store. Robots.txt is also only the first of the things that can stop an agent buying, and the others get an article to themselves.

Frequently asked questions

What is Google's shared AI training control?

A robots.txt group under the name __usage_ai-training that states a training preference to every AI company that reads it. Google announced on 30 October 2026 that it plans to support it, and its crawler page says "in early 2027".

Does __usage_ai-training replace Google-Extended?

No. Google says company-specific controls "will continue to function alongside this new joint signal". A Google-Extended group, if you have one, decides for Google.

Will opting out of AI training remove my store from Gemini?

From Gemini's grounded answers, it may. Without a Google-Extended group, Google plans to treat a token opt-out as disallowing Google-Extended, and Google-Extended covers grounding in Gemini Apps. Google Search is not affected.

Is __usage_ai-training a standard?

Not yet. It is defined in an Internet-Draft that expires in May 2027 unless it is revised, and the draft itself says drafts may be "updated, replaced, or obsoleted by other documents at any time."

Does it stop AI crawlers from fetching my pages?

No. The draft says the token "is not a directive for crawling behavior". To keep a crawler out, use a group for that crawler's own name.

The sources below were last checked on the date shown.