# Google’s shared AI training control: what the robots.txt token means

Google plans to read a shared __usage_ai-training token in robots.txt from 2027. What the token says, how Google maps it onto Google-Extended and Gemini grounding, and whether a store should change anything yet.

By Valerii Checha, 12 Nov 2026

Google-Extended, the robots.txt token Google already offers for AI training, also covers grounding in Gemini Apps: in Google's words, "providing content from the Google Search index to the model at prompt time". Google says a site that opts out of training with the new shared token, and does not mention Google-Extended, will be treated "as if the publisher had disallowed Google-Extended." So a store that opts out of training this way could, as we read Google's words, also step out of Gemini's grounded answers.

The new control is a robots.txt group under the name `__usage_ai-training`, defined in an IETF Internet-Draft published on 30 October 2026. One group expresses a training preference to every AI company that adopts it, and Google says it aims to honour it in 2027. Bing gives a date: "Starting February 1, 2027". It is a request about how content is used, not a block on crawling, and it changes nothing about Google Search.

## What did Google announce on 30 October 2026?

Google's post opens: "Today, we're announcing our plans to support a shared AI training control in robots.txt." Website owners "will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers", in place of one instruction per AI company.

The post does not print the token's name. Google's crawler documentation, updated the same day, does: Google "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: \_\_usage_ai-training") in early 2027." The post itself says "We aim to support this new control along with other AI providers in 2027". Here is how the pieces line up:

The shared AI training control, from Google-Extended to 2027.

Google-Extended (2023). What they say: Google calls it “the AI training control we’ve offered since 2023”. Its crawler page: Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Our reading: A Google-only token. It is no crawler of its own: Google says the token “is used in a control capacity”.

Sources: [Google Search Central: Supporting a shared AI training control (30 Oct 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control) and [Google: Google’s common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers).

The IETF definition (Sep 2026). What they say: The AIPREF vocabulary draft, revision 08, defines AI Training as “Using an asset to modify the learned parameters of a generative AI model.” Our reading: A working-group draft, not a standard. The new token borrows this definition.

Sources: [IETF: draft-ietf-aipref-vocab-08](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) and [IETF: draft-kelley-aipref-training-preference-signal-00](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html).

The shared token, drafted (30 Oct 2026). What they say: draft-kelley-aipref-training-preference-signal-00, by an Anthropic author, defines “the policy token \_\_usage_ai-training” and “introduces no new directive names and no new grammar”. Our reading: A proposal. The draft says “Intended Status: Standards Track”; the IETF datatracker lists no stream for it.

Source: [IETF: draft-kelley-aipref-training-preference-signal-00](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html).

Google’s plan (30 Oct 2026). What they say: When a site uses the token to opt out of training “(and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.” Our reading: An explicit Google-Extended group still wins for Google.

Source: [Google Search Central: Supporting a shared AI training control (30 Oct 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control).

Bing starts honouring it (1 Feb 2027). What they say: “Starting February 1, 2027, Microsoft will honor the new preferences as it crawls”. Our reading: A firm date, where Google gives a year.

Source: [Bing: clearer preferences for training and grounding (30 Oct 2026)](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/).

Google starts honouring it (early 2027). What they say: The post: “We aim to support this new control along with other AI providers in 2027”. The crawler page: Google “intends to support” it “in early 2027”. Our reading: An aim, with no date. Until then Google reads Google-Extended only.

Sources: [Google Search Central: Supporting a shared AI training control (30 Oct 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control) and [Google: Google’s common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers).

The draft expires (3 May 2027). What they say: “This Internet-Draft will expire on 3 May 2027.” Drafts “may be updated, replaced, or obsoleted by other documents at any time.” Our reading: Until it is revised or replaced, the token is a proposal, not a standard.

Source: [IETF: draft-kelley-aipref-training-preference-signal-00](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html).

Google calls its own text a "draft proposal". The underlying Internet-Draft says it is intended for the Standards Track, and the IETF datatracker lists no stream for it, so for now it is a proposal by one author, who works at Anthropic.

## What does `__usage_ai-training` mean in robots.txt?

It is a name in a `User-Agent` line that belongs to no crawler. The draft says it "introduces no new directive names and no new grammar", so a robots.txt parser that has never heard of it simply skips the group. Click through the draft's own example:

The draft’s example group, line by line.

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```

- \_\_usage_ai-training. In our words: A name in the User-Agent line that belongs to no crawler. It states a preference about use; it does not ask anyone to stay away. What the source says: “A policy token need not appear in the User-Agent header field … of any outbound request.” “This policy token is not a directive for crawling behavior.” And: “wildcard groups (i.e., User-Agent: \* ) MUST NOT be interpreted as matching the policy token.”
- Disallow: /. In our words: Every path: please do not train on this. What the source says: “If the lookup finds that the path is disallowed, a preference against the use described by the policy token has been expressed for that path.”
- Allow: /press-kit/. In our words: Except here: training on the press kit is welcome. The longest matching rule wins, as for any robots.txt group. What the source says: “This expresses a preference against AI training for all paths except those under /press-kit/ , for which it expresses a preference in favor”.

Source: [IETF: draft-kelley-aipref-training-preference-signal-00](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html).

Three rules from the draft matter for a store. The `*` group never counts: "wildcard groups (i.e., User-Agent: \* ) MUST NOT be interpreted as matching the policy token." A file with no `__usage_ai-training` group expresses no preference at all. And the token "is not a directive for crawling behavior": it asks AI companies not to train on a path, and it asks no crawler to stay away.

The name starts with two underscores. The draft explains that "the \_\_ prefix makes accidental matches unlikely", since some parsers match agent names loosely. It also makes the line look like something you should not edit casually, which is fair advice for most robots.txt files.

## How does the shared token map onto Google-Extended?

Google-Extended stays, and when you name it, it wins. Google's post says "Existing company-specific controls will continue to function alongside this new joint signal", and the translation applies only when "Google-Extended is not explicitly specified".

Google-Extended covers more than training. Google's crawler page says it manages whether content "may be used for training future generations of Gemini models" and "for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI." The post adds: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

Put those together and a store has three distinct choices with Google:

| If your robots.txt has                                                               | Google, once it reads the token                      | What that means, as we see it                                                        |
| ------------------------------------------------------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------------------------------ |
| No `__usage_ai-training` group and no Google-Extended group                          | Nothing changes                                      | Gemini may train on and ground with your pages, unless your `*` group blocks them    |
| `__usage_ai-training` disallowed, no Google-Extended group                           | "as if the publisher had disallowed Google-Extended" | No training, and no grounding in Gemini Apps either                                  |
| `__usage_ai-training` disallowed, plus `User-agent: Google-Extended` with `Allow: /` | Google-Extended's rules decide                       | AI companies that read the token get your opt-out; Google may still train and ground |

The third row is the one a store that wants Gemini's shoppers but not everyone's training runs should look at. It is also the most awkward to explain to a colleague.

## Does the shared token affect Google Search?

No, by Google's own words. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search", and the shared token only ever becomes Google-Extended. Googlebot's rules decide what Search crawls. If AI Overviews and AI Mode are what you are after, those are Search features, and our [GEO vs SEO article](https://storewalk.ai/blog/geo-vs-seo-for-online-stores) covers what Google says they need.

## Who else will read the token, and when?

Microsoft wrote on 30 October 2026 that "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls", and that "AI Training is the shared preference expressed in robots.txt for a site or path." Microsoft pairs it with a separate grounding preference of its own, set per page. The [robots.txt guide](https://storewalk.ai/guides/robots-txt-for-ai-agents) has Bing's details and a generator for the crawler side.

The draft's author works at Anthropic, and its acknowledgements thank people at Google, Microsoft, OpenAI, Amazon Web Services, Meta and others, listed, as the draft says, "for identification only". A thank-you is not a commitment. Of the pages we read, as of November 2026, only Google's and Microsoft's say they will read it.

## How would your robots.txt be read?

Paste your file, or start from one of the presets, and name a path. The first line follows the draft as written; the Google and Bing lines are our interpretation of their posts; the last runs our scanner's own robots.txt rule:

Read your robots.txt the way the draft, Google, Bing and our scan would.

On the page, you paste a robots.txt (or start from a preset), name a path, and read six lines: the draft's lookup, Google today, Google once it supports the token, Bing from 1 February 2027, Google Search, and our DISC-001. The presets, checked for /products/linen-duvet:

Shared token, opt out:

```
User-Agent: __usage_ai-training
Disallow: /

User-agent: *
Disallow: /checkout
```

- The shared token (the draft): A preference against AI training for this path.
- Google today (our reading): No Google-Extended group, so the \* group decides: allowed for this path. The shared token is not read yet.
- Google, once it supports the token (our reading): Treated “as if the publisher had disallowed Google-Extended”.
- Bing, from 1 February 2027 (our reading): Honours the preference against training for this path.
- Google Search (our reading): Decided by Googlebot’s rules, not by either training control: may crawl this path.
- Our DISC-001: WARN: no agent is fully blocked, but each inherits some Disallow lines, such as cart or checkout. That is normal for a store and costs little.

Opt out, but allow Google-Extended:

```
User-Agent: __usage_ai-training
Disallow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /checkout
```

- The shared token (the draft): A preference against AI training for this path.
- Google today (our reading): Reads Google-Extended: allowed for this path. The shared token is not read yet.
- Google, once it supports the token (our reading): Google-Extended is named, so its rules decide: allowed.
- Bing, from 1 February 2027 (our reading): Honours the preference against training for this path.
- Google Search (our reading): Decided by Googlebot’s rules, not by either training control: may crawl this path.
- Our DISC-001: WARN: no agent is fully blocked, but each inherits some Disallow lines, such as cart or checkout. That is normal for a store and costs little.

Google-Extended only:

```
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /checkout
```

- The shared token (the draft): No preference: no group names \_\_usage_ai-training, and the \* group does not count.
- Google today (our reading): Reads Google-Extended: disallowed for this path. The shared token is not read yet.
- Google, once it supports the token (our reading): Google-Extended is named, so its rules decide: disallowed.
- Bing, from 1 February 2027 (our reading): No robots.txt training preference to honour.
- Google Search (our reading): Decided by Googlebot’s rules, not by either training control: may crawl this path.
- Our DISC-001: WARN: only training crawlers and tokens are fully blocked, so search and shopping agents can still read the store. A fair choice, though some assistants may know less about you.

The draft’s example:

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

User-agent: *
Disallow: /checkout
```

- The shared token (the draft): A preference against AI training for this path.
- Google today (our reading): No Google-Extended group, so the \* group decides: allowed for this path. The shared token is not read yet.
- Google, once it supports the token (our reading): Treated “as if the publisher had disallowed Google-Extended”.
- Bing, from 1 February 2027 (our reading): Honours the preference against training for this path.
- Google Search (our reading): Decided by Googlebot’s rules, not by either training control: may crawl this path.
- Our DISC-001: WARN: no agent is fully blocked, but each inherits some Disallow lines, such as cart or checkout. That is normal for a store and costs little.

The first line follows the draft as written. The Google and Bing lines are our reading of their 30 October posts. The last line runs our scanner’s own robots.txt rule, which reads named agents on our list and no policy tokens. Nothing you type leaves the page.

Sources: [IETF: draft-kelley-aipref-training-preference-signal-00](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html), [Google Search Central: Supporting a shared AI training control (30 Oct 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control), [Bing: clearer preferences for training and grounding (30 Oct 2026)](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) and [Storewalk: DISC-001](https://storewalk.ai/checks/DISC-001).

Try the draft's example with `/press-kit/logo.png`: the longest matching rule wins, so the press kit is welcome for training while the rest of the site is not.

## Should a store change its robots.txt now?

Only if you already know your answer on training, and then with Google-Extended in mind. Neither Google nor Bing says it reads the token before 2027, so adding it now costs nothing and changes nothing yet. If you add it, decide at the same time whether you want a `Google-Extended` group, since without one Google plans to read your opt-out as covering Gemini's grounding too.

Our scan treats these lines the way it treats any robots.txt. [DISC-001](https://storewalk.ai/checks/DISC-001) reads the named agents on our list and no policy tokens, so a `__usage_ai-training` group alone changes nothing in it. Blocking Google-Extended is a training-only block, which DISC-001 grades as a warning: search and shopping agents can still read the store. [VER-004](https://storewalk.ai/checks/VER-004) reads `Content-Signal` and `Content-Usage` lines, which are a different way of stating usage preferences, and not the token.

The token says nothing about who may fetch your pages, and that is the question that costs a store sales. DISC-001 answers it for each named agent on our list whenever you [scan your store](https://storewalk.ai/). Robots.txt is also only the first of the things that can stop an agent buying, and [the others](https://storewalk.ai/blog/why-agents-cant-buy) get an article to themselves.

## Frequently asked questions

### What is Google's shared AI training control?

A robots.txt group under the name `__usage_ai-training` that states a training preference to every AI company that reads it. Google announced on 30 October 2026 that it plans to support it, and its crawler page says "in early 2027".

### Does `__usage_ai-training` replace Google-Extended?

No. Google says company-specific controls "will continue to function alongside this new joint signal". A Google-Extended group, if you have one, decides for Google.

### Will opting out of AI training remove my store from Gemini?

From Gemini's grounded answers, it may. Without a Google-Extended group, Google plans to treat a token opt-out as disallowing Google-Extended, and Google-Extended covers grounding in Gemini Apps. Google Search is not affected.

### Is `__usage_ai-training` a standard?

Not yet. It is defined in an Internet-Draft that expires in May 2027 unless it is revised, and the draft itself says drafts may be "updated, replaced, or obsoleted by other documents at any time."

### Does it stop AI crawlers from fetching my pages?

No. The draft says the token "is not a directive for crawling behavior". To keep a crawler out, use a group for that crawler's own name.

The sources below were last checked on the date shown.

## Sources

- [Google Search Central: Supporting a shared AI training control (30 Oct 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control)
- [Google: Google’s common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [IETF: draft-kelley-aipref-training-preference-signal-00](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html)
- [IETF: draft-ietf-aipref-vocab](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)
- [Bing: clearer preferences for training and grounding (30 Oct 2026)](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)

Last checked 4 Nov 2026
