robots.txt for AI agents: which bots to allow and block

Most AI companies run more than one bot, and each has its own name, called a product token, which is what you write after User-agent: in robots.txt. They fall into four main kinds. Search bots build the index an assistant searches when it answers. Fetchers for a user visit one page because someone asked a question just now. Training crawlers collect pages for training models. And two vendors, Google and Apple, publish control tokens, names that no bot sends but that you can address to opt out of training. One bot fits none of these: OpenAI's OAI-AdsBot, which checks pages submitted as ads on ChatGPT.

OpenAI says "Each setting is independent of the others", and Perplexity and Amazon say the same of theirs, so a block on one name does not reach the others. Highlight a kind to see who is in it:

Two names behave differently from what their spelling suggests. Google-Extended is not a crawler: Google calls it "a standalone product token" and says it "does not impact a site's inclusion in Google Search". Applebot-Extended is the same idea for Apple, which says it "does not crawl webpages". Apple also has a fallback: if your file has no group for Applebot but has one for Googlebot, Apple says "the Apple robot will follow Googlebot instructions."

Blocking a search bot costs you a place in that assistant's search results. OpenAI writes "we recommend allowing OAI-SearchBot", Perplexity says the same of PerplexityBot, and Anthropic says that disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".

Blocking a training crawler costs you nothing in search, by its vendors' accounts. OpenAI says "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models", and the setting is independent of OAI-SearchBot. Anthropic says a block on ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets". Whether you want your product copy in a training set is a business question, and we do not think there is one right answer.

Blocking a fetcher for a user is the odd one. It may not work. OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity says its user fetcher "generally ignores robots.txt rules", Google says the same of its user-triggered fetchers, Meta says its fetcher "may bypass robots.txt rules" and Amazon says Amzn-User "may not follow all robots.txt directives". Anthropic is the exception: it says its bots respect "do not crawl" signals, Claude-User included, and that disabling Claude-User "prevents our system from retrieving your content in response to a user query". For a store, a fetcher is a shopper's assistant reading your product page on the shopper's behalf, so we would not block it even where the block works.

A robots.txt is a list of groups. Each group starts with one or more User-agent: lines, followed by Allow: and Disallow: lines with paths. A bot uses the group that names it, or the * group if none does, and inside the group the longest matching path wins. So a bot named in its own group does not also read the * group. That catches people out when they block one bot and expect it to keep obeying the cart rules. RFC 9309, the standard, calls these rules ones "that crawlers are requested to honor", and our robots.txt check reads them the same way.

Changes are not instant. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust", Perplexity says up to 24 hours, and Amazon about the same. Anthropic supports a Crawl-delay line too, if you want to slow ClaudeBot down rather than stop it.

The generator below writes a file from your choices. Start from one of three presets, change any bot, and copy the result. Under the file it prints how our own robots.txt check reads it.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://example.com/sitemap.xml

Paste the result into your platform's robots.txt editor, or ask whoever runs your site to. On Shopify that is a theme template, and on BigCommerce and Magento a settings box; each platform guide has the path. Keep your platform's own lines for search and filter pages, since they keep crawlers out of URLs that would multiply forever.

Our advice, for a store that wants to be found and bought from:

• Allow the search bots, Googlebot and Applebot among them.
• Allow the fetchers for a user. Most of them may ignore the file anyway, and they are the closest thing to a shopper you have.
• Ask everyone to skip the cart, checkout and account pages. Our DISC-001 reports this as a WARN rather than a PASS, because every agent inherits those lines, and we consider that fine for a store.
• Decide training by preference. If you block it, use the training names and the control tokens, never User-agent: * with Disallow: /. The vendors say a training-only block does not stop their search or shopping bots, and our DISC-001 flags such a file as a warning rather than a failure. Block a search bot or a fetcher for a user, and it fails.
• List your sitemap.

The mistake we would most like to save you from is the blanket block. A Disallow: / under * asks every search engine to leave as well, which is a high price for keeping one training crawler out. Robots files are like shop signs: "No dogs" is fine, "No customers" is bad for business.

Robots.txt says where a bot may go. Two newer ideas say what it may do with what it reads.

The first is Cloudflare's Content Signals, a Content-Signal: line inside a group with three categories: search, ai-input (using your pages for real-time answers) and ai-train. Our VER-004 looks for it. Like the rest of the file, it states a preference and enforces nothing. Cloudflare's own docs add a fourth key on sites that use its managed file, use=reference, which it explains as "Index, excerpt, and link back".

The second is a shared token for training. Microsoft wrote on 30 October 2026 that "Starting February 1, 2027", Bing will honour a robots.txt group under the name __usage_ai-training, replacing its older NoArchive and NoCache tags for this purpose. The same day, an Internet-Draft by K. Kelley of Anthropic described the same idea; it says "It introduces no new directive names and no new grammar". Google wrote in October that when a site uses "this policy token" to opt out of training, and does not name Google-Extended, "Google plans to interpret that as if the publisher had disallowed Google-Extended". It adds "We aim to support this new control along with other AI providers in 2027". Google's post never names the token, but its crawler documentation does name __usage_ai-training. It is a draft and a set of plans, so the generator offers the group as an extra, off by default. Adding it costs nothing, because bots that do not know the name have no group to match.

It can. Cloudflare's managed robots.txt, which its docs say "is available on all plans", adds a block that disallows eight crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, with a Content-signal: search=yes, ai-train=no, use=reference line for everyone else. Its newer Bot Preference Sync writes the file from your bot settings in the dashboard. Cloudflare says it is "available to all customers from the Free tier to Enterprise", that its lines are "prepended to the existing material", and that "For all new customers, Bot Preference Sync will be on by default".

The managed block leaves the search bots alone, and Bot Preference Sync follows whatever you set in the dashboard. Either way, the file your store serves may not be the file you wrote. Open /robots.txt in a browser before you edit anything, and check what is there. Cloudflare is candid about the limits too: "robots.txt compliance is voluntary".

Paste your file below, type a path, and see which bots are asked to skip it and which rule decided. It starts with a sample from the generator.

To see whether bots follow your file, you need traffic data. Cloudflare's AI Crawl Control has a robots.txt tab whose violations table "identifies AI crawlers that have requested paths explicitly disallowed by your robots.txt file". Your server logs show the same if you match the user agents against each vendor's published IP list, since anyone can type a bot's name. Our scan reads your file with the robots.txt check, and the agent readiness checklist puts it in context with the rest of the store.

Should I block GPTBot on my store?

It is your choice. OpenAI says disallowing GPTBot means your content "should not be used in training", and that each of its settings is independent, so ChatGPT search (OAI-SearchBot) is unaffected. Block GPTBot by name if you do not want your pages in training; never block *. Our scan's robots.txt check flags a block like that as a warning, not a failure.

Does blocking Google-Extended remove my store from Google Search?

No. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". It only covers how Google may use your content for its AI models.

Can I block AI shopping agents with robots.txt?

Not reliably. Several vendors say their fetchers for a user may ignore robots.txt, because a person asked for the page. Anthropic says it honours the file for Claude-User. If you need to stop a bot, that takes a firewall or bot management, and we would think twice before turning a shopper's assistant away.

How long do robots.txt changes take to apply?

OpenAI says about 24 hours for its search systems, Perplexity up to 24 hours, and Amazon about 24 hours.

What is the __usage_ai-training token?

A shared name for opting out of AI training in robots.txt. Microsoft says Bing will honour it "Starting February 1, 2027". Google says it plans to support a shared training token in 2027, though its post does not name this one; its crawler page, updated on 30 October 2026, does: Google "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027." It is described in an Internet-Draft, which is a proposal, not yet a standard.

The sources below were last checked on the date shown.