robots.txt generator for AI bots
Pick who may crawl your store, copy the file, and see how our robots.txt check would grade it.
A robots.txt generator for a store
Start from
Nothing is blocked. Your pages can be used for search, for answers and for training.
Each bot
Allow or block each bot on its own. Changing one turns the preset into your own mix.
Extras
Left out unless it is a full http or https address.
robots.txt7 lines
User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Disallow: /cart Disallow: /checkout Disallow: /account Sitemap: https://example.com/sitemap.xml
How our scan reads it
- Worth fixing: DISC-001 WARN: no agent is fully blocked, but each inherits some Disallow lines, such as cart or checkout. That is normal for a store and costs little.
robots.txt is a request. RFC 9309 calls its rules ones “that crawlers are requested to honor”, and a bot that ignores the file is not stopped by it. The file is built in your browser; nothing is sent.
Check the rest of your store.
Free, no account, about five minutes.
What the generator does
Start from one of three presets, then set each AI bot to allow or block. The bots are grouped the way their vendors describe them: search bots that build the index an assistant searches, fetchers that open a page because someone asked about it just now, training crawlers, and control tokens, which are names no bot sends but that you can address to opt out of training. The file updates as you click, and under it the generator prints how our robots.txt check, DISC-001, would grade it.
Three options sit under the bots. One asks every bot to skip the cart, checkout and account pages, which is normal for a store. One adds a Content-Signal line that says yes or no to search, AI answers and training, following your choices. The last adds the proposed __usage_ai-training group, which Microsoft says Bing will honour from February 1, 2027. Type your sitemap address and it goes at the bottom.
How to use the file
Copy the file into your platform’s robots.txt editor, or send it to whoever runs your site. On Shopify robots.txt is a theme template, and on BigCommerce and Magento it is a settings box. Keep the lines your platform already writes for search and filter pages, since they keep crawlers out of URLs that multiply forever. OpenAI and Perplexity say a change can take up to a day to reach their systems, so there is no need to refresh ChatGPT every five minutes.
The one choice we would talk you out of is Disallow: / under User-agent: *. It asks every search engine to leave as well, which is a steep price for keeping one training crawler out. Our check fails a file that blocks a search bot or a fetcher for a user, and only warns about a block on training alone.
What robots.txt cannot do
robots.txt is a request. It asks, and it cannot enforce. Several vendors say their fetchers for a user may ignore it, because a person asked for the page, and we would not block those anyway: a fetcher is a shopper’s assistant reading your product page. If Cloudflare sits in front of your store, its managed robots.txt or Bot Preference Sync can add lines of its own, so open /robots.txt in a browser to see the file your store really serves. The generator builds the file in your browser and sends nothing.
Frequently asked questions
Should I block GPTBot?
It is your choice. OpenAI says blocking GPTBot means your pages should not be used for training, and that the setting is independent of OAI-SearchBot, which ChatGPT search uses. Block it by name if you want to, and never block every bot to get there.
Does blocking Google-Extended take my store out of Google Search?
No. Google says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. It only covers how Google may use your content for its AI models.
Can robots.txt stop AI shopping agents?
Not reliably. Several vendors say their fetchers for a user may ignore robots.txt because a person asked for the page. Anthropic says Claude-User respects it. Stopping a bot for real takes a firewall or bot management.
Is anything I choose sent to Storewalk?
No. The file is built in your browser, and nothing you pick or type leaves it.
The full guide
robots.txt for AI agents: which bots to allow and block
The AI bots a store meets, what each vendor says its bot is for and what robots.txt does for it, which to allow and which to block, with a generator for your file and a checker for the one you have.