robots.txt allows AI agents

What we check

We read your robots.txt and work out, for each of 22 agent tokens on our list (among them GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Google-Extended and CCBot), which group of rules applies to it and how much of the site it disallows. Each token is sorted by what its vendor says it is for: training a model, building a search index, or fetching a page because a person asked. The result lists agents blocked by a group that names them separately from agents blocked only by "User-agent: *". We record any Content-Signal or Content-Usage lines as evidence; they do not change the result.

Why it matters

robots.txt is the first file a well-behaved AI crawler or assistant reads. If it disallows everything for an agent, that agent should not read your product pages, so it cannot find, compare or recommend them. A blanket block under "User-agent: *" does this to every agent without its own group. The list also includes training-only crawlers and training tokens (such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot). Blocking those is a legitimate choice and does not stop search or shopping agents, but some assistants may have less to say about the store, so it shows up as a WARN rather than a FAIL.

How we check

We use the whole robots.txt captured during the crawl, up to the 512 KiB we read. A file longer than that is not graded, since a group past the cut could change any agent's result. For each agent we pick the group naming it, falling back to "*". In a rule only "*" and a final "$" are special, as RFC 9309 says, so Magento's default "Disallow: /*?" blocks addresses with a query string and nothing else. FAIL: the group of a search crawler or a user-triggered agent (such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot or Googlebot) disallows everything (for example "Disallow: /") and has no Allow lines. WARN: only training crawlers or training tokens are fully blocked, or nobody is fully blocked but some agent's group has any non-empty Disallow line, even a narrow one such as /cart. PASS: no agent's group disallows anything. SKIPPED: robots.txt did not return HTTP 200 during the crawl, including when the file does not exist.

How to fix it

1. Remove or narrow any rule that disallows everything for a search crawler or user-triggered agent named in the result. If you choose to keep blocking training-only crawlers, you can: the result stays a WARN, and search and shopping agents still get in.
2. If a blanket rule under "User-agent: *" is the cause, remove it or add groups with "Allow: /" for the agents you want to admit.
3. Narrow rules for cart, checkout or admin paths give a WARN. Shopify's default robots.txt disallows paths such as /cart and /checkout (editable through the theme's robots.txt.liquid template); WordPress's default disallows /wp-admin/. Decide whether to keep them.
4. Keep agent groups near the top of the file and serve it with HTTP 200.

What good looks like

User-agent: *
Allow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: https://www.example-shop.com/sitemap.xml

Sources

• RFC 9309: Robots Exclusion Protocol
• OpenAI: crawlers and user agents
• Anthropic: Claude crawlers
• Perplexity: crawlers
• Google: common crawlers (Googlebot, Google-Extended)
• Content Signals

Read more

• Bots, crawlers and agents: who is knocking

  A User-Agent string can be spoofed, and robots.txt rules are a request to crawlers rather than access authorization. Web Bot Auth, an IETF draft, would let an agent sign its requests so a server can verify who sent them.