Skip to content

DISC-001Find

robots.txt allows AI agents

Checks whether your robots.txt blocks, fully or partly, the AI crawlers and agents on our list.

What we check

We read your robots.txt and work out, for each of 22 agent tokens on our list (among them GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Google-Extended and CCBot), which group of rules applies to it and how much of the site it disallows. We report separately agents blocked by a group naming them and agents blocked only by "User-agent: *". Content-Signal or Content-Usage lines are recorded as evidence and do not change the result.

Why it matters

robots.txt is the first file a well-behaved AI crawler or assistant reads. If it disallows everything for an agent, that agent should not read your product pages, so it cannot find, compare or recommend them. A blanket block under "User-agent: *" does this to every agent without its own group. The list includes training-only crawlers (such as GPTBot, Google-Extended and CCBot), so a deliberate training opt-out also shows up here.

How we check

We use the robots.txt captured during the crawl (its first 4,000 characters). For each agent we pick the group naming it, falling back to "*". FAIL: an agent's group disallows everything (for example "Disallow: /") and has no Allow lines. WARN: nobody is fully blocked, but some agent's group has any non-empty Disallow line, even a narrow one such as /cart. PASS: no agent's group disallows anything. SKIPPED: robots.txt did not return HTTP 200 during the crawl, including when the file does not exist.

How to fix it

  1. Remove or narrow any rule that disallows everything for an agent named in the result. If you choose to keep blocking training-only crawlers, the result stays FAIL for those tokens.
  2. If a blanket rule under "User-agent: *" is the cause, remove it or add groups with "Allow: /" for the agents you want to admit.
  3. Narrow rules for cart, checkout or admin paths give a WARN. Shopify's default robots.txt disallows paths such as /cart and /checkout (editable through the theme's robots.txt.liquid template); WordPress's default disallows /wp-admin/. Decide whether to keep them.
  4. Keep agent groups near the top of the file and serve it with HTTP 200.

What good looks like

User-agent: *
Allow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: https://www.example-shop.com/sitemap.xml

Sources