Skip to content

VER-004Beside the path

Content Signals / AI usage preferences declared

Checks whether your robots.txt or home page headers state how AI systems may use your content, using Content Signals or the IETF AIPREF vocabulary.

What we check

We look for AI usage preferences in two vocabularies. Cloudflare's Content Signals Policy (a vendor convention published 2025-09-24) uses a robots.txt line such as "Content-Signal: search=yes, ai-input=yes, ai-train=no", with values yes or no. The IETF AIPREF drafts, still working group drafts (draft-ietf-aipref-vocab-08, 2026-09-14, and draft-ietf-aipref-attach-05, 2026-08-19), use "Content-Usage: search=y, ai-use=y, train-ai=n" in robots.txt or as an HTTP header. We read both forms in robots.txt and as Content-Signal or Content-Usage headers on your home page.

Why it matters

Shopping agents read your product pages to answer shoppers' questions. The setting that matters for shopping agents is ai-input (Content Signals) or ai-use (AIPREF). Refusing AI training with ai-train=no or train-ai=n does not stop agents using your pages to help shoppers.

How we check

This check makes no extra requests: it uses the robots.txt our crawl fetched and the headers of your home page. Comment lines are ignored.

PASS: at least one well-formed preference and no site-wide refusal of AI use. WARN: preferences are present but none is well-formed (for example the two vocabularies mixed, as in "Content-Signal: train-ai=n", or values such as "maybe"); ai-input=no or ai-use=n in the "User-agent: *" group or a response header, which tells every agent not to use your content; or such a refusal only in groups for named bots, which we report naming those bots. A refusal limited to one path, such as /private/, does not count as refusing the shop. FAIL: no Content-Signal or Content-Usage preference found. SKIPPED: robots.txt could not be read.

How to fix it

  1. Open robots.txt and, inside the "User-agent: *" group, state how AI systems may use your content, for example "Content-Signal: search=yes, ai-input=yes, ai-train=no".
  2. Or use the IETF AIPREF form, "Content-Usage: search=y, ai-use=y, train-ai=n"; do not mix the two vocabularies in one line.
  3. On Shopify: edit robots.txt through the robots.txt.liquid theme template.
  4. On WooCommerce: WordPress generates robots.txt automatically; add the line through your SEO plugin's robots.txt editor or by placing a robots.txt file in your site's root folder.

What good looks like

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# Or, using the IETF AIPREF vocabulary:
User-agent: *
Content-Usage: search=y, ai-use=y, train-ai=n
Allow: /

Sources