robots.txt lets agents read the UCP profile

What we check

We ask your robots.txt whether each agent may fetch /.well-known/ucp. The agents are our own crawler plus every agent on our list that browses or buys for a person, such as ChatGPT-User, Claude-User, Perplexity-User and Google-Agent. Crawlers that only train models or build search indexes are left out, since this check is about agents acting for a buyer.

Why it matters

An agent that respects robots.txt will not read a profile it is told to stay away from. If the rules keep it off /.well-known/ucp, it cannot use anything in your profile. UCP Checker asks the same question for its own bot before it reads a profile.

How we check

We read the robots.txt the scan already fetched and apply the matching rules from RFC 9309 to the path /.well-known/ucp for each agent. This makes no extra request. The score is the share of agents allowed: 80 or more is a PASS and 40 or more is a WARN. With no robots.txt at all (a 404 or 410) nothing is blocked, so it is a PASS. SKIPPED: there is no UCP profile, or we could not read robots.txt and so cannot tell what it says.

How to fix it

1. Add "Allow: /.well-known/ucp" to the group of each agent you want to buy from you, or to the * group.
2. Remove any Disallow line that covers /.well-known/ for those agents.

What good looks like

User-agent: ChatGPT-User
Allow: /.well-known/ucp
Disallow: /account

User-agent: *
Allow: /

Sources

• RFC 9309: Robots Exclusion Protocol
• UCP Checker: how we check

Read more

• UCP explained

  The Universal Commerce Protocol: the profile a store publishes at /.well-known/ucp, the capabilities it declares there, and the checkout flow from a new basket to a placed order.