# How agents discover what a store offers

Well-known documents, HTTP Link headers and API catalogs are places where a store can say what it offers, and content signals say how its pages may be used. Their status ranges from Standards Track RFC to proposal.

By Valerii Checha, 15 Oct 2026

An agent asked to buy something from your store may start with little more than your domain name. If it is to use a store's protocols or APIs, it first has to learn which ones the store offers and where they are, and it may also look for what the store says about how its pages may be used. A store can publish that in files at fixed addresses, in pointers on ordinary responses and in lines in robots.txt, and each of these has its own status.

It belongs to a series. [The hub article](https://storewalk.ai/blog/why-agents-cant-buy) sets out five walls between an agent and a finished purchase, and finding what the store offers is the third. Reading the page itself is the first, covered in [How an AI agent reads a web page](https://storewalk.ai/blog/how-agents-read-a-page). The commerce protocols named here have their own articles: [UCP explained](https://storewalk.ai/blog/ucp-explained), [ACP explained](https://storewalk.ai/blog/acp-explained) and [MCP, A2A and WebMCP for stores](https://storewalk.ai/blog/mcp-a2a-webmcp-for-stores).

Where we explain a term in plain words, the explanation is ours, not any specification's. Everything else comes from the documents linked in the text and listed under sources.

## Why an agent has to look first

The IETF (the Internet Engineering Task Force) develops internet standards and publishes them as numbered documents called RFCs. Its RFC on well-known addresses opens: "Some applications on the Web require the discovery of information about an origin … (sometimes called "site-wide metadata") before making a request." An origin is a site as a browser sees it, such as `https://example.com`. Its RFC on API catalogs says of APIs (documented ways for programs to talk to a service): "Such APIs must be discovered before they may be used, i.e., the external party needs to know what APIs a given Publisher exposes, their purpose, any policies for usage, and the endpoint to interact with each API."

ACP's discovery proposal calls it "a bootstrapping problem: agents need to learn the API base URL before they can call any endpoint." Its answer is that "Agents only need a domain".

In our reading, a site can get an agent from its domain to that information with a document at an address the agent already knows, a pointer added to an ordinary response, or lines in robots.txt.

## Well-known URIs and the IANA registry

A URI is an address on the web, such as `https://example.com/robots.txt`. [RFC 8615](https://www.rfc-editor.org/rfc/rfc8615), "Well-Known Uniform Resource Identifiers (URIs)", is a Standards Track RFC of May 2019, a document on the IETF's path to becoming a standard. It "defines a path prefix for "well-known locations", "/.well-known/"". The prefix sits at the top of the site: "/.well-known/example" is a well-known URI, "whereas "/foo/.well-known/example" is not."

The RFC defines only the prefix. It "does not define a format or media type for the resource located at "/.well-known/"", and clients "should not expect a resource to exist at that location". A media type is the label that says what format a response is in, such as `application/json`. Each application defines its own name and document. In our reading, there is no index to read first: an agent has to know each name in advance.

RFC 8615 says "Applications that wish to mint new well-known URIs MUST register them", in a registry kept by [IANA](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml), the body that keeps the internet's official lists of names and numbers. Names are "registered on the advice of one or more experts, with a Specification Required", and a registration references "a specification that defines the format and associated media type(s)". The RFC also asks applications to choose precise names: "squatting" on generic terms "is not encouraged".

We read the registry on 22 October 2026; IANA had last updated it on 7 October. Pick a path to see what it said.

Which well-known paths IANA’s registry lists.

- /.well-known/api-catalog: In the registry. Reference: RFC 9727; change controller IETF; registered 2024-12-23. What it is for: A list of the publisher’s APIs, “to facilitate automated discovery and usage of published Application Programming Interfaces (APIs)”. Defined by: RFC 9727. Status: Standards Track, June 2025. Sources: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727) and [IANA Well-Known URIs registry](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml).
- /.well-known/oauth-authorization-server: In the registry. Reference: RFC 8414, section 3; change controller IESG (the IETF’s steering group); registered 2018-03-27. What it is for: Metadata “that an OAuth 2.0 client can use to obtain the information needed to interact with an OAuth 2.0 authorization server” (in our words, the service that grants an app permission to act on an account). Defined by: RFC 8414. Status: Standards Track, June 2018. Sources: [RFC 8414: OAuth 2.0 Authorization Server Metadata](https://www.rfc-editor.org/rfc/rfc8414) and [IANA Well-Known URIs registry](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml).
- /.well-known/oauth-protected-resource: In the registry. Reference: RFC 9728, section 3; change controller IETF; registered 2024-10-22. What it is for: Metadata a client can use “to obtain the information needed to interact with an OAuth 2.0 protected resource” (in our words, an API that needs that permission). Defined by: RFC 9728. Status: Standards Track, April 2025. Sources: [RFC 9728: OAuth 2.0 Protected Resource Metadata](https://www.rfc-editor.org/rfc/rfc9728) and [IANA Well-Known URIs registry](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml).
- /.well-known/agent-card.json: In the registry. Reference: the A2A specification; change controller Linux Foundation; registered 2025-08-01. What it is for: An A2A agent card, “A self-describing manifest for an agent”: its identity, capabilities, skills, communication methods and security requirements. Defined by: A2A specification. Status: A2A, latest released version 1.0.0. Sources: [A2A specification](https://a2a-protocol.org/latest/specification/) and [IANA Well-Known URIs registry](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml).
- /.well-known/ucp: Not in the registry on 22 October 2026. What it is for: A store’s UCP business profile: protocol version, services, optional capabilities and payment handlers. “Businesses publish their profile at /.well-known/ucp.” Defined by: UCP specification. Status: UCP release v2026-08-25, marked Latest. Sources: [UCP specification overview](https://ucp.dev/latest/specification/overview/) and [UCP releases on GitHub](https://github.com/Universal-Commerce-Protocol/ucp/releases).
- /.well-known/acp.json: Not in the registry on 22 October 2026. What it is for: Whether a seller supports ACP, which version, which transports and which capabilities, “all before creating a checkout session and without requiring authentication”. Defined by: ACP discovery RFC. Status: ACP discovery RFC, Proposal, version unreleased. Source: [ACP RFC: Discovery (Proposal, unreleased)](https://github.com/agentic-commerce-protocol/agentic-commerce-protocol/blob/main/rfcs/rfc.discovery.md).
- /.well-known/ai-catalog.json: Not in the registry on 22 October 2026. What it is for: An AI Catalog that can list a host’s MCP server cards. Hosts “MAY publish one” here, and clients doing domain-level discovery “SHOULD attempt to retrieve this well-known URL”. Defined by: MCP server-card extension, discovery document. Status: MCP extension SEP-2127, Final, optional; its discovery document is marked draft. Sources: [MCP server-card extension: discovery (draft)](https://github.com/modelcontextprotocol/ext-server-card/blob/main/docs/discovery.md) and [MCP SEP-2127: Server Cards](https://modelcontextprotocol.io/seps/2127-mcp-server-cards).
- /.well-known/http-message-signatures-directory: Not in the registry on 22 October 2026. The Web Bot Auth draft asks for it in its IANA section. What it is for: In our words, the key directory an agent’s operator publishes, so a site can check the agent’s signed requests. Defined by: Web Bot Auth draft. Status: Internet-Draft draft-ietf-webbotauth-httpsig-protocol-00, work in progress. Source: [IETF draft-ietf-webbotauth-httpsig-protocol-00](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/).

The registry as it read on 22 October 2026 (last updated 7 October 2026). In our words, an entry’s change controller is the party responsible for it. RFC 8615 says “Applications that wish to mint new well-known URIs MUST register them”, and a registration references a specification that defines the format. In our reading, the registry tells you whether a name has been registered, not whether a site serves it: UCP’s profile path is missing from it, yet the UCP specification tells businesses to publish their profile there. Sources: [IANA Well-Known URIs registry](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml), [RFC 8615: Well-Known URIs](https://www.rfc-editor.org/rfc/rfc8615) and [UCP specification overview](https://ucp.dev/latest/specification/overview/).

The registry lists `api-catalog`, the two OAuth metadata paths (OAuth is a way for one program to get limited access to an account held elsewhere), and `agent-card.json`, registered for A2A on 1 August 2025 with the Linux Foundation as its change controller, the party responsible for the entry. It does not list `ucp`, `acp.json`, `ai-catalog.json` or `http-message-signatures-directory`; the Web Bot Auth draft asks for the last of these in its IANA section. A missing entry does not stop anyone serving the path. UCP's specification says "Businesses publish their profile at /.well-known/ucp", and that name is not in the registry.

## Link headers

HTTP is the protocol browsers and servers use to exchange pages, and headers are the named lines at the top of each message. Any response can carry a pointer in its headers. [RFC 8288](https://www.rfc-editor.org/rfc/rfc8288), "Web Linking", is a Standards Track RFC of October 2017. It defines "a model for the relationships between resources on the Web ("links")" and "the serialisation of such links in HTTP headers with the Link header field".

A link has "a link context, a link relation type, a link target, and optionally, target attributes", and the RFC reads it as a sentence: "link context has a link relation type resource at link target". Its own example is `Link: <http://example.com/TheBook/chapter2>; rel="previous"`. The relation type is the word that says what kind of link it is, and IANA keeps a registry of those too. On 22 October 2026 it held `api-catalog`, which "Refers to a list of APIs available from the Publisher of the link context", along with `alternate`, `describedby`, `item` and `service-desc`. It held no `llms-txt`, `mcp` or `sitemap`.

RFC 9727 uses a Link header to point at an API catalog from an ordinary page. A publisher "could include the api-catalog link relation in the HTTP header and/or content payload when responding to a request to <https://www.example.com>". In our reading, a program that only asked for the home page can learn from the answer where the catalog is. The figure takes the RFC's example response apart.

A Link header, part by part.

The response headers from RFC 9727’s section 3 example, the home page of its fictional publisher “example”. The HTML body that follows them is left out.

```http
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Location: /index.html
Link: </my_api_catalog.json>; rel=api-catalog
Content-Length: 356
```

Its parts:

- Content-Type: text/html; charset=UTF-8. In our words: This answer is an ordinary web page. The link to the API catalog travels with it, so a program that asked for the home page can learn about the catalog without asking for it. What the source says: RFC 9727: the Publisher “could include the api-catalog link relation in the HTTP header and/or content payload when responding to a request to https\://www\.example.com”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- Link. In our words: The header that carries links. One Link header can carry more than one. What the source says: RFC 8288: “The Link header field provides a means for serialising one or more links into HTTP headers.” A link has “a link context, a link relation type, a link target, and optionally, target attributes”. Source: [RFC 8288: Web Linking](https://www.rfc-editor.org/rfc/rfc8288).
- \</my_api_catalog.json>. In our words: Where the link points, inside angle brackets. This one is a path on the same site, so a program reads it against the page’s own address. RFC 9727 lets the publisher keep its catalog at any address it chooses. What the source says: RFC 8288: each link-value “conveys one target IRI as a URI-Reference … inside angle brackets”, and “If the URI-Reference is relative, parsers MUST resolve it”. RFC 9727: “The target resource URI may be /.well-known/api-catalog or any other URI chosen by the Publisher.” Sources: [RFC 8288: Web Linking](https://www.rfc-editor.org/rfc/rfc8288) and [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- rel=api-catalog. In our words: What kind of link this is. The value api-catalog is a registered relation type, so a program that knows RFC 9727 knows the target is a list of APIs. Writing rel="api-catalog" with quotes means the same. What the source says: RFC 8288: “The rel parameter MUST be present but MUST NOT appear more than once in a given link-value”, and recipients “MUST be able to parse both forms”, x=y and x="y". IANA: api-catalog “Refers to a list of APIs available from the Publisher of the link context.” Sources: [RFC 8288: Web Linking](https://www.rfc-editor.org/rfc/rfc8288) and [IANA Link Relations registry (CSV)](https://www.iana.org/assignments/link-relations/link-relations-1.csv).

The link has no anchor parameter, so its context is the page it came with: RFC 8288 says the context is by default “the URL of the representation it is associated with”, here https\://www\.example.com. RFC 9727 does not show the request that produced this answer, so it does not say which method was used. It separately requires that a HEAD request to /.well-known/api-catalog gets an answer with a Link header. Sources: [RFC 8288: Web Linking](https://www.rfc-editor.org/rfc/rfc8288) and [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).

The RFC shows only that response, so it does not say whether the request was a GET or a HEAD (a request for the headers alone). For the well-known address it is specific: a publisher "SHALL resolve an HTTPS HEAD request to /.well-known/api-catalog with a response including a Link header".

## API catalogs

[RFC 9727](https://www.rfc-editor.org/rfc/rfc9727), a Standards Track RFC of June 2025, "defines the "api-catalog" well-known URI and link relation. It is intended to facilitate automated discovery and usage of published Application Programming Interfaces (APIs)." A publisher "SHALL resolve an HTTPS GET request to /.well-known/api-catalog and return an API catalog document". The catalog file can live elsewhere: its location "is decided by the Publisher", and the well-known address "provides a convenient reference to that location".

The catalog is JSON (a plain-text format for structured data) in a format called Linkset, a list of links written as JSON. "The Publisher MUST publish the API catalog document in the Linkset format application/linkset+json", and the document "SHOULD include a profile parameter" whose value is the RFC's own address. "The API catalog MUST include hyperlinks to API endpoints." Beyond that, "It is RECOMMENDED that the API catalog also includes useful metadata, such as usage policies, API version information, links to the OpenAPI Specification \[OAS] definitions for each API". The RFC does not tell publishers where to put the APIs themselves: it "does not mandate paths for API endpoints". A catalog can also point to other catalogs with the `api-catalog` relation.

The figure is the example in the RFC's Appendix A.2, a catalog that lists three APIs as items.

An API catalog, key by key.

The example in RFC 9727, Appendix A.2, as the RFC prints it, with the response’s Date and Server headers left out. The Content-Type header is one line, split here as the RFC splits it.

```http
GET .well-known/api-catalog HTTP/1.1
Host: example.com
Accept: application/linkset+json

HTTP/1.1 200 OK
Content-Type: application/linkset+json;
    profile="https://www.rfc-editor.org/info/rfc9727"

{ "linkset":
 [
   { "anchor": "https://www.example.com/.well-known/api-catalog",
     "item": [
       {"href": "https://developer.example.com/apis/foo_api"},
       {"href": "https://developer.example.com/apis/bar_api"},
       {"href": "https://developer.example.com/apis/cantona_api"}
     ]
   }
 ]
}
```

Its parts:

- Accept: application/linkset+json. In our words: The client says which format it wants back: the Linkset format the RFC requires for catalogs. What the source says: RFC 9727, Appendix A.2, client request: “GET .well-known/api-catalog HTTP/1.1”, “Accept: application/linkset+json”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- application/linkset+json. In our words: The media type of the answer. A catalog in another format is allowed only alongside a Linkset. What the source says: RFC 9727: “The Publisher MUST publish the API catalog document in the Linkset format application/linkset+json”. A publisher with another format “MUST also implement a Linkset with, at minimum, hyperlinks to API endpoints”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- profile="https\://www\.rfc-editor.org/info/rfc9727". In our words: A label saying this Linkset is an API catalog in RFC 9727’s sense, and not some other list of links. What the source says: RFC 9727: “The Linkset SHOULD include a profile parameter … with a Profile URI \[RFC7284] value of "https\://www\.rfc-editor.org/info/rfc9727"”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- "linkset". In our words: The one member at the top: a list of link groups. In this example there is one group. What the source says: RFC 9727: the catalog is published “in the Linkset format application/linkset+json (Section 4.2 of \[RFC9264])”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- "anchor". In our words: What the links in this group are about, their context. Here it is the catalog itself. What the source says: RFC 9727, A.2: “Each link shares the same context anchor (the well-known URI of the API catalog)”. RFC 8288: the anchor parameter overrides the default link context. Sources: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727) and [RFC 8288: Web Linking](https://www.rfc-editor.org/rfc/rfc8288).
- "item". In our words: The relation type of the links that follow: each one is an API in the catalog. What the source says: RFC 9727: “the "item" \[RFC6573] link relation identifies a target resource that represents an API that is a member of the API catalog.” Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).
- "href". In our words: The address of one API. This is the part a catalog cannot do without; the RFC recommends more, such as usage policies and versions. What the source says: RFC 9727: “The API catalog MUST include hyperlinks to API endpoints. It is RECOMMENDED that the API catalog also includes useful metadata, such as usage policies, API version information, links to the OpenAPI Specification \[OAS] definitions for each API”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).

Appendix A is informative: it shows ways to write a catalog, not the only way. Its first example, A.1, gives each API its own group, with service-desc for “a description of the API that is primarily intended for machine consumption”, service-doc for documentation “primarily intended for human consumption” and status for the API’s status. Where the catalog file itself lives “is decided by the Publisher”; /.well-known/api-catalog “provides a convenient reference to that location”. Source: [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727).

In our reading, an API catalog tells an agent where a publisher's APIs are and, if the publisher adds it, how to use them. Nothing in RFC 9727 is specific to shopping; the word commerce appears in it only inside an example host name. The commerce protocols define documents of their own.

## Commerce protocols, agent cards and server cards

[UCP](https://ucp.dev/latest/specification/overview/) puts its document at a well-known address: "Businesses publish their profile at /.well-known/ucp." The profile "is a JSON object with a required `ucp` member" holding the "protocol version, services, optional capabilities, and payment handlers", and "The services and payment_handlers registries MUST be present even when empty". Platforms (in our reading, UCP's word for the agent's side) "MUST resolve schemas following this sequence", and its first step is "Discovery: Fetch business profile from /.well-known/ucp". The current UCP release on GitHub is `v2026-08-25`, marked "Latest". [UCP explained](https://storewalk.ai/blog/ucp-explained) goes through the profile field by field.

ACP's discovery file exists only as a proposal. `/.well-known/acp.json` is defined in a [discovery RFC](https://github.com/agentic-commerce-protocol/agentic-commerce-protocol/blob/main/rfcs/rfc.discovery.md) marked "Status: Proposal" and "Version: unreleased", and it is not part of the released `2026-04-17` specification. The proposal says the document would let agents "determine whether a seller supports ACP, which protocol version is available, what transports are offered, and what capabilities the seller provides, all before creating a checkout session and without requiring authentication". It "MUST be served at the origin root per RFC 8615", and the proposal also describes a seller platform (its example is Stripe) hosting one document for many sellers. Payment methods are left out of it and negotiated per checkout session. The proposal also lists "Discovering what a merchant sells (products, inventory, pricing)" as "out of scope".

Two protocols that are not about commerce have discovery documents too. [A2A](https://a2a-protocol.org/latest/specification/), whose latest released version is 1.0.0, is "an open standard designed to facilitate communication and interoperability between independent, potentially opaque AI agent systems." Its agent card is "A self-describing manifest for an agent", with "the agent's identity, capabilities, skills, supported communication methods, and security requirements". The specification lists three ways to find one: the well-known URI `https://{server_domain}/.well-known/agent-card.json`, "Registries/Catalogs" and "Direct Configuration". `agent-card.json` is the one name in this section that IANA's registry lists.

The core [MCP specification](https://modelcontextprotocol.io/specification/latest) defines no discovery file. MCP server cards come from [SEP-2127](https://modelcontextprotocol.io/seps/2127-mcp-server-cards), an optional extension marked "Final", which offers "a standardized, self-contained format to describe MCP servers for pre-connection discovery". The cards "intentionally" leave out tools, resources and prompts. The extension's [discovery document](https://github.com/modelcontextprotocol/ext-server-card/blob/main/docs/discovery.md), marked "Protocol Revision: draft", says hosts "MAY publish" an AI Catalog at `/.well-known/ai-catalog.json`, and "Clients performing domain-level discovery SHOULD attempt to retrieve this well-known URL." Where MCP, A2A and WebMCP each fit a store is the subject of [MCP, A2A and WebMCP for stores](https://storewalk.ai/blog/mcp-a2a-webmcp-for-stores).

The map adds robots.txt, the API catalog and the Link header to these addresses. None of the documents cited here sets an order across these addresses; the map's order is ours.

What an agent can ask a store for, address by address.

The figure asks in an order of our choosing.

- GET /robots.txt (Crawl rules): Which paths each named crawler is asked not to fetch, grouped by product token. The rules are a request, not access control. Status: RFC 9309, Standards Track. Source: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).
- GET /.well-known/ucp (UCP business profile): The business’s UCP profile: protocol version, services, optional capabilities and payment handlers. Status: UCP release v2026-08-25, marked Latest. Not in the IANA well-known URI registry on 2026-10-22. Source: [UCP overview](https://ucp.dev/latest/specification/overview/). Status source: [UCP releases](https://github.com/Universal-Commerce-Protocol/ucp/releases).
- GET /.well-known/acp.json (ACP discovery): Whether the seller supports ACP, which version, which transports and which capabilities, before a checkout session and without signing in. Status: Proposal, version unreleased. Not in the IANA well-known URI registry on 2026-10-22. Source: [ACP discovery RFC](https://github.com/agentic-commerce-protocol/agentic-commerce-protocol/blob/main/rfcs/rfc.discovery.md).
- GET /.well-known/api-catalog (API catalog): A list of the publisher’s APIs, as a Linkset (application/linkset+json) with links to each API. Status: RFC 9727, Standards Track. In the IANA well-known URI registry on 2026-10-22. Source: [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727).
- GET / (Link header on the home page): A Link header on an ordinary response. With rel="api-catalog" it points to the publisher’s list of APIs, wherever the publisher keeps it. RFC 9727 does not state the method; GET is our choice. Status: RFC 9727 and RFC 8288, Standards Track. api-catalog is a registered link relation. Example: Link: \</my_api_catalog.json>; rel=api-catalog. Source: [RFC 9727, section 3](https://www.rfc-editor.org/rfc/rfc9727). Status source: [RFC 8288](https://www.rfc-editor.org/rfc/rfc8288).
- GET /.well-known/agent-card.json (A2A agent card): An A2A agent card: the agent’s identity, capabilities, skills, supported communication methods and security requirements. Status: A2A, latest released version 1.0.0. In the IANA well-known URI registry on 2026-10-22. Source: [A2A specification](https://a2a-protocol.org/latest/specification/).
- GET /.well-known/ai-catalog.json (AI Catalog with MCP server cards): An AI Catalog listing the host’s MCP server cards, which describe a server before connecting. Clients doing domain-level discovery should try this address. Cards carry no tool list. Status: MCP extension SEP-2127, Final, optional; its discovery doc is marked draft. Not in the IANA well-known URI registry on 2026-10-22. Source: [Server-card extension: discovery](https://github.com/modelcontextprotocol/ext-server-card/blob/main/docs/discovery.md). Status source: [SEP-2127](https://modelcontextprotocol.io/seps/2127-mcp-server-cards).

## Content Signals and AIPREF: statements about use

A robots.txt file holds rules "that crawlers are requested to honor", grouped by the crawler's name, and [Bots, crawlers and agents](https://storewalk.ai/blog/bots-crawlers-agents) covers how they work. Those rules are about fetching. Two newer vocabularies add lines about what a page may be used for.

The first is Cloudflare's. Its [managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) documentation, last updated 22 October 2026, says "Content Signals are a set of machine-readable directives in a robots.txt file that categorize how crawlers may use your content. The three categories are search (building a search index), ai-input (feeding content into AI models for real-time answers), and ai-train (training or fine-tuning AI models)." The policy text on that page reads: "If a content-signal = yes, you may collect content for the corresponding use", "If a content-signal = no, you may not collect content for the corresponding use", and where a signal is missing the operator "neither grants nor restricts permission". The same text describes restrictions expressed this way as "EXPRESS RESERVATIONS OF RIGHTS" under Article 4 of the EU's Directive 2019/790 on copyright. Cloudflare is also "testing" a fourth field, `use`, with three values from `immediate` to `full`. The [Content Signals site](https://contentsignals.org/) shows no version or date. Cloudflare's sample response for its Markdown for Agents feature, covered in [How an AI agent reads a web page](https://storewalk.ai/blog/how-agents-read-a-page), carries a `content-signal` header with the same three category names.

The second is the IETF's. Its AIPREF working group has two Internet-Drafts, working texts that can still change and may be cited only as work in progress. [draft-ietf-aipref-vocab-08](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt), dated 14 September 2026, defines three categories: "AI Training | train-ai", "AI Use | ai-use" and "Search | search". AI Training is "Using an asset to modify the learned parameters of a generative AI model". AI Use is "Using an asset as input to a generative AI model, where the asset is not directly provided by the user". Search "does not include the use of assets to generate summaries" and "overrides any usage that falls into other categories". Each category is allowed or disallowed, written `y` or `n`, and a category the statement leaves out is "unknown". [draft-ietf-aipref-attach-05](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt), dated 19 August 2026, attaches those statements to content "using the Robots Exclusion Protocol and HTTP header fields", as a `Content-Usage` line or header, and would update the robots.txt RFC if approved. Both are working-group documents intended for Standards Track. The vocabulary draft adds that its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part."

The attachment draft separates the two questions in a robots.txt group: "Acquisition relies on Allow/Disallow rules; usage preference relies on Content-Usage rules." Usage preferences "apply only to those resources that can be crawled". Compare the two sets of lines here.

Two ways to state how content may be used, in robots.txt.

Cloudflare Content Signals. The managed lines Cloudflare adds to a robots.txt, from its managed robots.txt page.

```text
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
```

- Content-signal. In our words: A line inside a robots.txt group that says for which uses the content may be collected or used. What the source says: Cloudflare: “Content Signals are a set of machine-readable directives in a robots.txt file that categorize how crawlers may use your content.” Source: [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/).
- search=yes. In our words: Yes to search: building a search index and showing results with links and short excerpts. What the source says: Cloudflare’s policy text: “search: building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website’s contents). Search does not include providing AI-generated search summaries.” “If a content-signal = yes, you may collect content for the corresponding use.” Source: [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/).
- ai-train=no. In our words: No to training AI models on the content. The line says nothing about ai-input, the third category, so for that one the file neither grants nor restricts. What the source says: Cloudflare’s policy text: “ai-train: training or fine-tuning AI models.” “If a content-signal = no, you may not collect content for the corresponding use.” “If the website operator does not include a content signal for a corresponding use, the website operator neither grants nor restricts permission via content signal with respect to the corresponding use.” Source: [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/).
- use=reference. In our words: A fourth field that Cloudflare is testing: what a crawler may keep and reuse after it has the content. Reference is the middle of three values. What the source says: Cloudflare: “Cloudflare is testing content-use, an optional extension to Content Signals”. Values: use=immediate “Interact, but store and reuse nothing.”, use=reference “Index, excerpt, and link back.”, use=full “Summarize and reproduce.” Source: [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/).

Status: a vendor’s documentation, last updated 22 October 2026. The Content Signals site shows no version or date, and content-use is a test. Sources: [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) and [contentsignals.org](https://contentsignals.org/).

IETF AIPREF drafts. The robots.txt example from the AIPREF attachment draft, section 1.2.

```text
User-Agent: *
Allow: /
Content-Usage: train-ai=n
```

- Allow: /. In our words: The ordinary robots.txt rule: crawlers in this group may fetch everything. In the draft, fetching and use are two separate questions. What the source says: AIPREF attachment draft: “This creates a two-stage arrangement that distinguishes acquisition and usage. Acquisition relies on Allow/Disallow rules; usage preference relies on Content-Usage rules.” “Usage preferences apply only to those resources that can be crawled”. Source: [draft-ietf-aipref-attach-05 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt).
- Content-Usage. In our words: The draft’s line for usage preferences. The same words can also travel as an HTTP header on a single response. What the source says: AIPREF attachment draft: “This document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields.” Header example: “Content-Usage: train-ai=n”. Source: [draft-ietf-aipref-attach-05 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt).
- train-ai=n. In our words: No to AI training. Categories the line leaves out, AI use and search, stay unknown. What the source says: AIPREF vocabulary draft: AI Training is “Using an asset to modify the learned parameters of a generative AI model.” Allow and disallow “are mapped to single byte Tokens … of y and n, respectively.” Its own example allows training and disallows search, “with the preference for other categories being unknown”. Source: [draft-ietf-aipref-vocab-08 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt).

Status: two Internet-Drafts of the IETF’s AIPREF working group, draft-ietf-aipref-vocab-08 (14 September 2026) and draft-ietf-aipref-attach-05 (19 August 2026), both intended for Standards Track. The vocabulary draft says its contents “DO NOT REFLECT CONSENSUS of the Working Group either in whole or part.” Sources: [draft-ietf-aipref-vocab-08 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt), [draft-ietf-aipref-attach-05 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt) and [IETF AIPREF working group documents](https://datatracker.ietf.org/wg/aipref/documents/).

Neither kind of line stops a crawler. The AIPREF vocabulary draft says “Preferences are not a security mechanism.” and that a recipient “has a choice whether to follow those preferences”. Cloudflare’s page says “robots.txt compliance is voluntary”. Sources: [draft-ietf-aipref-vocab-08 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt) and [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/).

Cloudflare writes `ai-train` and `ai-input` with `yes` and `no`; the drafts write `train-ai` and `ai-use` with `y` and `n`. Neither kind of line stops a crawler. The vocabulary draft says it does not "ensure that preferences are followed", that a recipient "has a choice whether to follow those preferences", and that "Preferences are not a security mechanism." The Content Signals site says that "some may ignore these Content-Signal directives."

## What you can check today

Replace `your-domain` with your store's domain in each of these.

- Open `https://your-domain/.well-known/ucp` in a browser. A JSON document with a `ucp` member means something already publishes a UCP profile for your store.
- Open `https://your-domain/.well-known/api-catalog`. If your store publishes an API catalog under RFC 9727, the RFC requires a Linkset here (`application/linkset+json`).
- Run `curl -I https://your-domain/.well-known/api-catalog`, which sends a HEAD request. Under RFC 9727, a publisher that supports the address answers with a `Link` header.
- Run `curl -s -D - -o /dev/null https://your-domain/` to print the headers of your home page, and look for `Link:` lines and their `rel` values.
- If you run an A2A agent or an MCP server, open `https://your-domain/.well-known/agent-card.json` or `https://your-domain/.well-known/ai-catalog.json`.
- Open `https://your-domain/robots.txt` and look for `Content-signal` or `Content-Usage` lines. If your site is on Cloudflare, its managed robots.txt documentation shows the lines it adds.
- Ask your store platform which of these documents it publishes for you, and at which addresses.

## What is not settled

- ACP's `/.well-known/acp.json` is a proposal, version unreleased, outside the released specification.
- MCP's core specification defines no discovery file. Server cards come from an optional extension marked Final, and the document that gives the AI Catalog's address is marked draft.
- On 22 October 2026 IANA's registry did not list `ucp`, `acp.json`, `ai-catalog.json` or `http-message-signatures-directory`, although RFC 8615 says new well-known URIs "MUST" be registered.
- RFC 9727's home-page example shows a Link header without the request that produced it, so the method is not stated.
- AIPREF's two documents are Internet-Drafts, and the vocabulary draft says its contents do not reflect the working group's consensus. Content Signals is a vendor's vocabulary with no published version, and its `use` field is a test. The two name their categories differently.
- The AIPREF vocabulary draft leaves the choice to follow a preference with the recipient, and Cloudflare calls robots.txt compliance voluntary.

Finding what the store offers is the third wall in [the hub](https://storewalk.ai/blog/why-agents-cant-buy). The fourth, getting through checkout, is the subject of [Why agents stall at checkout](https://storewalk.ai/blog/why-agents-stall-at-checkout).

## Sources

- [RFC 8615: Well-Known Uniform Resource Identifiers (URIs)](https://www.rfc-editor.org/rfc/rfc8615)
- [IANA Well-Known URIs registry (fetched 2026-10-22)](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml)
- [RFC 8288: Web Linking](https://www.rfc-editor.org/rfc/rfc8288)
- [IANA Link Relations registry, CSV (fetched 2026-10-22)](https://www.iana.org/assignments/link-relations/link-relations-1.csv)
- [RFC 9727: api-catalog](https://www.rfc-editor.org/rfc/rfc9727)
- [RFC 8414: OAuth 2.0 Authorization Server Metadata](https://www.rfc-editor.org/rfc/rfc8414)
- [RFC 9728: OAuth 2.0 Protected Resource Metadata](https://www.rfc-editor.org/rfc/rfc9728)
- [UCP specification overview](https://ucp.dev/latest/specification/overview/)
- [UCP releases on GitHub](https://github.com/Universal-Commerce-Protocol/ucp/releases)
- [ACP RFC: Discovery (Proposal, unreleased)](https://github.com/agentic-commerce-protocol/agentic-commerce-protocol/blob/main/rfcs/rfc.discovery.md)
- [A2A specification](https://a2a-protocol.org/latest/specification/)
- [MCP specification](https://modelcontextprotocol.io/specification/latest)
- [MCP SEP-2127: Server Cards](https://modelcontextprotocol.io/seps/2127-mcp-server-cards)
- [MCP server-card extension: Discovery](https://github.com/modelcontextprotocol/ext-server-card/blob/main/docs/discovery.md)
- [IETF datatracker: draft-ietf-webbotauth-httpsig-protocol](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)
- [draft-ietf-aipref-vocab-08: A Vocabulary For Expressing AI Usage Preferences](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt)
- [draft-ietf-aipref-attach-05: Associating AI Usage Preferences with Content in HTTP](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt)
- [IETF AIPREF working group documents](https://datatracker.ietf.org/wg/aipref/documents/)
- [Cloudflare docs: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- [contentsignals.org (fetched 2026-10-22)](https://contentsignals.org/)

Last checked 22 Oct 2026