How agents discover what a store offers

The IETF (the Internet Engineering Task Force) develops internet standards and publishes them as numbered documents called RFCs. Its RFC on well-known addresses opens: "Some applications on the Web require the discovery of information about an origin … (sometimes called "site-wide metadata") before making a request." An origin is a site as a browser sees it, such as https://example.com. Its RFC on API catalogs says of APIs (documented ways for programs to talk to a service): "Such APIs must be discovered before they may be used, i.e., the external party needs to know what APIs a given Publisher exposes, their purpose, any policies for usage, and the endpoint to interact with each API."

ACP's discovery proposal calls it "a bootstrapping problem: agents need to learn the API base URL before they can call any endpoint." Its answer is that "Agents only need a domain".

In our reading, a site can get an agent from its domain to that information with a document at an address the agent already knows, a pointer added to an ordinary response, or lines in robots.txt.

A URI is an address on the web, such as https://example.com/robots.txt. RFC 8615, "Well-Known Uniform Resource Identifiers (URIs)", is a Standards Track RFC of May 2019, a document on the IETF's path to becoming a standard. It "defines a path prefix for "well-known locations", "/.well-known/"". The prefix sits at the top of the site: "/.well-known/example" is a well-known URI, "whereas "/foo/.well-known/example" is not."

The RFC defines only the prefix. It "does not define a format or media type for the resource located at "/.well-known/"", and clients "should not expect a resource to exist at that location". A media type is the label that says what format a response is in, such as application/json. Each application defines its own name and document. In our reading, there is no index to read first: an agent has to know each name in advance.

RFC 8615 says "Applications that wish to mint new well-known URIs MUST register them", in a registry kept by IANA, the body that keeps the internet's official lists of names and numbers. Names are "registered on the advice of one or more experts, with a Specification Required", and a registration references "a specification that defines the format and associated media type(s)". The RFC also asks applications to choose precise names: "squatting" on generic terms "is not encouraged".

We read the registry on 22 October 2026; IANA had last updated it on 7 October. Pick a path to see what it said.

The registry lists api-catalog, the two OAuth metadata paths (OAuth is a way for one program to get limited access to an account held elsewhere), and agent-card.json, registered for A2A on 1 August 2025 with the Linux Foundation as its change controller, the party responsible for the entry. It does not list ucp, acp.json, ai-catalog.json or http-message-signatures-directory; the Web Bot Auth draft asks for the last of these in its IANA section. A missing entry does not stop anyone serving the path. UCP's specification says "Businesses publish their profile at /.well-known/ucp", and that name is not in the registry.

HTTP is the protocol browsers and servers use to exchange pages, and headers are the named lines at the top of each message. Any response can carry a pointer in its headers. RFC 8288, "Web Linking", is a Standards Track RFC of October 2017. It defines "a model for the relationships between resources on the Web ("links")" and "the serialisation of such links in HTTP headers with the Link header field".

A link has "a link context, a link relation type, a link target, and optionally, target attributes", and the RFC reads it as a sentence: "link context has a link relation type resource at link target". Its own example is Link: <http://example.com/TheBook/chapter2>; rel="previous". The relation type is the word that says what kind of link it is, and IANA keeps a registry of those too. On 22 October 2026 it held api-catalog, which "Refers to a list of APIs available from the Publisher of the link context", along with alternate, describedby, item and service-desc. It held no llms-txt, mcp or sitemap.

RFC 9727 uses a Link header to point at an API catalog from an ordinary page. A publisher "could include the api-catalog link relation in the HTTP header and/or content payload when responding to a request to https://www.example.com". In our reading, a program that only asked for the home page can learn from the answer where the catalog is. The figure takes the RFC's example response apart.

HTTP/1.1 200 OKContent-Type: text/html; charset=UTF-8Location: /index.htmlLink: </my_api_catalog.json>; rel=api-catalogContent-Length: 356

The RFC shows only that response, so it does not say whether the request was a GET or a HEAD (a request for the headers alone). For the well-known address it is specific: a publisher "SHALL resolve an HTTPS HEAD request to /.well-known/api-catalog with a response including a Link header".

RFC 9727, a Standards Track RFC of June 2025, "defines the "api-catalog" well-known URI and link relation. It is intended to facilitate automated discovery and usage of published Application Programming Interfaces (APIs)." A publisher "SHALL resolve an HTTPS GET request to /.well-known/api-catalog and return an API catalog document". The catalog file can live elsewhere: its location "is decided by the Publisher", and the well-known address "provides a convenient reference to that location".

The catalog is JSON (a plain-text format for structured data) in a format called Linkset, a list of links written as JSON. "The Publisher MUST publish the API catalog document in the Linkset format application/linkset+json", and the document "SHOULD include a profile parameter" whose value is the RFC's own address. "The API catalog MUST include hyperlinks to API endpoints." Beyond that, "It is RECOMMENDED that the API catalog also includes useful metadata, such as usage policies, API version information, links to the OpenAPI Specification [OAS] definitions for each API". The RFC does not tell publishers where to put the APIs themselves: it "does not mandate paths for API endpoints". A catalog can also point to other catalogs with the api-catalog relation.

The figure is the example in the RFC's Appendix A.2, a catalog that lists three APIs as items.

GET .well-known/api-catalog HTTP/1.1Host: example.comAccept: application/linkset+jsonHTTP/1.1 200 OKContent-Type: application/linkset+json;    profile="https://www.rfc-editor.org/info/rfc9727"{ "linkset": [   { "anchor": "https://www.example.com/.well-known/api-catalog",     "item": [       {"href": "https://developer.example.com/apis/foo_api"},       {"href": "https://developer.example.com/apis/bar_api"},       {"href": "https://developer.example.com/apis/cantona_api"}     ]   } ]}

In our reading, an API catalog tells an agent where a publisher's APIs are and, if the publisher adds it, how to use them. Nothing in RFC 9727 is specific to shopping; the word commerce appears in it only inside an example host name. The commerce protocols define documents of their own.

UCP puts its document at a well-known address: "Businesses publish their profile at /.well-known/ucp." The profile "is a JSON object with a required ucp member" holding the "protocol version, services, optional capabilities, and payment handlers", and "The services and payment_handlers registries MUST be present even when empty". Platforms (in our reading, UCP's word for the agent's side) "MUST resolve schemas following this sequence", and its first step is "Discovery: Fetch business profile from /.well-known/ucp". The current UCP release on GitHub is v2026-08-25, marked "Latest". UCP explained goes through the profile field by field.

ACP's discovery file exists only as a proposal. /.well-known/acp.json is defined in a discovery RFC marked "Status: Proposal" and "Version: unreleased", and it is not part of the released 2026-04-17 specification. The proposal says the document would let agents "determine whether a seller supports ACP, which protocol version is available, what transports are offered, and what capabilities the seller provides, all before creating a checkout session and without requiring authentication". It "MUST be served at the origin root per RFC 8615", and the proposal also describes a seller platform (its example is Stripe) hosting one document for many sellers. Payment methods are left out of it and negotiated per checkout session. The proposal also lists "Discovering what a merchant sells (products, inventory, pricing)" as "out of scope".

Two protocols that are not about commerce have discovery documents too. A2A, whose latest released version is 1.0.0, is "an open standard designed to facilitate communication and interoperability between independent, potentially opaque AI agent systems." Its agent card is "A self-describing manifest for an agent", with "the agent's identity, capabilities, skills, supported communication methods, and security requirements". The specification lists three ways to find one: the well-known URI https://{server_domain}/.well-known/agent-card.json, "Registries/Catalogs" and "Direct Configuration". agent-card.json is the one name in this section that IANA's registry lists.

The core MCP specification defines no discovery file. MCP server cards come from SEP-2127, an optional extension marked "Final", which offers "a standardized, self-contained format to describe MCP servers for pre-connection discovery". The cards "intentionally" leave out tools, resources and prompts. The extension's discovery document, marked "Protocol Revision: draft", says hosts "MAY publish" an AI Catalog at /.well-known/ai-catalog.json, and "Clients performing domain-level discovery SHOULD attempt to retrieve this well-known URL." Where MCP, A2A and WebMCP each fit a store is the subject of MCP, A2A and WebMCP for stores.

The map adds robots.txt, the API catalog and the Link header to these addresses. None of the documents cited here sets an order across these addresses; the map's order is ours.

A robots.txt file holds rules "that crawlers are requested to honor", grouped by the crawler's name, and Bots, crawlers and agents covers how they work. Those rules are about fetching. Two newer vocabularies add lines about what a page may be used for.

The first is Cloudflare's. Its managed robots.txt documentation, last updated 22 October 2026, says "Content Signals are a set of machine-readable directives in a robots.txt file that categorize how crawlers may use your content. The three categories are search (building a search index), ai-input (feeding content into AI models for real-time answers), and ai-train (training or fine-tuning AI models)." The policy text on that page reads: "If a content-signal = yes, you may collect content for the corresponding use", "If a content-signal = no, you may not collect content for the corresponding use", and where a signal is missing the operator "neither grants nor restricts permission". The same text describes restrictions expressed this way as "EXPRESS RESERVATIONS OF RIGHTS" under Article 4 of the EU's Directive 2019/790 on copyright. Cloudflare is also "testing" a fourth field, use, with three values from immediate to full. The Content Signals site shows no version or date. Cloudflare's sample response for its Markdown for Agents feature, covered in How an AI agent reads a web page, carries a content-signal header with the same three category names.

The second is the IETF's. Its AIPREF working group has two Internet-Drafts, working texts that can still change and may be cited only as work in progress. draft-ietf-aipref-vocab-08, dated 14 September 2026, defines three categories: "AI Training | train-ai", "AI Use | ai-use" and "Search | search". AI Training is "Using an asset to modify the learned parameters of a generative AI model". AI Use is "Using an asset as input to a generative AI model, where the asset is not directly provided by the user". Search "does not include the use of assets to generate summaries" and "overrides any usage that falls into other categories". Each category is allowed or disallowed, written y or n, and a category the statement leaves out is "unknown". draft-ietf-aipref-attach-05, dated 19 August 2026, attaches those statements to content "using the Robots Exclusion Protocol and HTTP header fields", as a Content-Usage line or header, and would update the robots.txt RFC if approved. Both are working-group documents intended for Standards Track. The vocabulary draft adds that its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part."

The attachment draft separates the two questions in a robots.txt group: "Acquisition relies on Allow/Disallow rules; usage preference relies on Content-Usage rules." Usage preferences "apply only to those resources that can be crawled". Compare the two sets of lines here.

User-Agent: *Content-signal: search=yes, ai-train=no, use=referenceAllow: /

User-Agent: *Allow: /Content-Usage: train-ai=n

Cloudflare writes ai-train and ai-input with yes and no; the drafts write train-ai and ai-use with y and n. Neither kind of line stops a crawler. The vocabulary draft says it does not "ensure that preferences are followed", that a recipient "has a choice whether to follow those preferences", and that "Preferences are not a security mechanism." The Content Signals site says that "some may ignore these Content-Signal directives."

Replace your-domain with your store's domain in each of these.

• Open https://your-domain/.well-known/ucp in a browser. A JSON document with a ucp member means something already publishes a UCP profile for your store.
• Open https://your-domain/.well-known/api-catalog. If your store publishes an API catalog under RFC 9727, the RFC requires a Linkset here (application/linkset+json).
• Run curl -I https://your-domain/.well-known/api-catalog, which sends a HEAD request. Under RFC 9727, a publisher that supports the address answers with a Link header.
• Run curl -s -D - -o /dev/null https://your-domain/ to print the headers of your home page, and look for Link: lines and their rel values.
• If you run an A2A agent or an MCP server, open https://your-domain/.well-known/agent-card.json or https://your-domain/.well-known/ai-catalog.json.
• Open https://your-domain/robots.txt and look for Content-signal or Content-Usage lines. If your site is on Cloudflare, its managed robots.txt documentation shows the lines it adds.
• Ask your store platform which of these documents it publishes for you, and at which addresses.

• ACP's /.well-known/acp.json is a proposal, version unreleased, outside the released specification.
• MCP's core specification defines no discovery file. Server cards come from an optional extension marked Final, and the document that gives the AI Catalog's address is marked draft.
• On 22 October 2026 IANA's registry did not list ucp, acp.json, ai-catalog.json or http-message-signatures-directory, although RFC 8615 says new well-known URIs "MUST" be registered.
• RFC 9727's home-page example shows a Link header without the request that produced it, so the method is not stated.
• AIPREF's two documents are Internet-Drafts, and the vocabulary draft says its contents do not reflect the working group's consensus. Content Signals is a vendor's vocabulary with no published version, and its use field is a test. The two name their categories differently.
• The AIPREF vocabulary draft leaves the choice to follow a preference with the recipient, and Cloudflare calls robots.txt compliance voluntary.

Finding what the store offers is the third wall in the hub. The fourth, getting through checkout, is the subject of Why agents stall at checkout.