# Bots, crawlers and agents: who is knocking

A User-Agent string can be spoofed, and robots.txt rules are a request to crawlers rather than access authorization. Web Bot Auth, an IETF draft, would let an agent sign its requests so a server can verify who sent them.

By Valerii Checha, 14 Oct 2026

Every request that reaches your store comes from a program, and the request can carry that program's name. A browser names itself, and so do the crawlers and agents of the AI companies we checked. Your server, or a service in front of it, then has to decide what to believe and what to do. This article covers the ways a visitor can name itself and what each one shows.

It belongs to a series. [The hub article](https://storewalk.ai/blog/why-agents-cant-buy) sets out five walls between an agent and a finished purchase, and the second is proving who the agent is. The first, reading the page, is covered in [How an AI agent reads a web page](https://storewalk.ai/blog/how-agents-read-a-page). What happens when a bot check stops an agent at checkout is the subject of [Why agents stall at checkout](https://storewalk.ai/blog/why-agents-stall-at-checkout).

Where we explain a term in plain words, the explanation is ours, not any specification's. Everything else comes from the documents linked in the text and listed under sources.

## Crawlers, search bots and agents

Three of the AI companies we checked, OpenAI, Anthropic and Perplexity, sort their bots by purpose. Their pages, as they read on 22 October 2026, describe them like this:

- Training crawlers. OpenAI's [GPTBot](https://developers.openai.com/api/docs/bots) "is used to crawl content that may be used in training our generative AI foundation models". Anthropic's [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) collects "web content that could potentially contribute to their training".
- Search crawlers. OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features". Claude-SearchBot "navigates the web to improve search result quality for users". [PerplexityBot](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."
- Fetchers that act on a user's request. OpenAI: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent", and "ChatGPT-User is not used for crawling the web in an automatic fashion." Anthropic: "When individuals ask questions to Claude, it may access websites using a Claude-User agent". Perplexity, of Perplexity-User: "When users ask Perplexity a question, it might visit a web page to help provide an accurate answer".

Google documents one more. Its page on [user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), last updated 19 August 2026, says: "Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Cloudflare has a similar split in its [verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) documentation, with an "Agent" category for "User-directed agents visiting a page on behalf of a human" and a "Transact" category for "Checkout or other transaction actions on behalf of users".

In our reading, a shopping agent is the last kind: a program that visits because one person asked it to, at the moment they asked. A crawler visits on its own schedule. A store may want to treat the two differently, if it can tell them apart.

## What a User-Agent string says

HTTP is the protocol browsers and servers use to exchange pages. An HTTP request is the message a client sends to ask for one, and its headers are the named lines at the top of that message. One of them is `User-Agent`. The IETF (the Internet Engineering Task Force) develops internet standards and publishes them as numbered documents called RFCs. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110), its standard for HTTP semantics (June 2022), says the header "contains information about the user agent originating the request".

The value has a grammar: "one or more product identifiers, each followed by zero or more comments", where "Each product identifier consists of a name and optional version." The figure takes the full string OpenAI publishes for ChatGPT-User.

A User-Agent string, part by part.

The full User-Agent string OpenAI publishes for ChatGPT-User, copied from its crawler page on 22 October 2026.

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
```

Its parts:

- Mozilla/5.0. In our words: A name and a version, with a slash between them. In our reading it is the name of browser software, not of OpenAI’s agent. OpenAI’s page does not explain it. What the source says: RFC 9110: the value “consists of one or more product identifiers, each followed by zero or more comments”, and “Each product identifier consists of a name and optional version.” Sources: [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110) and [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots).
- AppleWebKit/537.36. In our words: Another name and version. In our reading it names a browser engine, again not OpenAI’s own software. RFC 9110 advises against using other implementations’ product tokens to declare compatibility. OpenAI’s page does not explain this part of its string. What the source says: RFC 9110: “implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them”. Source: [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110).
- (KHTML, like Gecko). In our words: A comment, marked by the brackets. In our reading it names two more browser engines. What the source says: RFC 9110: “Comments can be included in some HTTP fields by surrounding the comment text with parentheses.” Source: [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110).
- ChatGPT-User/1.0. In our words: The part that names OpenAI’s agent and its version. A robots.txt group for this agent is written against this name. What the source says: OpenAI: “When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” RFC 9309: a crawler’s product token “SHOULD be a substring in the User-Agent header”. Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) and [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309).
- +https\://openai.com/bot. In our words: A web address carried in the string. Like the rest of the line, it is text the sender wrote, so it tells you who the sender says it is. What the source says: OpenAI’s page prints it as part of the full string and does not explain it. Source: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots).

Any program can send this exact line. The Web Bot Auth draft says: “User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent.” Under the string, OpenAI also publishes the IP addresses ChatGPT-User uses, at openai.com/chatgpt-user.json. Sources: [IETF draft-ietf-webbotauth-httpsig-protocol-00](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/) and [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots).

The header holds whatever the client chooses to write. RFC 9110 says implementations "are encouraged not to use the product tokens of other implementations in order to declare compatibility with them". The Web Bot Auth draft, covered below, puts it in security terms: "User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent." A scraper can send `ChatGPT-User/1.0` as easily as OpenAI can.

OpenAI and Perplexity publish a second signal next to each string: a list of the IP addresses (the numeric addresses of the machines a request comes from) that the bot uses. OpenAI lists ChatGPT-User's at `openai.com/chatgpt-user.json`, and Perplexity publishes one for each of its two bots. A request that claims to be ChatGPT-User but comes from an address outside that list is, in our reading, a reason for doubt. The draft still counts "IP allowlisting, User-Agent strings, or shared API keys" among methods that "have significant limitations in security, scalability, and manageability."

## robots.txt and what it can ask

The oldest way for a site to address crawlers is a text file at `/robots.txt`. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), a Standards Track RFC of September 2022 (in our words, a document on the IETF's path to becoming a standard), describes its contents as rules "that crawlers are requested to honor when accessing URIs. These rules are not a form of access authorization." It also says the file is "not a substitute for valid content security measures". In our reading, a crawler that ignores the file can still fetch the pages.

The rules are grouped under `User-agent` lines, and a crawler finds its group by name. "Crawlers set their own name, which is called a product token, to find relevant groups", and that token "SHOULD be a substring in the User-Agent header". If no group names the crawler, it "MUST obey the group with a user-agent line with the "\*" value, if present". Inside a group, the most specific matching rule wins, which RFC 9309 defines as the longest match, counted in bytes. When an `allow` and a `disallow` rule are equivalent, the `allow` rule "SHOULD be used".

If the server answers the request for robots.txt with a 4xx status (a client error, such as 404 Not Found), the crawler "MAY access any resources on the server"; a 5xx status (a server error) or a network error means it "MUST assume complete disallow". OpenAI says it can take "\~24 hours" for its search systems to pick up a change.

Paste your own file into the tester to see which named agents it asks to stay out of the home page and of a product page. The tester checks the file in your browser and sends nothing you paste.

Which named agents a robots.txt asks to stay out.

On the page you can paste a robots.txt and see, for each agent token our scan checks, whether it may fetch / and a product page (/products/example). The figure reads it with the same parser our scan uses; this figure checks it in your browser and sends nothing you paste. The three examples:

Allow all:

```text
User-agent: *
Disallow:
```

| Agent token          | Vendor       | /       | Product page | Decided by   |
| -------------------- | ------------ | ------- | ------------ | ------------ |
| GPTBot               | OpenAI       | Allowed | Allowed      | the \* group |
| OAI-SearchBot        | OpenAI       | Allowed | Allowed      | the \* group |
| ChatGPT-User         | OpenAI       | Allowed | Allowed      | the \* group |
| chatgpt.com          | OpenAI       | Allowed | Allowed      | the \* group |
| ClaudeBot            | Anthropic    | Allowed | Allowed      | the \* group |
| Claude-User          | Anthropic    | Allowed | Allowed      | the \* group |
| Claude-SearchBot     | Anthropic    | Allowed | Allowed      | the \* group |
| PerplexityBot        | Perplexity   | Allowed | Allowed      | the \* group |
| Perplexity-User      | Perplexity   | Allowed | Allowed      | the \* group |
| Googlebot            | Google       | Allowed | Allowed      | the \* group |
| Google-Extended      | Google       | Allowed | Allowed      | the \* group |
| Google-Agent         | Google       | Allowed | Allowed      | the \* group |
| Amazonbot            | Amazon       | Allowed | Allowed      | the \* group |
| Amzn-SearchBot       | Amazon       | Allowed | Allowed      | the \* group |
| Amzn-User            | Amazon       | Allowed | Allowed      | the \* group |
| meta-externalagent   | Meta         | Allowed | Allowed      | the \* group |
| meta-externalfetcher | Meta         | Allowed | Allowed      | the \* group |
| Applebot             | Apple        | Allowed | Allowed      | the \* group |
| Applebot-Extended    | Apple        | Allowed | Allowed      | the \* group |
| bingbot              | Microsoft    | Allowed | Allowed      | the \* group |
| Bytespider           | ByteDance    | Allowed | Allowed      | the \* group |
| CCBot                | Common Crawl | Allowed | Allowed      | the \* group |

Block GPTBot only:

```text
User-agent: GPTBot
Disallow: /

User-agent: *
Disallow:
```

| Agent token          | Vendor       | /       | Product page | Decided by    |
| -------------------- | ------------ | ------- | ------------ | ------------- |
| GPTBot               | OpenAI       | Blocked | Blocked      | its own group |
| OAI-SearchBot        | OpenAI       | Allowed | Allowed      | the \* group  |
| ChatGPT-User         | OpenAI       | Allowed | Allowed      | the \* group  |
| chatgpt.com          | OpenAI       | Allowed | Allowed      | the \* group  |
| ClaudeBot            | Anthropic    | Allowed | Allowed      | the \* group  |
| Claude-User          | Anthropic    | Allowed | Allowed      | the \* group  |
| Claude-SearchBot     | Anthropic    | Allowed | Allowed      | the \* group  |
| PerplexityBot        | Perplexity   | Allowed | Allowed      | the \* group  |
| Perplexity-User      | Perplexity   | Allowed | Allowed      | the \* group  |
| Googlebot            | Google       | Allowed | Allowed      | the \* group  |
| Google-Extended      | Google       | Allowed | Allowed      | the \* group  |
| Google-Agent         | Google       | Allowed | Allowed      | the \* group  |
| Amazonbot            | Amazon       | Allowed | Allowed      | the \* group  |
| Amzn-SearchBot       | Amazon       | Allowed | Allowed      | the \* group  |
| Amzn-User            | Amazon       | Allowed | Allowed      | the \* group  |
| meta-externalagent   | Meta         | Allowed | Allowed      | the \* group  |
| meta-externalfetcher | Meta         | Allowed | Allowed      | the \* group  |
| Applebot             | Apple        | Allowed | Allowed      | the \* group  |
| Applebot-Extended    | Apple        | Allowed | Allowed      | the \* group  |
| bingbot              | Microsoft    | Allowed | Allowed      | the \* group  |
| Bytespider           | ByteDance    | Allowed | Allowed      | the \* group  |
| CCBot                | Common Crawl | Allowed | Allowed      | the \* group  |

Block all:

```text
User-agent: *
Disallow: /
```

| Agent token          | Vendor       | /       | Product page | Decided by   |
| -------------------- | ------------ | ------- | ------------ | ------------ |
| GPTBot               | OpenAI       | Blocked | Blocked      | the \* group |
| OAI-SearchBot        | OpenAI       | Blocked | Blocked      | the \* group |
| ChatGPT-User         | OpenAI       | Blocked | Blocked      | the \* group |
| chatgpt.com          | OpenAI       | Blocked | Blocked      | the \* group |
| ClaudeBot            | Anthropic    | Blocked | Blocked      | the \* group |
| Claude-User          | Anthropic    | Blocked | Blocked      | the \* group |
| Claude-SearchBot     | Anthropic    | Blocked | Blocked      | the \* group |
| PerplexityBot        | Perplexity   | Blocked | Blocked      | the \* group |
| Perplexity-User      | Perplexity   | Blocked | Blocked      | the \* group |
| Googlebot            | Google       | Blocked | Blocked      | the \* group |
| Google-Extended      | Google       | Blocked | Blocked      | the \* group |
| Google-Agent         | Google       | Blocked | Blocked      | the \* group |
| Amazonbot            | Amazon       | Blocked | Blocked      | the \* group |
| Amzn-SearchBot       | Amazon       | Blocked | Blocked      | the \* group |
| Amzn-User            | Amazon       | Blocked | Blocked      | the \* group |
| meta-externalagent   | Meta         | Blocked | Blocked      | the \* group |
| meta-externalfetcher | Meta         | Blocked | Blocked      | the \* group |
| Applebot             | Apple        | Blocked | Blocked      | the \* group |
| Applebot-Extended    | Apple        | Blocked | Blocked      | the \* group |
| bingbot              | Microsoft    | Blocked | Blocked      | the \* group |
| Bytespider           | ByteDance    | Blocked | Blocked      | the \* group |
| CCBot                | Common Crawl | Blocked | Blocked      | the \* group |

robots.txt is a request. RFC 9309 calls its rules ones “that crawlers are requested to honor” and says they “are not a form of access authorization”. Source: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

Notes on particular tokens:

- ChatGPT-User: OpenAI: “robots.txt rules may not apply”, because a user starts these actions. Source: [OpenAI crawlers](https://developers.openai.com/api/docs/bots).
- Perplexity-User: Perplexity: this fetcher “generally ignores robots.txt rules”, because a user requested the fetch. Source: [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers).
- ClaudeBot, Claude-User, Claude-SearchBot: Anthropic says its bots honour robots.txt directives. Source: [Anthropic crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler).
- chatgpt.com: Not an RFC 9309 product token (letters, \_ and - only); shown because our scan checks it by this name. Source: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

Whether a bot follows the file is up to its operator, and operators say different things about fetches a user started. OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules", because "a user requested the fetch". Google says the same of its user-triggered fetchers. Anthropic says its bots "respect "do not crawl" signals by honoring industry standard directives in robots.txt", and the bots its page lists include Claude-User. OpenAI also says each of its settings "is independent of the others", so blocking GPTBot does not block OAI-SearchBot.

## Bot management

In our words, bot management is a service that sits in front of a site, sorts incoming requests into people, known bots and the rest, and blocks, slows or challenges them as its rules say.

Such a service needs a way to tell a real ChatGPT-User request from a copy. Cloudflare's [verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) page, updated 8 October 2026, names three ways a bot can declare "who it is deterministically": "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS". Reverse DNS, in our words, means looking up the name registered for the address a request came from. A verified bot must also behave: it "obeys robots.txt and crawl directives, maintains reasonable request rates". The page adds that "Signed agents are now Verified As of July 1, 2026".

Anthropic's page warns that "Alternate methods like blocking IP address(es) … may not work correctly or persistently guarantee an opt-out". And verification does not oblige anyone to let a bot in. Amazon's documentation for its [AgentCore Browser](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html), which can sign its requests, says: "Domain owners retain full control over their bot policies and may block, monitor, or rate-limit agent traffic regardless of cryptographic signatures".

## What a signed request carries

The first of Cloudflare's three ways is a signature. [Web Bot Auth](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/) is an Internet-Draft, `draft-ietf-webbotauth-httpsig-protocol-00`, dated 1 September 2026, a working-group document of the IETF's Web Bot Auth group with an intended status of Standards Track. In our words, an Internet-Draft is a working text that can still change; the IETF's own boilerplate says it may be cited only as "work in progress". Its aim: "to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence."

It builds on [RFC 9421](https://www.rfc-editor.org/rfc/rfc9421), HTTP Message Signatures (February 2024). The idea rests on a key pair. In our words, the agent's operator keeps a private key secret and publishes the matching public key; a signature made with the private key can be checked by anyone holding the public one, and cannot be made without the private one. The agent signs parts of each request, and the server checks the signature against the published key.

A signed request "carries three headers": Signature, Signature-Input and Signature-Agent. The draft requires the signature to cover `@authority` or `@target-uri`, and to include the parameters `created`, `expires`, `keyid` and `tag`, with `tag` set to `web-bot-auth`. The figure is the draft's own example.

A signed request, header by header.

The example request from the Web Bot Auth draft, revision -00, section 5.2.5. Signature-Input is one header, split over several lines here only for width.

```http
GET /path/to/resource HTTP/1.1
Host: origin.example.com
Signature: sig=abc==
Signature-Input: sig=(
  "@authority" "signature-agent";key="sig");
  created=1700000000;
  expires=1700011111;
  keyid="ba3e64==";
  tag="web-bot-auth"
Signature-Agent: sig="https://signer.example.com"
```

Its headers and parameters:

- Signature. In our words: The signature itself, filed under the label sig. The draft’s example puts a short stand-in value here. What the source says: RFC 9421: each member’s value “is a Byte Sequence containing the signature value for the message signature identified by the label”. Sources: [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) and [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
- Signature-Input. In our words: What was signed and with which settings, under the same label sig. The verifier rebuilds the signed text from this line. What the source says: RFC 9421: the field holds “the metadata for one or more message signatures”; “The member’s key is the label that uniquely identifies the message signature within the HTTP message. The member’s value is the covered components ordered set”. Source: [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421).
- "@authority". In our words: The site the request is addressed to, origin.example.com, which HTTP/1.1 usually carries in the Host line. Signing it ties the signature to this site. What the source says: Draft: “Agents MUST include at least one of the following components: @authority … @target-uri”. RFC 9421: @authority “refers to the authority component of the target URI of the HTTP request message”. Sources: [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt) and [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421).
- "signature-agent";key="sig". In our words: Signs the sig entry of the Signature-Agent header, so the pointer to the keys is covered by the signature too. What the source says: Draft: the Signature-Agent header’s “member keyed to the signature label MUST be signed as a component”. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
- created. In our words: When the signature was made, as a count of seconds since the start of 1970 (UTC). What the source says: RFC 9421: “Creation time as a UNIX timestamp value of type Integer.” The draft makes it required. Sources: [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) and [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
- expires. In our words: When the signature stops being valid. In the example it is 11,111 seconds after created, a little over three hours. What the source says: RFC 9421: “Expiration time as a UNIX timestamp value of type Integer.” The draft makes it required and says: “It is RECOMMENDED that expiry be no more than 24 hours.” Sources: [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) and [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
- keyid. In our words: Which public key to check the signature with. A thumbprint, in our words, is a short fingerprint computed from the key. What the source says: Draft: keyid “MUST be a base64url JWK SHA-256 Thumbprint”, and “The key used to verify the signature is selected by the keyid parameter of the corresponding Signature-Input member.” Source: [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
- tag. In our words: Marks this as a Web Bot Auth signature, so a server can pick out the signatures meant for this protocol. What the source says: RFC 9421: “An application-specific tag for the signature”. Draft: tag “MUST be web-bot-auth”, and an origin “MAY discard signatures for which the tag is not set to web-bot-auth”. Sources: [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) and [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
- Signature-Agent. In our words: Where the keys can be found: an https address, filed under the label sig like the other two headers. What the source says: Draft: “Signature-Agent is a Dictionary Structured Header … Its member values MUST be String Items that contain a \[URI], whose scheme MUST be https”, and “Signers MUST send the dictionary form.” Cloudflare’s docs (updated 8 October 2026) still say: “Ensure your Signature-Agent header uses a structured string, not a dictionary.” Sources: [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt) and [Cloudflare docs: Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/).

This example signs only the site and the key pointer. The draft warns that “A signature covering @authority alone verifies against any method, path, or body sent to that authority until it expires”, and says agents that want to narrow it “SHOULD also cover” @method, @path, @target-uri or @query-param. “No component covers the body”: an agent that needs one must send and cover a Content-Digest header. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).

The Signature-Agent header is where the draft and one verifier currently disagree. The draft makes it a Dictionary (in our words, a header made of named entries) and says "Signers MUST send the dictionary form", while allowing that "Earlier versions of this protocol defined Signature-Agent as a bare String, and deployments still send it". Cloudflare's [Web Bot Auth documentation](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/), updated 8 October 2026, cites older individual drafts and tells bot operators: "Ensure your Signature-Agent header uses a structured string, not a dictionary." It also says it does not support the `key` component parameter, which the draft's example uses. In our reading, an operator writing a signer today has to pick which one to follow.

## How a store checks a signature

The draft's story has four parties: a user, the agent, the origin (the store's server) and a directory that holds the agent's keys. "A User initiates an action requiring the Agent to perform an HTTP request. The Agent constructs the request, generates a signature using its signing key, and includes it in the request … along with the Signature-Agent header for discovery of its verification key. Upon receiving the request, the Origin ensures it has the verification key for the Agent, validates the signature, and processes the request if the signature is valid."

The sequence plays it through, with one case for an origin that has not fetched the keys yet and one for an unsigned request.

A request signed with Web Bot Auth, and how the store checks it.

1. User → Agent: Start an action. In the draft’s story a user starts an action that needs the agent to make an HTTP request. The draft is revision -00, a working-group document. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
2. Agent → Origin: Signed GET /path/to/resource. Per the draft, a signed request carries Signature, Signature-Input (covering @authority and signature-agent, tag="web-bot-auth") and Signature-Agent, which the draft makes a Dictionary: Signature-Agent: sig="https\://signer.example.com". Cloudflare’s docs (Oct 8, 2026) still ask for a structured string, not a dictionary. Sources: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt) and [Cloudflare docs: Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/).
3. Origin → Agent: Response. With keys it already holds (the draft has later requests verify from cache), the origin checks the signature with RFC 9421’s algorithm: it rebuilds the signature base from Signature-Input and verifies it. A key it does not know or trust fails verification. Sources: [RFC 9421, HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) and [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).

If keys not resolved:

1. Agent → Origin: Signed GET, keys not yet known. On a first request, before the keys are cached: the same signed request, to an origin that has not resolved this agent’s keys. Per the draft, the origin learns the key URL from the request, so it resolves keys after the first request rather than before it. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
2. Origin → Directory: GET /.well-known/http-message-signatures-directory. Per the draft, an origin that does not know the keyid may fetch keys where Signature-Agent points. The fetch must get a 200 and must not follow redirects. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
3. Directory → Origin: Keys (JWK set). The agent publishes its keys here. The draft’s example returns Ed25519 keys as application/http-message-signatures-directory+json with Cache-Control: max-age=86400. Lookups are keyed on the (URL, key) pair. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
4. Origin → Agent: Response, once verified. Per the draft’s Figure 2, the origin responds after resolving the keys; it verifies with RFC 9421 and processes the request if the signature is valid. Later requests verify from cache. Sources: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt) and [RFC 9421, HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421).

If the request is unsigned:

1. Agent → Origin: Unsigned GET /path/to/resource. When the request carries no signature. Per the draft, an origin that sees none has learned nothing about the sender, and the draft neither requires it to treat unsigned requests differently nor gives it grounds to. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).
2. Origin → Agent: 403 + Accept-Signature. The draft lets an origin ask a client that did not sign for a signature with RFC 9421’s Accept-Signature, and says the status should be 403 Forbidden. Source: [draft-ietf-webbotauth-httpsig-protocol-00 (WG Document, work in progress)](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt).

The draft and RFC 9421 add these details:

- The verification follows RFC 9421's algorithm, which the draft says "happens at the HTTP layer, once headers are received". A key "that the verifier does not know or trust" makes verification fail.
- By default, the keys live at `/.well-known/http-message-signatures-directory` on the host that Signature-Agent names (the draft also defines other types of entry), as JSON (a plain-text format for data), with the media type (the label that says what format a response is in) `application/http-message-signatures-directory+json`. Fetches "MUST be served with a 200 (OK)" and "MUST NOT automatically follow HTTP redirects".
- "The Origin learns the URL from the request, so it resolves keys after the first request rather than before it. Later requests verify from cache".
- An origin may ask a client that did not sign to do so, with RFC 9421's `Accept-Signature`, and "The status code SHOULD be 403 Forbidden".
- A missing signature tells the store nothing either way: "A verifier that sees none has learned nothing about the sender: not that it is automated, not that it is human, not that it is evading anything." The draft "neither requires an origin to treat unsigned requests differently nor gives it grounds to."

In our reading, a valid signature shows that whoever sent the request holds the private key matching a key in that directory. It does not say what the agent was asked to do or for whom, and the draft says "Delegation and chaining are out of scope for this document".

## Who signs requests today

We looked for each vendor's own statement on 22 October 2026.

- Google has a page titled "Web Bot Auth (experimental)", last updated 13 October 2026. "A subset of requests made by the Google-Agent are signed with Web Bot Auth; in these cases, they are authenticated as <https://agent.bot.goog>." It also says "We don't sign every request of a particular agent" and tells verifiers: "Don't forget to fall back to IP-based verification, as not all requests are signed."
- Amazon says that when Web Bot Auth is enabled, AgentCore Browser, a preview feature, "automatically signs each HTTP request with a private key and includes verification headers".
- OpenAI's crawler page does not mention signatures. A help article on its help site, whose address names ChatGPT agent allowlisting, returned HTTP 403 (access refused) when we tried to fetch it, so we cannot say what it states.
- Anthropic's and Perplexity's crawler pages do not mention signatures.

On 22 October 2026, `https://chatgpt.com/.well-known/http-message-signatures-directory` returned a signed key directory holding one Ed25519 key (Ed25519 is a signature algorithm), and Google's at `agent.bot.goog` returned a signed directory with several. That shows the directories exist, though not which requests are signed with them.

## What you can check today

- Open `https://your-domain/robots.txt` and paste it into the tester above. Check that the agents you want on your product pages are not asked to stay out.
- In your server's logs, or those of the service in front of your site, look for agent names in the User-Agent field, such as `ChatGPT-User` and `Perplexity-User`, which appear in the full strings their vendors publish. For a request that claims one of them, compare the address it came from with that vendor's published IP list.
- Ask whoever runs your bot protection three things: whether it verifies Web Bot Auth signatures, how it treats verified agents and user-directed agents, and whether its rules challenge them at checkout.
- To see a key directory for yourself, run `curl -i https://agent.bot.goog/.well-known/http-message-signatures-directory` and look at the `content-type` and the `keys` in the answer.

## What is not settled

- Web Bot Auth is at revision -00, an Internet-Draft that may be cited only as "work in progress".
- The draft requires the Dictionary form of Signature-Agent, and Cloudflare's documentation still asks for a string.
- Among the vendors we checked, signing is documented for Google-Agent (some of its requests) and AgentCore Browser (when enabled). OpenAI's crawler page does not mention it, and a help article whose address names ChatGPT agent allowlisting could not be fetched on 22 October 2026. Anthropic's and Perplexity's pages say nothing about it.
- How robots.txt applies to fetchers a user started is each operator's own policy, and the policies differ.
- The draft leaves delegation and chaining out of scope, and it gives a store no grounds to treat an unsigned request differently.

Proving identity is the second wall in [the hub](https://storewalk.ai/blog/why-agents-cant-buy). A verified agent can still be stopped later, by a challenge page or a sign-in it cannot complete; [Why agents stall at checkout](https://storewalk.ai/blog/why-agents-stall-at-checkout) covers those.

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google: User-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [Cloudflare docs: Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/)
- [AWS docs: Reducing CAPTCHAs with Web Bot Auth (AgentCore Browser)](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html)
- [IETF datatracker: draft-ietf-webbotauth-httpsig-protocol](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)
- [draft-ietf-webbotauth-httpsig-protocol-00: HTTP Message Signatures for automated traffic](https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-00.txt)
- [IETF Web Bot Auth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/)
- [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421)
- [Cloudflare docs: Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/)
- [Google: Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)
- [Google-Agent key directory (fetched 2026-10-22)](https://agent.bot.goog/.well-known/http-message-signatures-directory)
- [chatgpt.com key directory (fetched 2026-10-22)](https://chatgpt.com/.well-known/http-message-signatures-directory)

Last checked 22 Oct 2026