Bots, crawlers and agents: who is knocking

Three of the AI companies we checked, OpenAI, Anthropic and Perplexity, sort their bots by purpose. Their pages, as they read on 22 October 2026, describe them like this:

• Training crawlers. OpenAI's GPTBot "is used to crawl content that may be used in training our generative AI foundation models". Anthropic's ClaudeBot collects "web content that could potentially contribute to their training".
• Search crawlers. OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features". Claude-SearchBot "navigates the web to improve search result quality for users". PerplexityBot "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."
• Fetchers that act on a user's request. OpenAI: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent", and "ChatGPT-User is not used for crawling the web in an automatic fashion." Anthropic: "When individuals ask questions to Claude, it may access websites using a Claude-User agent". Perplexity, of Perplexity-User: "When users ask Perplexity a question, it might visit a web page to help provide an accurate answer".

Google documents one more. Its page on user-triggered fetchers, last updated 19 August 2026, says: "Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Cloudflare has a similar split in its verified bots documentation, with an "Agent" category for "User-directed agents visiting a page on behalf of a human" and a "Transact" category for "Checkout or other transaction actions on behalf of users".

In our reading, a shopping agent is the last kind: a program that visits because one person asked it to, at the moment they asked. A crawler visits on its own schedule. A store may want to treat the two differently, if it can tell them apart.

HTTP is the protocol browsers and servers use to exchange pages. An HTTP request is the message a client sends to ask for one, and its headers are the named lines at the top of that message. One of them is User-Agent. The IETF (the Internet Engineering Task Force) develops internet standards and publishes them as numbered documents called RFCs. RFC 9110, its standard for HTTP semantics (June 2022), says the header "contains information about the user agent originating the request".

The value has a grammar: "one or more product identifiers, each followed by zero or more comments", where "Each product identifier consists of a name and optional version." The figure takes the full string OpenAI publishes for ChatGPT-User.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

The header holds whatever the client chooses to write. RFC 9110 says implementations "are encouraged not to use the product tokens of other implementations in order to declare compatibility with them". The Web Bot Auth draft, covered below, puts it in security terms: "User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent." A scraper can send ChatGPT-User/1.0 as easily as OpenAI can.

OpenAI and Perplexity publish a second signal next to each string: a list of the IP addresses (the numeric addresses of the machines a request comes from) that the bot uses. OpenAI lists ChatGPT-User's at openai.com/chatgpt-user.json, and Perplexity publishes one for each of its two bots. A request that claims to be ChatGPT-User but comes from an address outside that list is, in our reading, a reason for doubt. The draft still counts "IP allowlisting, User-Agent strings, or shared API keys" among methods that "have significant limitations in security, scalability, and manageability."

The oldest way for a site to address crawlers is a text file at /robots.txt. RFC 9309, a Standards Track RFC of September 2022 (in our words, a document on the IETF's path to becoming a standard), describes its contents as rules "that crawlers are requested to honor when accessing URIs. These rules are not a form of access authorization." It also says the file is "not a substitute for valid content security measures". In our reading, a crawler that ignores the file can still fetch the pages.

The rules are grouped under User-agent lines, and a crawler finds its group by name. "Crawlers set their own name, which is called a product token, to find relevant groups", and that token "SHOULD be a substring in the User-Agent header". If no group names the crawler, it "MUST obey the group with a user-agent line with the "*" value, if present". Inside a group, the most specific matching rule wins, which RFC 9309 defines as the longest match, counted in bytes. When an allow and a disallow rule are equivalent, the allow rule "SHOULD be used".

If the server answers the request for robots.txt with a 4xx status (a client error, such as 404 Not Found), the crawler "MAY access any resources on the server"; a 5xx status (a server error) or a network error means it "MUST assume complete disallow". OpenAI says it can take "~24 hours" for its search systems to pick up a change.

Paste your own file into the tester to see which named agents it asks to stay out of the home page and of a product page. The tester checks the file in your browser and sends nothing you paste.

Whether a bot follows the file is up to its operator, and operators say different things about fetches a user started. OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules", because "a user requested the fetch". Google says the same of its user-triggered fetchers. Anthropic says its bots "respect "do not crawl" signals by honoring industry standard directives in robots.txt", and the bots its page lists include Claude-User. OpenAI also says each of its settings "is independent of the others", so blocking GPTBot does not block OAI-SearchBot.

In our words, bot management is a service that sits in front of a site, sorts incoming requests into people, known bots and the rest, and blocks, slows or challenges them as its rules say.

Such a service needs a way to tell a real ChatGPT-User request from a copy. Cloudflare's verified bots page, updated 8 October 2026, names three ways a bot can declare "who it is deterministically": "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS". Reverse DNS, in our words, means looking up the name registered for the address a request came from. A verified bot must also behave: it "obeys robots.txt and crawl directives, maintains reasonable request rates". The page adds that "Signed agents are now Verified As of July 1, 2026".

Anthropic's page warns that "Alternate methods like blocking IP address(es) … may not work correctly or persistently guarantee an opt-out". And verification does not oblige anyone to let a bot in. Amazon's documentation for its AgentCore Browser, which can sign its requests, says: "Domain owners retain full control over their bot policies and may block, monitor, or rate-limit agent traffic regardless of cryptographic signatures".

The first of Cloudflare's three ways is a signature. Web Bot Auth is an Internet-Draft, draft-ietf-webbotauth-httpsig-protocol-00, dated 1 September 2026, a working-group document of the IETF's Web Bot Auth group with an intended status of Standards Track. In our words, an Internet-Draft is a working text that can still change; the IETF's own boilerplate says it may be cited only as "work in progress". Its aim: "to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence."

It builds on RFC 9421, HTTP Message Signatures (February 2024). The idea rests on a key pair. In our words, the agent's operator keeps a private key secret and publishes the matching public key; a signature made with the private key can be checked by anyone holding the public one, and cannot be made without the private one. The agent signs parts of each request, and the server checks the signature against the published key.

A signed request "carries three headers": Signature, Signature-Input and Signature-Agent. The draft requires the signature to cover @authority or @target-uri, and to include the parameters created, expires, keyid and tag, with tag set to web-bot-auth. The figure is the draft's own example.

GET /path/to/resource HTTP/1.1Host: origin.example.comSignature: sig=abc==Signature-Input: sig=(  "@authority" "signature-agent";key="sig");  created=1700000000;  expires=1700011111;  keyid="ba3e64==";  tag="web-bot-auth"Signature-Agent: sig="https://signer.example.com"

The Signature-Agent header is where the draft and one verifier currently disagree. The draft makes it a Dictionary (in our words, a header made of named entries) and says "Signers MUST send the dictionary form", while allowing that "Earlier versions of this protocol defined Signature-Agent as a bare String, and deployments still send it". Cloudflare's Web Bot Auth documentation, updated 8 October 2026, cites older individual drafts and tells bot operators: "Ensure your Signature-Agent header uses a structured string, not a dictionary." It also says it does not support the key component parameter, which the draft's example uses. In our reading, an operator writing a signer today has to pick which one to follow.

The draft's story has four parties: a user, the agent, the origin (the store's server) and a directory that holds the agent's keys. "A User initiates an action requiring the Agent to perform an HTTP request. The Agent constructs the request, generates a signature using its signing key, and includes it in the request … along with the Signature-Agent header for discovery of its verification key. Upon receiving the request, the Origin ensures it has the verification key for the Agent, validates the signature, and processes the request if the signature is valid."

The sequence plays it through, with one case for an origin that has not fetched the keys yet and one for an unsigned request.

The draft and RFC 9421 add these details:

• The verification follows RFC 9421's algorithm, which the draft says "happens at the HTTP layer, once headers are received". A key "that the verifier does not know or trust" makes verification fail.
• By default, the keys live at /.well-known/http-message-signatures-directory on the host that Signature-Agent names (the draft also defines other types of entry), as JSON (a plain-text format for data), with the media type (the label that says what format a response is in) application/http-message-signatures-directory+json. Fetches "MUST be served with a 200 (OK)" and "MUST NOT automatically follow HTTP redirects".
• "The Origin learns the URL from the request, so it resolves keys after the first request rather than before it. Later requests verify from cache".
• An origin may ask a client that did not sign to do so, with RFC 9421's Accept-Signature, and "The status code SHOULD be 403 Forbidden".
• A missing signature tells the store nothing either way: "A verifier that sees none has learned nothing about the sender: not that it is automated, not that it is human, not that it is evading anything." The draft "neither requires an origin to treat unsigned requests differently nor gives it grounds to."

In our reading, a valid signature shows that whoever sent the request holds the private key matching a key in that directory. It does not say what the agent was asked to do or for whom, and the draft says "Delegation and chaining are out of scope for this document".

We looked for each vendor's own statement on 22 October 2026.

• Google has a page titled "Web Bot Auth (experimental)", last updated 13 October 2026. "A subset of requests made by the Google-Agent are signed with Web Bot Auth; in these cases, they are authenticated as https://agent.bot.goog." It also says "We don't sign every request of a particular agent" and tells verifiers: "Don't forget to fall back to IP-based verification, as not all requests are signed."
• Amazon says that when Web Bot Auth is enabled, AgentCore Browser, a preview feature, "automatically signs each HTTP request with a private key and includes verification headers".
• OpenAI's crawler page does not mention signatures. A help article on its help site, whose address names ChatGPT agent allowlisting, returned HTTP 403 (access refused) when we tried to fetch it, so we cannot say what it states.
• Anthropic's and Perplexity's crawler pages do not mention signatures.

On 22 October 2026, https://chatgpt.com/.well-known/http-message-signatures-directory returned a signed key directory holding one Ed25519 key (Ed25519 is a signature algorithm), and Google's at agent.bot.goog returned a signed directory with several. That shows the directories exist, though not which requests are signed with them.

• Open https://your-domain/robots.txt and paste it into the tester above. Check that the agents you want on your product pages are not asked to stay out.
• In your server's logs, or those of the service in front of your site, look for agent names in the User-Agent field, such as ChatGPT-User and Perplexity-User, which appear in the full strings their vendors publish. For a request that claims one of them, compare the address it came from with that vendor's published IP list.
• Ask whoever runs your bot protection three things: whether it verifies Web Bot Auth signatures, how it treats verified agents and user-directed agents, and whether its rules challenge them at checkout.
• To see a key directory for yourself, run curl -i https://agent.bot.goog/.well-known/http-message-signatures-directory and look at the content-type and the keys in the answer.

• Web Bot Auth is at revision -00, an Internet-Draft that may be cited only as "work in progress".
• The draft requires the Dictionary form of Signature-Agent, and Cloudflare's documentation still asks for a string.
• Among the vendors we checked, signing is documented for Google-Agent (some of its requests) and AgentCore Browser (when enabled). OpenAI's crawler page does not mention it, and a help article whose address names ChatGPT agent allowlisting could not be fetched on 22 October 2026. Anthropic's and Perplexity's pages say nothing about it.
• How robots.txt applies to fetchers a user started is each operator's own policy, and the policies differ.
• The draft leaves delegation and chaining out of scope, and it gives a store no grounds to treat an unsigned request differently.

Proving identity is the second wall in the hub. A verified agent can still be stopped later, by a challenge page or a sign-in it cannot complete; Why agents stall at checkout covers those.