GPTBot vs OAI-SearchBot vs ChatGPT-User: which OpenAI bot does what

OpenAI's crawler page lists four user agents: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. It describes them as "web crawlers ("robots") and user agents to perform actions for its products, either automatically or triggered by user request." A fifth visitor has its own help page: ChatGPT Work's Cloud browser, which browses for a user and signs its requests instead of naming itself.

Four of them have a token you can write in robots.txt. They do not share one switch, which is the most useful fact on the page. "Each setting is independent of the others", OpenAI says, and its example is the one most stores want: allow OAI-SearchBot "in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models."

Pick a visitor to see what it does, what robots.txt does to it, and how to check it is genuine:

It collects training material. OpenAI says "GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models." A block is a signal about training and nothing else: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."

Of the four, GPTBot matters least to sales: it does not put products in front of shoppers. If you want OpenAI's future models to know your brand, let it in; if you would rather not feed a model your product copy, block it. Either way, your ChatGPT search presence is decided elsewhere.

One detail from OpenAI cuts the other way: "If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling." Allowing both can mean one visit, which your server may appreciate.

It is the one that puts you in ChatGPT search. OpenAI is plain about it: "OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." And about the opt-out: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

OpenAI asks for it outright: "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges below." Note the IP ranges. A firewall rule or a bot manager that turns OpenAI's IP ranges away has the same effect as a Disallow line, and robots.txt will not tell you it happened. The bot protection guide covers that side.

Changes are not instant. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Make the edit a day before the sale starts, not on the morning of it.

It fetches a page for a person, live. "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." OpenAI adds that "ChatGPT-User is not used for crawling the web in an automatic fashion."

robots.txt is a weaker lever here. "Because these actions are initiated by a user, robots.txt rules may not apply." OpenAI also separates it from search: "ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl."

For a store, this is the visit closest to a sale. A shopper has asked about your product and ChatGPT is reading your page to answer. Blocking it saves you very little bandwidth and costs you that answer.

OAI-AdsBot only matters if you advertise on ChatGPT. "OAI-AdsBot is used to validate the safety of web pages submitted as ads on ChatGPT." It "only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models." OpenAI's page makes no robots.txt statement for it.

The Cloud browser is a different animal. Its help page says "ChatGPT Work's Cloud browser uses Web Bot Auth to sign outbound HTTP requests, allowing website operators to verify that requests genuinely originate from ChatGPT." Each request carries "A Signature-Agent header set to "https://chatgpt.com"", and the keys are published at chatgpt.com/.well-known/http-message-signatures-directory. The page gives no robots.txt token, and it sets a limit worth knowing: "at launch, Cloud browser cannot sign in to websites or complete payments." So, by its help page, it launched able to browse your store for a user but not to pay. How signed requests work is in bots, crawlers and agents.

Here is the short version, all from OpenAI's crawler page:

Choose what you want from each bot and the builder writes the file. It also shows the verdict our own robots.txt check would give, computed with the scanner's rule:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /cart
Disallow: /checkout

Our DISC-001 sorts agents by kind. A full block of GPTBot alone gets a warning, because search and shopping agents can still read the store. A full block of OAI-SearchBot or ChatGPT-User fails, because those are the visits that put you in front of shoppers. The cart and checkout lines in the builder are a partial block, which is normal for a store and costs little.

No. By OpenAI's account it keeps your pages out of training and nothing more, because the settings are independent. The rule that hides you is the OAI-SearchBot one. A file that blocks GPTBot and stops there leaves the store eligible for ChatGPT search, by OpenAI's description.

The mistake that does hide a store usually looks like this:

User-agent: *
Disallow: /

A blanket block under * catches every bot that has no group of its own, OAI-SearchBot included. It is easy to carry over from a staging site, where it belongs, to the live store, where it does not.

To check what your own file does, paste it in. Each OpenAI token gets its verdict for a path and the rule that decided it:

Not by its User-Agent string alone, which anyone can copy. OpenAI publishes an IP list for each bot: openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json. All four returned 200 when we fetched them. A request that says GPTBot but comes from an address outside its list is someone borrowing the name.

OpenAI also marks some robots.txt fetches: "When fetching robots.txt files, we may use a user-agent string with an additional robots.txt marker." That helps when your logs show no paths. For the Cloud browser, the check is its signature against the published keys, not an IP list. The AI agent traffic guide shows how to count each of these in your own logs.

RFC 9309, the standard for robots.txt, says its rules "are not a form of access authorization." It is a request. Your logs show whether a bot honours it; the file itself cannot make it.

Our order, from OpenAI's statements above:

1. Allow OAI-SearchBot, in robots.txt and in your firewall. It is the bot tied to ChatGPT search.
2. Decide on GPTBot separately. Blocking it is a fair choice about training and does not affect search.
3. Leave ChatGPT-User alone. robots.txt may not apply to it anyway, and it is a visit a shopper asked for.
4. Check that no Disallow: / under * is catching OpenAI's bots by accident.
5. Verify visitors by IP list or signature before you trust or block them by name.

For every other vendor's bots and a generator for the whole file, see the robots.txt guide. For what ChatGPT reads beyond your pages, product feeds included, see how ChatGPT, Perplexity and Google shop.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content "that may be used in training our generative AI foundation models". OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features". OpenAI says each setting is independent, so you can allow one and block the other.

Does blocking GPTBot remove my store from ChatGPT search?

No. Blocking GPTBot signals that your content should not be used for training. Blocking OAI-SearchBot is what removes you: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."

Does ChatGPT-User follow robots.txt?

Not necessarily. OpenAI says "Because these actions are initiated by a user, robots.txt rules may not apply," and that ChatGPT-User is not used to decide what appears in search.

How long does a robots.txt change take to reach ChatGPT search?

OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust."

How do I know a request claiming to be GPTBot is real?

Check its IP address against OpenAI's published list for that bot, such as openai.com/gptbot.json. The User-Agent string alone proves nothing. Our agent readiness scanner checks what your robots.txt says to OpenAI's bots and to the rest of the agents on our list.

The sources below were last checked on the date shown.