# Bot protection that doesn’t block good AI agents How to keep scrapers out without turning away search bots and shopping agents: a firewall rule planner, where to put a challenge, and what each vendor says it does about agents. By Valerii Checha, 3 Nov 2026 How do you protect a store from bots without blocking good AI agents? Leave product pages open, put the stricter controls on checkout, use a check that scores the visit instead of a puzzle, and allow bots that can prove who they are instead of matching names in the User-Agent. A rule that matches names stops the honest bots that use them and lets through anyone who borrows one. This guide covers what tells a good bot from a bad one, a planner that shows which crawlers and agents a firewall rule would stop, where to put a challenge, and what the main vendors say they do about agents. Storewalk is ours: we build the scan, and the [bot management](https://storewalk.ai/checks/VER-001), [checkout controls](https://storewalk.ai/checks/VER-003) and [CAPTCHA](https://storewalk.ai/checks/TRX-002) checks below are ours. The vendors are quoted from their own pages, read on 3 November 2026. For how a bot identifies itself in the first place, read [bots, crawlers and agents](https://storewalk.ai/blog/bots-crawlers-agents); for the other walls between an agent and a sale, [the hub article](https://storewalk.ai/blog/why-agents-cant-buy). ## What separates a good bot from a bad one? Whether it tells you the truth about itself, mostly. Cloudflare defines a verified bot as "a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does". Proving it takes more than a name. A bot can publish the IP addresses it crawls from, or sign its requests with Web Bot Auth, a draft standard for cryptographic signatures on bot traffic. Signed agents, in Cloudflare's words, "are agents that are generally directed by an end user instead of a single company or entity", and since 1 July 2026 "the distinction between a Verified bot and a signed agent is expressed by a new metadata field". Cloudflare's behaviour list for these bots includes "User-directed agents visiting a page on behalf of a human" and "Checkout or other transaction actions on behalf of users". The sender of a request writes its User-Agent itself, so a scraper can call itself GPTBot as easily as GPTBot can. Signatures and IP lists are the part a scraper cannot copy. What you do with a verified bot is still up to you. AWS, whose browser for agents signs requests, writes that "Domain owners retain full control over their bot policies and may block, monitor, or rate-limit agent traffic regardless of cryptographic signatures." ## Which agents would your firewall rule stop? The planner runs a rule against our own list of AI crawlers and agents, the list our robots.txt check reads. Write a rule the way a firewall would: a word in the User-Agent, a kind of bot, or anything its vendor gives you no way to verify. Then choose block, challenge or log, and the whole site or the cart and checkout only. A firewall rule planner: which AI crawlers and agents your rule would stop. On the page, you write a rule (a User-Agent word, a kind of bot, or anything its vendor gives no way to verify), choose block, challenge or log, and the whole site or the cart and checkout, and see each crawler and agent on our list. The list: | Bot | Vendor | Kind | Verified by | | -------------------- | ------------ | -------------------- | ---------------------- | | GPTBot | OpenAI | Training crawlers | published IP list | | OAI-SearchBot | OpenAI | Search bots | published IP list | | ChatGPT-User | OpenAI | Fetchers for a user | published IP list | | ChatGPT agent | OpenAI | Buying agents | Web Bot Auth signature | | ClaudeBot | Anthropic | Training crawlers | published IP list | | Claude-User | Anthropic | Fetchers for a user | published IP list | | Claude-SearchBot | Anthropic | Search bots | published IP list | | PerplexityBot | Perplexity | Search bots | published IP list | | Perplexity-User | Perplexity | Fetchers for a user | published IP list | | Googlebot | Google | Search bots | reverse DNS | | Google-Extended | Google | robots.txt name only | no published method | | Google-Agent | Google | Fetchers for a user | reverse DNS | | Amazonbot | Amazon | Training crawlers | published IP list | | Amzn-SearchBot | Amazon | Search bots | published IP list | | Amzn-User | Amazon | Fetchers for a user | published IP list | | meta-externalagent | Meta | Training crawlers | no published method | | meta-externalfetcher | Meta | Fetchers for a user | no published method | | Applebot | Apple | Search bots | reverse DNS | | Applebot-Extended | Apple | robots.txt name only | no published method | | bingbot | Microsoft | Search bots | published IP list | | Bytespider | ByteDance | Training crawlers | no published method | | CCBot | Common Crawl | Training crawlers | reverse DNS | - Block “GPTBot, CCBot, Bytespider” in the User-Agent, whole site: stops GPTBot, Bytespider, CCBot. The ChatGPT agent sends a browser’s User-Agent and identifies itself with a signature, so no word rule picks it out by name; a browser word catches it along with shoppers. Anyone can type a bot’s name. A User-Agent rule stops honest bots and lets a liar through. - Block search bots, whole site: stops OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Amzn-SearchBot, Applebot, bingbot. It stops search bots. OpenAI writes “we recommend allowing OAI-SearchBot”, and Perplexity says the same of PerplexityBot. - Challenge anything its vendor gives no way to verify, cart and checkout only: stops meta-externalagent, meta-externalfetcher, Bytespider. It stops assistants working for a shopper right now. “Verifiable” here means the vendor publishes a way to check; your bot manager has to use it. Sources: [Cloudflare docs: AI Crawl Control, manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) and [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots). Type "bot" and you stop every search bot on our list and most training crawlers, plus some assistants fetching for a shopper, which costs you search results. OpenAI writes "we recommend allowing OAI-SearchBot". Type "Chrome" and you stop shoppers, because ordinary browsers send that word too. The ChatGPT agent sends a browser's User-Agent and identifies itself with a signature, so on our list no word rule picks it out by name. Two names on the list, Google-Extended and Applebot-Extended, are robots.txt tokens that no crawler sends, so a firewall rule never sees them; their place is the [robots.txt file](https://storewalk.ai/guides/robots-txt-for-ai-agents). Cloudflare's AI Crawl Control works per crawler: "For each AI crawler, you can choose to allow or block access", and when you block one, "the system creates or updates a WAF custom rule on your zone to enforce that block." That is the planner's middle option with a vendor's directory behind it. ## Where should the challenge go? Where money moves, and as late as you can. Two of our checks pull in different directions here, and the figure shows both. [VER-003](https://storewalk.ai/checks/VER-003) wants your checkout guarded more strictly than your product pages. [TRX-002](https://storewalk.ai/checks/TRX-002) fails a CAPTCHA a visitor has to click on the cart or checkout entry, because an agent stops there. The shape that suits both is a check that scores the visit without asking anything at the checkout entry, and a puzzle, if you want one at all, only when the order is submitted. Choose a control for each page, or start from a preset. Where to put the challenge: what an agent meets, and what our checks say. On the page, you choose a control for the product pages, the cart, the checkout entry and the order submit, and whether a bot manager sits in front of the site. What an agent meets is our reading. The presets: - Doors open (Product pages: Nothing; Cart: Nothing; Checkout entry: Nothing; Order submit: Nothing; bot manager no): TRX-002 PASS, No CAPTCHA widget on cart or checkout for an anonymous visitor; VER-003 FAIL, No automation controls detected on checkout; VER-001 FAIL, No bot management or CDN protection detected. - Checkout like a bank vault (Product pages: Nothing; Cart: reCAPTCHA checkbox (v2); Checkout entry: reCAPTCHA checkbox (v2); Order submit: reCAPTCHA checkbox (v2); bot manager no): TRX-002 FAIL, interactive CAPTCHA before payment; VER-003 PASS, Checkout is protected more strictly than product pages; VER-001 WARN, CAPTCHA present but no bot management in front of the site. - The same wall everywhere (Product pages: Bot manager’s challenge page; Cart: Bot manager’s challenge page; Checkout entry: Bot manager’s challenge page; Order submit: Bot manager’s challenge page; bot manager yes): TRX-002 FAIL, interactive CAPTCHA before payment; VER-003 WARN, Checkout and product pages carry the same controls; VER-001 PASS, Bot management detected; Most other checks SKIPPED, With product pages behind a wall, the checks that read them skip, since what they got was the wall. - Open shelves, a quiet till (Product pages: Nothing; Cart: Nothing; Checkout entry: Score-based reCAPTCHA (v3); Order submit: reCAPTCHA checkbox (v2); bot manager yes): TRX-002 WARN, invisible/score-based CAPTCHA; it may refuse an agent without showing a challenge; VER-003 PASS, Checkout is protected more strictly than product pages; VER-001 PASS, Bot management detected. Our scan never submits an order, so a challenge on the order submit is invisible to it, and to VER-003. A bank-vault checkout with a checkbox on every step passes VER-003 and fails TRX-002, and it would stop the agent at the door. A wall on every page passes VER-001, and VER-003 warns, because checkout and product pages carry the same control. Our checks also read any Turnstile widget on the cart or checkout as interactive, whatever its mode, which the [guest checkout guide](https://storewalk.ai/guides/guest-checkout-ai-agents) explains. Anthropic says "we will not attempt to bypass CAPTCHAs for the sites we crawl." Cloudflare's challenge pages fail "when the browser expects a non-HTML response, such as an AJAX or XHR (fetch) request", and in our reading an agent that calls your store's API is that kind of client. hCaptcha's advice is closest to the shape above: "Adding a new payment method, changing the delivery address, or submitting an order introduces a different consequence", and "Sensitive activity can wait for confirmation." ## What do bot protection vendors offer for AI agents? Pick a vendor to see what its own pages say about agents. Where a page says nothing on a point, the figure says so. What each bot protection vendor says it does about AI agents. Cloudflare. Tells agents apart: A verified bot is “a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does”. Since 1 July 2026, “the distinction between a Verified bot and a signed agent is expressed by a new metadata field”. Its behaviour list includes “User-directed agents visiting a page on behalf of a human” and “Checkout or other transaction actions on behalf of users”. Per-agent control: In AI Crawl Control, “For each AI crawler, you can choose to allow or block access”, and for a block “the system creates or updates a WAF custom rule on your zone to enforce that block.” Watch out for: Bot Fight Mode: “You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules.” It “may challenge API or mobile app traffic”. Its page does not say how verified bots fare. Defaults: “Historically, Verified bots have been excluded in default bot configurations across all plans.” Sources: [Cloudflare docs: Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), [Cloudflare docs: AI Crawl Control, manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) and [Cloudflare docs: Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/). Google Cloud Fraud Defense (reCAPTCHA). Tells agents apart: Its Agent overview dashboard separates “Verified agent sessions” from suspected ones, which “self-declare as agents, such as through HTTP headers, or exhibit agentic behavior, but whose identity can’t be officially verified”. It verifies by “Web Bot Auth (WBA)” or by User-Agent and IP. Per-agent control: The page is about monitoring: it is there so “you can refine your security policies without disrupting valuable automated traffic like shopping assistants or search indexers.” Watch out for: Not stated on the page. Defaults: Not stated on the page. Source: [Google Cloud Fraud Defense: Monitor agent traffic](https://docs.cloud.google.com/recaptcha/docs/monitor-agents). hCaptcha. Tells agents apart: “hCaptcha also supports Web Bot Auth cryptographic verification for partner platforms that sign their requests.” “A signed request remains subject to hCaptcha’s risk assessment.” Per-agent control: “Permit catalog searches and other low-risk activity from agents that meet your policy.” “Restrict agents on flows where automation is prohibited, even when the request has a valid signature.” Watch out for: It treats checkout steps as their own decision: “Adding a new payment method, changing the delivery address, or submitting an order introduces a different consequence”. Defaults: Not stated on the page. Sources: [hCaptcha: Agent Controls](https://www.hcaptcha.com/agent-controls) and [hCaptcha: AI agent verification](https://www.hcaptcha.com/learning/ai-agents/ai-agent-verification/). Shopify Plus bot protection. Tells agents apart: Not stated. It “acts as a fairness control by blocking known bots from your checkout”. Per-agent control: None for agents. You schedule an event and “must select a checkpoint challenge”: reCAPTCHA or hCaptcha. The page does not mention AI agents. Watch out for: “The maximum duration for a bot protection event is 60 minutes.” Defaults: In an event, “The setting to stop bots from auto-completing the checkout is active by default”. The feature is “available only for merchants on a Shopify Plus plan.” Source: [Shopify Help Center: Bot protection](https://help.shopify.com/en/manual/checkout-settings/bot-protection). WooCommerce with Turnstile. Tells agents apart: Not stated. Turnstile “protects your forms from spam and bot abuse without showing CAPTCHAs or puzzles to real users.” Per-agent control: None stated. You choose the forms, “Classic Checkout” among them, and the docs say to choose managed widget mode. Watch out for: The same page suggests Modern Login, where “the checkout form is only shown after the user is logged in”. Defaults: Not stated; it is part of the Enhanced Customer Accounts documentation. Source: [WooCommerce docs: Configuring Cloudflare Turnstile (Enhanced Customer Accounts)](https://woocommerce.com/document/enhanced-customer-accounts/configuring-cloudflare-turnstile/). Cloudflare's Bot Fight Mode takes no exceptions from your own rules: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules", and it "may challenge API or mobile app traffic". If you want to let some agents through and stop others, it is the wrong tool. Shopify's bot protection "is available only for merchants on a Shopify Plus plan". It runs as an event of at most 60 minutes with a reCAPTCHA or hCaptcha challenge, so that "your customers have the opportunity to purchase scarce products during flash sales", and inside an event "The setting to stop bots from auto-completing the checkout is active by default". Its page does not mention AI agents. For a flash sale of limited trainers that may be what you want; on an ordinary Tuesday it is a bouncer at an empty door. ## How do you set it up, step by step? Our suggested order, smallest risk first: 1. Look at your traffic before you write rules. Cloudflare says AI Crawl Control is "Available on all plans"; your server logs show the same if you match user agents against each vendor's published IP list. 2. Allow verified bots and signed agents. Cloudflare writes that "Historically, Verified bots have been excluded in default bot configurations across all plans", so check that your own rules have not undone that. 3. Decide about training crawlers separately, in robots.txt and, if you want it enforced, per crawler. Blocking a training crawler does not touch its vendor's search bot, by the vendors' own accounts. 4. Keep product pages, search and policy pages free of challenges. An agent has to read them before it can buy. 5. On the cart and checkout entry, use a check that scores the visit without asking for a click. 6. If you want a puzzle, put it on the order submit, and look for a vendor that lets you exempt verified agents there. 7. Run a scan to see what our checks find, then watch the traffic again for a week. ## What do our checks say about bot protection? [VER-001](https://storewalk.ai/checks/VER-001) passes a store where we detect bot management, or where a bot manager answered one of our requests in place of the store. It warns when we find only a CAPTCHA vendor, or only a CDN (a network that caches your pages), and fails when we find neither. [VER-003](https://storewalk.ai/checks/VER-003) compares the strongest control on your cart and checkout with the one on your product pages: stricter at checkout passes, equal warns, none fails. [TRX-002](https://storewalk.ai/checks/TRX-002) fails an interactive CAPTCHA on the cart or checkout entry and warns on a score-based one. Their limits are ours to state. VER-001 detects that a bot manager is installed, not what it enforces: a store whose Cloudflare challenges nobody looks the same to it as one in block mode. Neither VER-003 nor TRX-002 sees a challenge that only appears on the order submit. And our scan identifies itself as Storewalk/1.0. If your rules stop it, most checks that needed the page skip rather than fail, though a challenge page on your cart or checkout still counts against TRX-002. Our [crawler page](https://storewalk.ai/bot) says how to recognise the scan or block it. ## Frequently asked questions ### Should I block AI bots to protect my store? Block the ones you have a reason to block, by name or per crawler, and leave search bots and shoppers' assistants alone. A rule that matches names stops only bots honest enough to use their real name. In our view, abuse is better handled by rate limits and by checks on checkout. ### What is a verified bot? In Cloudflare's words, "a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does". In practice that means a bot whose requests can be checked against something it publishes, such as an IP list or a signing key. ### Will a CAPTCHA on checkout stop AI agents? A puzzle can. Anthropic says its bots will not try to get past one, and our TRX-002 fails an interactive CAPTCHA on the cart or checkout entry. A score-based check asks for nothing, though it can still refuse a visit; TRX-002 warns on it. ### Can I exempt AI agents from Cloudflare Bot Fight Mode? Not with WAF custom rules or Page Rules; Cloudflare says Bot Fight Mode cannot be bypassed that way. Cloudflare's page does not say how it treats verified bots, so if you need exceptions, use a product that takes them. ### Does Shopify's bot protection block AI agents? It "acts as a fairness control by blocking known bots from your checkout" during an event you schedule, and its page does not mention AI agents. It is available on Shopify Plus only and lasts at most 60 minutes per event. The sources below were last checked on the date shown. ## Sources - [Cloudflare docs: Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) - [Cloudflare blog: signed agents (28 August 2025)](https://blog.cloudflare.com/signed-agents) - [Cloudflare docs: AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) - [Cloudflare docs: AI Crawl Control, manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) - [Cloudflare docs: Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/) - [Cloudflare docs: Challenge pages](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/) - [Cloudflare docs: Turnstile widgets](https://developers.cloudflare.com/turnstile/concepts/widget/) - [Google Cloud Fraud Defense: Monitor agent traffic](https://docs.cloud.google.com/recaptcha/docs/monitor-agents) - [hCaptcha: Agent Controls](https://www.hcaptcha.com/agent-controls) - [hCaptcha: AI agent verification](https://www.hcaptcha.com/learning/ai-agents/ai-agent-verification/) - [Shopify Help Center: Bot protection](https://help.shopify.com/en/manual/checkout-settings/bot-protection) - [WooCommerce docs: Configuring Cloudflare Turnstile (Enhanced Customer Accounts)](https://woocommerce.com/document/enhanced-customer-accounts/configuring-cloudflare-turnstile/) - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) - [Anthropic: Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - [AWS: Reducing CAPTCHAs with Web Bot Auth](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html) Last checked 3 Nov 2026