Bot protection that doesn’t block good AI agents

Whether it tells you the truth about itself, mostly. Cloudflare defines a verified bot as "a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does". Proving it takes more than a name. A bot can publish the IP addresses it crawls from, or sign its requests with Web Bot Auth, a draft standard for cryptographic signatures on bot traffic. Signed agents, in Cloudflare's words, "are agents that are generally directed by an end user instead of a single company or entity", and since 1 July 2026 "the distinction between a Verified bot and a signed agent is expressed by a new metadata field". Cloudflare's behaviour list for these bots includes "User-directed agents visiting a page on behalf of a human" and "Checkout or other transaction actions on behalf of users".

The sender of a request writes its User-Agent itself, so a scraper can call itself GPTBot as easily as GPTBot can. Signatures and IP lists are the part a scraper cannot copy. What you do with a verified bot is still up to you. AWS, whose browser for agents signs requests, writes that "Domain owners retain full control over their bot policies and may block, monitor, or rate-limit agent traffic regardless of cryptographic signatures."

The planner runs a rule against our own list of AI crawlers and agents, the list our robots.txt check reads. Write a rule the way a firewall would: a word in the User-Agent, a kind of bot, or anything its vendor gives you no way to verify. Then choose block, challenge or log, and the whole site or the cart and checkout only.

Type "bot" and you stop every search bot on our list and most training crawlers, plus some assistants fetching for a shopper, which costs you search results. OpenAI writes "we recommend allowing OAI-SearchBot". Type "Chrome" and you stop shoppers, because ordinary browsers send that word too. The ChatGPT agent sends a browser's User-Agent and identifies itself with a signature, so on our list no word rule picks it out by name. Two names on the list, Google-Extended and Applebot-Extended, are robots.txt tokens that no crawler sends, so a firewall rule never sees them; their place is the robots.txt file.

Cloudflare's AI Crawl Control works per crawler: "For each AI crawler, you can choose to allow or block access", and when you block one, "the system creates or updates a WAF custom rule on your zone to enforce that block." That is the planner's middle option with a vendor's directory behind it.

Where money moves, and as late as you can. Two of our checks pull in different directions here, and the figure shows both. VER-003 wants your checkout guarded more strictly than your product pages. TRX-002 fails a CAPTCHA a visitor has to click on the cart or checkout entry, because an agent stops there. The shape that suits both is a check that scores the visit without asking anything at the checkout entry, and a puzzle, if you want one at all, only when the order is submitted. Choose a control for each page, or start from a preset.

Our scan never submits an order, so a challenge on the order submit is invisible to it, and to VER-003. A bank-vault checkout with a checkbox on every step passes VER-003 and fails TRX-002, and it would stop the agent at the door. A wall on every page passes VER-001, and VER-003 warns, because checkout and product pages carry the same control. Our checks also read any Turnstile widget on the cart or checkout as interactive, whatever its mode, which the guest checkout guide explains.

Anthropic says "we will not attempt to bypass CAPTCHAs for the sites we crawl." Cloudflare's challenge pages fail "when the browser expects a non-HTML response, such as an AJAX or XHR (fetch) request", and in our reading an agent that calls your store's API is that kind of client. hCaptcha's advice is closest to the shape above: "Adding a new payment method, changing the delivery address, or submitting an order introduces a different consequence", and "Sensitive activity can wait for confirmation."

Pick a vendor to see what its own pages say about agents. Where a page says nothing on a point, the figure says so.

Cloudflare's Bot Fight Mode takes no exceptions from your own rules: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules", and it "may challenge API or mobile app traffic". If you want to let some agents through and stop others, it is the wrong tool. Shopify's bot protection "is available only for merchants on a Shopify Plus plan". It runs as an event of at most 60 minutes with a reCAPTCHA or hCaptcha challenge, so that "your customers have the opportunity to purchase scarce products during flash sales", and inside an event "The setting to stop bots from auto-completing the checkout is active by default". Its page does not mention AI agents. For a flash sale of limited trainers that may be what you want; on an ordinary Tuesday it is a bouncer at an empty door.

Our suggested order, smallest risk first:

1. Look at your traffic before you write rules. Cloudflare says AI Crawl Control is "Available on all plans"; your server logs show the same if you match user agents against each vendor's published IP list.
2. Allow verified bots and signed agents. Cloudflare writes that "Historically, Verified bots have been excluded in default bot configurations across all plans", so check that your own rules have not undone that.
3. Decide about training crawlers separately, in robots.txt and, if you want it enforced, per crawler. Blocking a training crawler does not touch its vendor's search bot, by the vendors' own accounts.
4. Keep product pages, search and policy pages free of challenges. An agent has to read them before it can buy.
5. On the cart and checkout entry, use a check that scores the visit without asking for a click.
6. If you want a puzzle, put it on the order submit, and look for a vendor that lets you exempt verified agents there.
7. Run a scan to see what our checks find, then watch the traffic again for a week.

VER-001 passes a store where we detect bot management, or where a bot manager answered one of our requests in place of the store. It warns when we find only a CAPTCHA vendor, or only a CDN (a network that caches your pages), and fails when we find neither. VER-003 compares the strongest control on your cart and checkout with the one on your product pages: stricter at checkout passes, equal warns, none fails. TRX-002 fails an interactive CAPTCHA on the cart or checkout entry and warns on a score-based one.

Their limits are ours to state. VER-001 detects that a bot manager is installed, not what it enforces: a store whose Cloudflare challenges nobody looks the same to it as one in block mode. Neither VER-003 nor TRX-002 sees a challenge that only appears on the order submit. And our scan identifies itself as Storewalk/1.0. If your rules stop it, most checks that needed the page skip rather than fail, though a challenge page on your cart or checkout still counts against TRX-002. Our crawler page says how to recognise the scan or block it.

Should I block AI bots to protect my store?

Block the ones you have a reason to block, by name or per crawler, and leave search bots and shoppers' assistants alone. A rule that matches names stops only bots honest enough to use their real name. In our view, abuse is better handled by rate limits and by checks on checkout.

What is a verified bot?

In Cloudflare's words, "a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does". In practice that means a bot whose requests can be checked against something it publishes, such as an IP list or a signing key.

Will a CAPTCHA on checkout stop AI agents?

A puzzle can. Anthropic says its bots will not try to get past one, and our TRX-002 fails an interactive CAPTCHA on the cart or checkout entry. A score-based check asks for nothing, though it can still refuse a visit; TRX-002 warns on it.

Can I exempt AI agents from Cloudflare Bot Fight Mode?

Not with WAF custom rules or Page Rules; Cloudflare says Bot Fight Mode cannot be bypassed that way. Cloudflare's page does not say how it treats verified bots, so if you need exceptions, use a product that takes them.

Does Shopify's bot protection block AI agents?

It "acts as a fairness control by blocking known bots from your checkout" during an event you schedule, and its page does not mention AI agents. It is available on Shopify Plus only and lasts at most 60 minutes per event.

The sources below were last checked on the date shown.