Skip to content
Sign in

VER-001Beside the path

Bot management present

Checks whether anything in front of your shop can tell automated clients apart from shoppers, so you can let verified agents in and keep scrapers out.

What we check

We look for bot management: a product that sorts automated traffic from human traffic. We recognise vendors from response headers, cookies and page markup, including Cloudflare Bot Management, DataDome, Akamai Bot Manager, HUMAN, Kasada, Imperva and Shopify's bot protection. We also count it as present if, during our crawl, something other than your shop answered one of our requests with a bot wall. A CDN on its own (Cloudflare, Akamai, Fastly, CloudFront, Vercel, Bunny) is caching and delivery, not bot management, and is reported separately. We also note CAPTCHA products such as reCAPTCHA or hCaptcha.

Why it matters

You will want to allow agents acting for real shoppers while limiting scrapers. That choice needs something in front of your site that can tell them apart; without it you can only leave everything open or block broadly, turning away agents that would have bought.

How we check

We make no extra requests; we read the fingerprints and bot-wall answers our crawl recorded.

PASS: a bot management vendor was fingerprinted, or a bot wall answered at least one crawl request. WARN: a CAPTCHA product is present but no bot management; or a CDN is in front of the site but no bot management was detected. FAIL: no bot management, CAPTCHA product or CDN detected. SKIPPED: no page we requested was answered by your shop itself.

A fingerprint shows that a product is installed, not how strictly it is configured: a shop that sets a vendor's cookie but challenges nobody looks the same to this check as one that blocks aggressively.

How to fix it

  1. Turn on your CDN's or firewall's bot management (for example Cloudflare Bot Management, Akamai Bot Manager, DataDome or HUMAN) in monitoring mode first.
  2. After about a week of traffic, move it to challenge mode on cart, checkout and search.
  3. Keep an allow path for verified agent traffic, such as requests signed with Web Bot Auth or coming from an agent operator's published IP ranges, so real shopping agents still work.

What good looks like

Response headers from a shop behind Cloudflare with bot management on:

HTTP/2 200
server: cloudflare
cf-ray: 8c1a2b3c4d5e6f70-LHR
set-cookie: __cf_bm=...; path=/; HttpOnly; Secure; SameSite=None

Or from a shop using DataDome:

HTTP/2 200
x-datadome: protected
set-cookie: datadome=...; Path=/; Secure

Sources

Read more

  • Bots, crawlers and agents: who is knocking

    A User-Agent string can be spoofed, and robots.txt rules are a request to crawlers rather than access authorization. Web Bot Auth, an IETF draft, would let an agent sign its requests so a server can verify who sent them.

Could an agent buy from your store?

The free scan runs all 51 checks on your store in about five minutes.

Scan my store
The storewalk.ai app on a phone, scanning a clothing store: it checks a product, views its details and compares options.