Skip to content

VER-001Beside the path

Bot management present

Checks whether anything in front of your shop can tell automated clients apart from shoppers, so you can let verified agents in and keep scrapers out.

What we check

We look for bot management: a product that sorts automated traffic from human traffic. We recognise vendors from response headers, cookies and page markup, including Cloudflare Bot Management, DataDome, Akamai Bot Manager, HUMAN, Kasada, Imperva and Shopify's bot protection. We also count it as present if, during our crawl, something other than your shop answered one of our requests with a bot wall. A CDN on its own (Cloudflare, Akamai, Fastly, CloudFront, Vercel, Bunny) is caching and delivery, not bot management, and is reported separately. CAPTCHA products such as reCAPTCHA or hCaptcha are noted too.

Why it matters

You will want to allow agents acting for real shoppers while limiting scrapers. That choice needs something in front of your site that can tell them apart; without it you can only leave everything open or block broadly, turning away agents that would have bought.

How we check

We make no extra requests; we read the fingerprints and bot-wall answers our crawl recorded.

PASS: a bot management vendor was fingerprinted, or a bot wall answered at least one crawl request. WARN: a CAPTCHA product is present but no bot management; or a CDN is in front of the site but no bot management was detected. FAIL: no bot management, CAPTCHA product or CDN detected. SKIPPED: no page we requested was answered by your shop itself.

A fingerprint shows that a product is installed, not how strictly it is configured: a shop that sets a vendor's cookie but challenges nobody looks the same to this check as one that blocks aggressively.

How to fix it

  1. Turn on your CDN's or firewall's bot management (for example Cloudflare Bot Management, Akamai Bot Manager, DataDome or HUMAN) in monitoring mode first.
  2. After about a week of traffic, move it to challenge mode on cart, checkout and search.
  3. Keep an allow path for verified agent traffic, such as requests signed with Web Bot Auth or coming from an agent operator's published IP ranges, so real shopping agents still work.

What good looks like

Response headers from a shop behind Cloudflare with bot management on:

HTTP/2 200
server: cloudflare
cf-ray: 8c1a2b3c4d5e6f70-LHR
set-cookie: __cf_bm=...; path=/; HttpOnly; Secure; SameSite=None

Or from a shop using DataDome:

HTTP/2 200
x-datadome: protected
set-cookie: datadome=...; Path=/; Secure

Sources