Skip to content

MEAS-001Beside the path

Analytics distinguishes AI agents from humans

Asks whether your analytics can tell AI agents apart from human visitors; this is only measurable from your own logs, so the scan currently reports it as not measured.

What we check

Whether you can see how much of your traffic comes from AI agents, which agents they are, and what they do on your site. That can only be answered from your server, CDN or analytics logs, not from outside. Log upload is not available in this version of the scan, so this check does not look at your site.

Why it matters

Agents that search, compare and buy for shoppers are a separate channel from human visitors. Without separating them you cannot see whether agents reach your product pages, where they stop, or whether changes you make for them have any effect. Some browse with an ordinary browser user agent, which makes them easy to miss.

How we check

The scanner makes no requests for this check. It is always reported as SKIPPED with the reason NO_TRAFFIC_UPLOAD ("Agent share of traffic is only measurable from uploaded server logs"). We do not estimate or guess agent traffic, and the MEASURE stage is shown as not assessed rather than as a failure.

How to fix it

  1. Keep access logs from your server, CDN or hosting provider with the timestamp, full User-Agent, path, status code and client IP.
  2. Group requests by the user-agent tokens vendors publish, for example GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity), and Googlebot. Keep training crawlers, search crawlers and user-triggered fetchers apart.
  3. Verify claimed identities against the IP lists or reverse DNS the vendors publish; a User-Agent alone can be spoofed.
  4. Some agents sign requests with Web Bot Auth (an IETF working-group draft, -00, 2026-09-01); the Signature-Agent header identifies the signer, for example https://chatgpt.com for OpenAI's ChatGPT agent. Log it if your stack allows.
  5. On hosted platforms without raw log access, use your CDN's or platform's own bot reporting where available.

What good looks like

Log fields to keep per request:
  timestamp, client IP, method, path, status, User-Agent, Signature-Agent

User-agent tokens to group by (from vendor documentation):
  training crawlers:        GPTBot, ClaudeBot
  search crawlers:          OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot
  user-triggered fetchers:  ChatGPT-User, Claude-User, Perplexity-User

Verification sources:
  https://openai.com/gptbot.json
  https://claude.com/crawling/bots.json
  https://www.perplexity.ai/perplexitybot.json
  Googlebot: reverse DNS ending in googlebot.com

Sources