How to see AI agent traffic in analytics and Cloudflare

Because GA4 hears about a visit from the page itself. Google tells you to "Place the Google tag snippet immediately after the opening HTML tag on every page you want to measure", and that snippet is JavaScript. A crawler that downloads your HTML and never runs its scripts sends GA4 nothing. The vendors' crawler pages do not say whether their bots run JavaScript, so that step is our reading of how the tag works, and it is why we would never measure agents with GA4 alone.

GA4 also filters on purpose. Google's help page says "traffic from known bots and spiders is automatically excluded", using "a combination of Google research and the International Spiders and Bots List", and "At this time, you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded." So a bot that does run your tag may be dropped before you see it, and you cannot tell how many were. Asking GA4 how many bots visited is like asking the guest book how many burglars came in.

Pick a visitor and see which tool records it:

Open AI Crawl Control in the dashboard. Cloudflare says "AI Crawl Control (formerly AI Audit) gives you visibility into which AI services are accessing your content", and it is "Available on all plans". In our reading it works from the requests that pass through Cloudflare, so it needs no script in your pages. Its metrics, from Cloudflare's own list, cover:

• total, allowed and unsuccessful requests, and referrals, which are "(paid plans only)";
• crawlers grouped by operator ("OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta");
• the status codes they got, the format of what they fetched, and the paths they read most.

Its robots.txt view has a violations table that "identifies AI crawlers that have requested paths explicitly disallowed by your robots.txt file", which is a fast way to find out who ignores your file. Cloudflare's bot reference names the crawlers it means, "A selection of crawlers from major AI operators", each with a category: GPTBot and ClaudeBot are "AI Crawler", OAI-SearchBot and PerplexityBot "AI Search", and ChatGPT-User, Claude-User and Perplexity-User "AI Assistant". For an up-to-date list of verified bots, it points to the Cloudflare Radar bots directory.

Cloudflare's separate Web Analytics product works the other way. It needs "a JavaScript snippet on your page", which "(also known as a beacon) collects metrics using the Performance API", so it has the same blind spot as GA4.

Search the User-Agent field. nginx's default combined format, which "The configuration always includes", already records it: log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';. Apache documents its "NCSA extended/combined log format" as "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"", the same fields. Every request lands in the log whether or not the client runs JavaScript, so this is where the crawlers and assistants show up.

Paste a few lines from your own log. The example is invented, on documentation addresses:

A User-Agent is a claim, and anyone can type "GPTBot". To check a line, compare its address with the vendor's published list, or do a reverse DNS lookup where the vendor uses one; the classifier names the method each vendor gives. A request signed with Web Bot Auth can be checked cryptographically, which the bots, crawlers and agents article explains. And an agent that drives a real browser sends a browser's User-Agent, so in a plain log it looks like a shopper unless it signs its requests.

With a short command on the server. The builder writes one over the combined format, using the names on our agent list:

awk -F'"' 'match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|Google-Agent|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|meta-externalfetcher|Applebot|bingbot|Bytespider|CCBot/) { print substr($6, RSTART, RLENGTH) }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

Run it over a week of logs. Crawlers come in bursts, and a single quiet hour says very little. If your store sits behind Cloudflare or another CDN, count there as well: your server only sees the requests the CDN passes on, so a page served from the CDN's cache never reaches your log.

Start with the status codes. A run of 403 or 429 answers to search crawlers or assistants means a firewall rule or a rate limit is turning them away, and those are the agents that can show your products to shoppers. The last line of the example log is a 403 for ClaudeBot at /cart, a training crawler at a page you may well have meant to keep private, so not every refusal is a mistake. The bot protection guide shows how to let the right ones through.

Then split the assistants from the crawlers. An assistant line is a fetch that a person asked for: OpenAI says of ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply", and Anthropic describes Claude-User's visits as "user-initiated requests". A rise in those lines is people asking assistants about your products, which in our view is worth more to a shop than a crawler's routine pass.

Last, look at the paths. Product and collection pages, robots.txt, and any discovery file you publish, such as /.well-known/ucp or /llms.txt, are what agents come for. If an assistant reads your product pages but your policy pages never appear, it may be answering delivery questions without them.

Here GA4 is useful, because the visitor is a person with a browser. GA4 has an AI Assistant channel: "The medium is set to "ai-assistant" and the campaign is set to "(ai-assistant)" if the referrer matches a list of AI Assistants." Google does not publish that list on the page. In AI Crawl Control, referrals from AI platforms are a paid-plan view. In your logs, the referrer field shows the assistant's site when its app sends one, as the second line of the example log does.

Keep the two numbers apart: a crawler reading your product page and a shopper clicking a link in an answer are different events.

Our scan reads your store from the outside and never sees your logs, so our MEAS-001 check, "Analytics distinguishes AI agents from humans", is currently reported as not measured. What the scan does read is the other side: whether your robots.txt lets these agents in, and whether a CAPTCHA on your checkout stands in their way. The robots.txt guide and the bot protection guide cover those.

Does Google Analytics track AI crawlers?

Not in our reading. GA4 counts what its tag reports, the tag is a script, and a crawler that does not run the page's scripts sends nothing. GA4 also excludes known bots automatically, and Google says "you cannot disable known bot traffic exclusion".

Is Cloudflare AI Crawl Control free?

It is "Available on all plans". Some views are not: referral counts are marked "(paid plans only)".

How do I find GPTBot in my server logs?

Search the User-Agent field for GPTBot, or paste a few lines into the classifier above. Then check the address against OpenAI's published list, since anyone can send that name.

Can I see people who came from ChatGPT?

Yes, in GA4's AI Assistant channel when the referrer matches Google's list, and in your logs when the assistant's app sends a referrer.

Does Storewalk measure my AI traffic?

No. Our scan cannot see your logs, so MEAS-001 is reported as not measured. Our agent readiness scanner checks whether agents can get in and find what they need.

The sources below were last checked on the date shown.