# How to see AI agent traffic in analytics and Cloudflare

Where AI crawlers and assistants show up and where they do not: Cloudflare AI Crawl Control, your server logs and Google Analytics, with a log classifier and a command builder.

By Valerii Checha, 4 Nov 2026

How do you see AI agent traffic? Look where the requests arrive: at your CDN and in your server logs. Cloudflare's AI Crawl Control, "Available on all plans", shows which AI crawlers and assistants reach your site, and a server log records every request with its User-Agent. Google Analytics counts visits through a script in the page, so in our reading it misses the agents that never run it, though it does show the people who click through from an assistant.

This guide covers what each tool can and cannot see, how to read AI traffic in Cloudflare, how to find agents in your own logs, and how to count the people who arrive from ChatGPT and other assistants. It comes with a classifier for log lines and a command builder that run in this page. Storewalk is ours: we build the scan, and the agent list the classifier uses is the one [our robots.txt check](https://storewalk.ai/checks/DISC-001) reads. Cloudflare, Google and nginx are quoted from their own pages, read on 4 November 2026. For who these bots are, read [bots, crawlers and agents](https://storewalk.ai/blog/bots-crawlers-agents), and for the five walls between an agent and a sale, [the hub article](https://storewalk.ai/blog/why-agents-cant-buy).

## Why doesn't Google Analytics show AI agents?

Because GA4 hears about a visit from the page itself. Google tells you to "Place the Google tag snippet immediately after the opening  HTML tag on every page you want to measure", and that snippet is JavaScript. A crawler that downloads your HTML and never runs its scripts sends GA4 nothing. The vendors' crawler pages do not say whether their bots run JavaScript, so that step is our reading of how the tag works, and it is why we would never measure agents with GA4 alone.

GA4 also filters on purpose. Google's help page says "traffic from known bots and spiders is automatically excluded", using "a combination of Google research and the International Spiders and Bots List", and "At this time, you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded." So a bot that does run your tag may be dropped before you see it, and you cannot tell how many were. Asking GA4 how many bots visited is like asking the guest book how many burglars came in.

Pick a visitor and see which tool records it:

Which tool sees which visitor: Cloudflare, your logs and Google Analytics.

On the page, you pick a kind of visitor and see which tool records it. Every pair:

A training crawler (for example GPTBot, ClaudeBot, CCBot):

- Cloudflare AI Crawl Control: Sees it. It shows which AI services access your content, by crawler and operator; GPTBot is in its bot reference as an AI Crawler.
- Server or CDN logs: Sees it. nginx’s combined format records the client address, the request, the status, the referrer and the User-Agent of every request.
- Google Analytics 4: Misses it. GA4 counts what its tag reports, and the tag is a script in the page (“Place the Google tag snippet immediately after the opening \<head> HTML tag”). A client that does not run the page’s scripts sends it nothing: our reading of how the tag works.
- Cloudflare Web Analytics: Misses it. Web Analytics is “a JavaScript snippet on your page”, so it has GA4’s blind spot: our reading.

An AI search crawler (for example OAI-SearchBot, Claude-SearchBot, PerplexityBot):

- Cloudflare AI Crawl Control: Sees it. OAI-SearchBot and Claude-SearchBot are in its bot reference, as AI Search.
- Server or CDN logs: Sees it. nginx’s combined format records the client address, the request, the status, the referrer and the User-Agent of every request.
- Google Analytics 4: Misses it. GA4 counts what its tag reports, and the tag is a script in the page (“Place the Google tag snippet immediately after the opening \<head> HTML tag”). A client that does not run the page’s scripts sends it nothing: our reading of how the tag works.
- Cloudflare Web Analytics: Misses it. Web Analytics is “a JavaScript snippet on your page”, so it has GA4’s blind spot: our reading.

An assistant fetching a page for a user (for example ChatGPT-User, Claude-User, Perplexity-User):

- Cloudflare AI Crawl Control: Sees it. ChatGPT-User and Claude-User are in its bot reference, as AI Assistant.
- Server or CDN logs: Sees it. nginx’s combined format records the client address, the request, the status, the referrer and the User-Agent of every request.
- Google Analytics 4: Misses it. GA4 counts what its tag reports, and the tag is a script in the page (“Place the Google tag snippet immediately after the opening \<head> HTML tag”). A client that does not run the page’s scripts sends it nothing: our reading of how the tag works.
- Cloudflare Web Analytics: Misses it. Web Analytics is “a JavaScript snippet on your page”, so it has GA4’s blind spot: our reading.

An agent driving a real browser (for example An agent that loads pages and runs their scripts):

- Cloudflare AI Crawl Control: Sometimes. Only if Cloudflare identifies it: its reference is “A selection of crawlers from major AI operators”, and a browser’s User-Agent names no crawler.
- Server or CDN logs: Sometimes. nginx’s combined format records the client address, the request, the status, the referrer and the User-Agent of every request. Telling it from a person takes a signature or the vendor’s address list.
- Google Analytics 4: Sometimes. If it runs the tag, GA4 can count it as a visit, unless GA4 drops it as a known bot, which “you cannot disable” and whose size you cannot see.
- Cloudflare Web Analytics: Sometimes. If it runs the beacon, it is counted like a visitor: our reading.

A person clicking a link in an assistant (for example A shopper following a link from a chat answer):

- Cloudflare AI Crawl Control: Sometimes. Referrals are shown on paid plans only.
- Server or CDN logs: Sees it. The combined format keeps the referrer, so the assistant’s site shows when its app sends one.
- Google Analytics 4: Sees it. The AI Assistant channel: “The medium is set to "ai-assistant" and the campaign is set to "(ai-assistant)" if the referrer matches a list of AI Assistants.”
- Cloudflare Web Analytics: Sees it. A person’s browser runs the beacon.

Sources: [Cloudflare: AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/), [Cloudflare: Analyze AI traffic](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), [Cloudflare: AI Crawl Control bot reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/), [Google tag platform: gtag.js](https://developers.google.com/tag-platform/gtagjs), [Google Analytics: Known bot-traffic exclusion](https://support.google.com/analytics/answer/9888366), [Google Analytics: Default channel group](https://support.google.com/analytics/answer/9756891), [Cloudflare Web Analytics: About](https://developers.cloudflare.com/web-analytics/about/) and [nginx: ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html).

## How do you see AI crawlers in Cloudflare?

Open AI Crawl Control in the dashboard. Cloudflare says "AI Crawl Control (formerly AI Audit) gives you visibility into which AI services are accessing your content", and it is "Available on all plans". In our reading it works from the requests that pass through Cloudflare, so it needs no script in your pages. Its metrics, from Cloudflare's own list, cover:

- total, allowed and unsuccessful requests, and referrals, which are "(paid plans only)";
- crawlers grouped by operator ("OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta");
- the status codes they got, the format of what they fetched, and the paths they read most.

Its robots.txt view has a violations table that "identifies AI crawlers that have requested paths explicitly disallowed by your robots.txt file", which is a fast way to find out who ignores your file. Cloudflare's bot reference names the crawlers it means, "A selection of crawlers from major AI operators", each with a category: GPTBot and ClaudeBot are "AI Crawler", OAI-SearchBot and PerplexityBot "AI Search", and ChatGPT-User, Claude-User and Perplexity-User "AI Assistant". For an up-to-date list of verified bots, it points to the Cloudflare Radar bots directory.

Cloudflare's separate Web Analytics product works the other way. It needs "a JavaScript snippet on your page", which "(also known as a beacon) collects metrics using the Performance API", so it has the same blind spot as GA4.

## How do you find AI agents in your server logs?

Search the User-Agent field. nginx's default `combined` format, which "The configuration always includes", already records it: `log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';`. Apache documents its "NCSA extended/combined log format" as `"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""`, the same fields. Every request lands in the log whether or not the client runs JavaScript, so this is where the crawlers and assistants show up.

Paste a few lines from your own log. The example is invented, on documentation addresses:

A log line classifier: which requests claim to be an AI agent.

On the page, you paste access log lines and each User-Agent is matched against our agent list (the longest matching name wins). For an invented log:

```text
192.0.2.10 - - [04/Nov/2026:09:12:01 +0000] "GET /products/linen-duvet HTTP/1.1" 200 18234 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)"
198.51.100.7 - - [04/Nov/2026:09:12:04 +0000] "GET /products/linen-duvet HTTP/1.1" 200 18234 "https://chatgpt.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"
203.0.113.25 - - [04/Nov/2026:09:12:09 +0000] "GET /robots.txt HTTP/1.1" 200 412 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
192.0.2.44 - - [04/Nov/2026:09:12:15 +0000] "GET /collections/bedding HTTP/1.1" 200 22410 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)"
198.51.100.90 - - [04/Nov/2026:09:12:20 +0000] "GET /.well-known/ucp HTTP/1.1" 200 2140 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"
203.0.113.61 - - [04/Nov/2026:09:12:31 +0000] "GET /products/linen-sheet-set HTTP/1.1" 200 17002 "-" "Mozilla/5.0 (compatible; Amzn-SearchBot/0.1; +https://developer.amazon.com/amazonbot)"
192.0.2.77 - - [04/Nov/2026:09:12:40 +0000] "GET /cart HTTP/1.1" 403 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
```

- Line 1: GPTBot (Training crawler); to check the claim, check the address against the vendor’s IP list.
- Line 2: no agent on our list; a person, or an agent with a browser's User-Agent.
- Line 3: OAI-SearchBot (Search crawler); to check the claim, check the address against the vendor’s IP list.
- Line 4: Claude-User (Assistant, for a user); to check the claim, check the address against the vendor’s IP list.
- Line 5: Perplexity-User (Assistant, for a user); to check the claim, check the address against the vendor’s IP list.
- Line 6: Amzn-SearchBot (Search crawler); to check the claim, check the address against the vendor’s IP list.
- Line 7: ClaudeBot (Training crawler); to check the claim, check the address against the vendor’s IP list.

Sources: [Our agent list (Storewalk)](https://storewalk.ai/checks/DISC-001) and [nginx: ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html).

A User-Agent is a claim, and anyone can type "GPTBot". To check a line, compare its address with the vendor's published list, or do a reverse DNS lookup where the vendor uses one; the classifier names the method each vendor gives. A request signed with Web Bot Auth can be checked cryptographically, which the [bots, crawlers and agents article](https://storewalk.ai/blog/bots-crawlers-agents) explains. And an agent that drives a real browser sends a browser's User-Agent, so in a plain log it looks like a shopper unless it signs its requests.

## How do you count AI agents in an access log?

With a short command on the server. The builder writes one over the combined format, using the names on our agent list:

A command to count AI agents in your own access log.

On the page, you choose the kinds of agent and what to count, and copy a command for an nginx or Apache combined log. With every kind chosen:

Requests per agent:

```sh
awk -F'"' 'match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|Google-Agent|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|meta-externalfetcher|Applebot|bingbot|Bytespider|CCBot/) { print substr($6, RSTART, RLENGTH) }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn
```

Top paths they read:

```sh
awk -F'"' '$6 ~ /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|Google-Agent|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|meta-externalfetcher|Applebot|bingbot|Bytespider|CCBot/ { split($2, r, " "); print r[2] }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20
```

Status codes they got:

```sh
awk -F'"' '$6 ~ /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|Google-Agent|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|meta-externalfetcher|Applebot|bingbot|Bytespider|CCBot/ { split($3, s, " "); print s[1] }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn
```

The command matches each name as it is written, case and all, once per line. A User-Agent is a claim, so check the addresses before you trust a count.

Sources: [nginx: ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html) and [Our agent list (Storewalk)](https://storewalk.ai/checks/DISC-001).

Run it over a week of logs. Crawlers come in bursts, and a single quiet hour says very little. If your store sits behind Cloudflare or another CDN, count there as well: your server only sees the requests the CDN passes on, so a page served from the CDN's cache never reaches your log.

## What should you look for in the numbers?

Start with the status codes. A run of 403 or 429 answers to search crawlers or assistants means a firewall rule or a rate limit is turning them away, and those are the agents that can show your products to shoppers. The last line of the example log is a 403 for ClaudeBot at `/cart`, a training crawler at a page you may well have meant to keep private, so not every refusal is a mistake. The [bot protection guide](https://storewalk.ai/guides/bot-protection-without-blocking-ai-agents) shows how to let the right ones through.

Then split the assistants from the crawlers. An assistant line is a fetch that a person asked for: OpenAI says of ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply", and Anthropic describes Claude-User's visits as "user-initiated requests". A rise in those lines is people asking assistants about your products, which in our view is worth more to a shop than a crawler's routine pass.

Last, look at the paths. Product and collection pages, `robots.txt`, and any discovery file you publish, such as `/.well-known/ucp` or `/llms.txt`, are what agents come for. If an assistant reads your product pages but your policy pages never appear, it may be answering delivery questions without them.

## How do you see people who arrive from ChatGPT and other assistants?

Here GA4 is useful, because the visitor is a person with a browser. GA4 has an AI Assistant channel: "The medium is set to "ai-assistant" and the campaign is set to "(ai-assistant)" if the referrer matches a list of AI Assistants." Google does not publish that list on the page. In AI Crawl Control, referrals from AI platforms are a paid-plan view. In your logs, the referrer field shows the assistant's site when its app sends one, as the second line of the example log does.

Keep the two numbers apart: a crawler reading your product page and a shopper clicking a link in an answer are different events.

## What can our scan tell you about your traffic?

Our scan reads your store from the outside and never sees your logs, so our [MEAS-001](https://storewalk.ai/checks/MEAS-001) check, "Analytics distinguishes AI agents from humans", is currently reported as not measured. What the scan does read is the other side: whether [your robots.txt](https://storewalk.ai/checks/DISC-001) lets these agents in, and whether [a CAPTCHA on your checkout](https://storewalk.ai/checks/TRX-002) stands in their way. The [robots.txt guide](https://storewalk.ai/guides/robots-txt-for-ai-agents) and the [bot protection guide](https://storewalk.ai/guides/bot-protection-without-blocking-ai-agents) cover those.

## Frequently asked questions

### Does Google Analytics track AI crawlers?

Not in our reading. GA4 counts what its tag reports, the tag is a script, and a crawler that does not run the page's scripts sends nothing. GA4 also excludes known bots automatically, and Google says "you cannot disable known bot traffic exclusion".

### Is Cloudflare AI Crawl Control free?

It is "Available on all plans". Some views are not: referral counts are marked "(paid plans only)".

### How do I find GPTBot in my server logs?

Search the User-Agent field for `GPTBot`, or paste a few lines into the classifier above. Then check the address against OpenAI's published list, since anyone can send that name.

### Can I see people who came from ChatGPT?

Yes, in GA4's AI Assistant channel when the referrer matches Google's list, and in your logs when the assistant's app sends a referrer.

### Does Storewalk measure my AI traffic?

No. Our scan cannot see your logs, so MEAS-001 is reported as not measured. Our [agent readiness scanner](https://storewalk.ai/) checks whether agents can get in and find what they need.

The sources below were last checked on the date shown.

## Sources

- [Cloudflare: AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
- [Cloudflare: Analyze AI traffic](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/)
- [Cloudflare: AI Crawl Control directives](https://developers.cloudflare.com/ai-crawl-control/features/track-robots-txt/)
- [Cloudflare: AI Crawl Control bot reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/)
- [Cloudflare Web Analytics: About](https://developers.cloudflare.com/web-analytics/about/)
- [Google Analytics: Known bot-traffic exclusion](https://support.google.com/analytics/answer/9888366)
- [Google Analytics: Default channel group](https://support.google.com/analytics/answer/9756891)
- [Google tag platform: gtag.js](https://developers.google.com/tag-platform/gtagjs)
- [nginx: ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html)
- [Apache HTTP Server: mod_log_config](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Last checked 4 Nov 2026