Our crawler
About Storewalk’s crawler
Storewalk checks how well an online store works for AI shopping agents. A person asks us to scan a public store address. We then read a small number of its pages, including the cart and checkout, the way an agent would find them. We only read: nothing is added to a cart and no form is filled in.
If you found this address in your server logs, our crawler put it there: every request we make names this page. Part of what we report to a merchant is which AI agent crawlers their own robots.txt blocks, so an account of what our crawler does, and how to refuse it, is the least we owe you.
What the crawler does
In the order it does it. Every line here is a fact about the code that runs, and the page below repeats all five in plain markup.
robots.txt
robots.txt
Read first, every scan, before anything else — and respected for the crawler user agent below. If it refuses us we fetch your home page once, so the report can say “blocked” rather than “saw nothing”, and stop.
The budget
The budget
At most 75 pages for the whole scan, no faster than 2 page requests a second per host. A page opened in a real browser is charged against both, exactly like a plain fetch; the scripts and styles that page then loads come with it, the way a browser loads them. It may also read a few pages on your shop’s own subdomains, such as an order-status or returns page, and reads each subdomain’s own robots.txt first.
The four nevers
The four nevers
Never enters payment details, never creates accounts, never solves CAPTCHAs, never uploads files. A scan reads pages and does nothing else.
The checkout
The checkout
Cart and checkout pages are read as they load. Nothing is added to a cart and no form is filled in, so no order is ever started. On up to three product pages and the cart that browser also loads the page's images and fonts, so the report can show the page close to how a person sees it.
Then it is deleted
Then it is deleted
Screenshots and captured page data are kept for 30 days and then expire on a storage lifecycle rule, rather than on someone remembering.
The string in your log
Storewalk/1.0 (+https://storewalk.ai/bot)
One name, on plain requests and on the pages we render alike. We do not rotate it and we do not dress the crawler up as an ordinary browser to get past a rule you set for it.
User agent
Our checks identify as Storewalk/1.0 (+https://storewalk.ai/bot).
That string is the only name our crawler uses, on plain requests and on the pages it renders alike. We do not rotate user agents and we do not disguise the crawler as an ordinary browser to get past a rule you set for it.
How to block it
Add this to your robots.txt. The crawler will then fetch only your homepage, once, to explain the block in the report.
User-agent: Storewalk Disallow: /
You do not need to ask us first and you do not need to give a reason. We read robots.txt at the start of every scan, so a rule you add today applies to the next scan of your store.
This deployment publishes no contact address. The rule above is the route that needs nothing from us: it is read at the start of every scan and no one has to agree to it.
In short
first
robots.txt
Read at the start of every scan and respected for the crawler user agent above.
75
pages, at most
Per scan, and never faster than 2 page requests a second, per host. A rendered page loads its own scripts and styles the way a browser does.
never
pays or signs in
No payment details, no accounts, no CAPTCHAs solved, no files uploaded — ever.
none
orders started
Cart and checkout pages are read as they load. Nothing is added to a cart and no form is filled in. Up to three product pages and the cart also load their images and fonts, so the report can show the page close to how a person sees it.
30
days, then deleted
Screenshots and captured page data expire on a storage rule, not on someone remembering.
one
user agent, never disguised
The same name on plain requests and rendered pages. We do not rotate it.