Our crawler
About Storewalk’s crawler
Storewalk checks how well an online store works for AI shopping agents. A person asks us to scan a public store address. We then read a small number of its pages, including the cart and checkout, the way an agent would find them. We only read: nothing is added to a cart and no form is filled in.
If you found this address in your server logs, our crawler put it there: every request we make names this page. We tell merchants which AI agent crawlers their own robots.txt blocks, so you should be able to see what ours does and how to block it.
What the crawler does
Step by step, in the order the running code does it.
robots.txt
robots.txt
Every scan starts here, before any other page, and the crawler follows its rules for the user agent shown below. If it blocks us, we fetch your home page once so the report can say “blocked” instead of “saw nothing”, and then stop.
Request and rate limits
Request and rate limits
Each scan has a fixed request budget set in the crawler’s code, and the crawler refuses any request past it before it reaches the network. No host gets more than 2 requests a second. A page opened in a real browser counts against both limits, just like a plain fetch, and loads its own scripts and styles as any browser would. The crawler may also read a few pages on your shop’s own subdomains, such as an order-status or returns page. It reads each subdomain’s robots.txt first.
What it never does
What it never does
A scan reads pages, and sends at most one read-only request to an MCP server your UCP profile declares. It never enters payment details, creates accounts, solves CAPTCHAs, uploads files or calls a tool.
Your MCP server and UCP links
Your MCP server and UCP links
If your /.well-known/ucp declares an MCP server, we send it at most one request per scan: a JSON-RPC tools/list call, which asks for the list of tools and changes nothing. It goes only to the endpoint your own profile names, over HTTPS, even when that endpoint is on another domain. Your report shows the tools it listed and how long it took. To check that the specification and schema links in your profile work, we fetch them only from the specification sites we allow (ucp.dev, shopify.dev and pay.google.com), never from your shop, and we reuse a working or broken answer for a day.
Cart and checkout
Cart and checkout
The crawler reads cart and checkout pages as they load. It adds nothing to a cart and fills in no form, so it never starts an order. On up to three product pages and the cart, the browser also loads images and fonts, so the report can show those pages close to how a person sees them.
How long we keep it
How long we keep it
We keep screenshots and captured page data for 30 days. A storage lifecycle rule then deletes them automatically.
The string in your log
Storewalk/1.0 (+https://storewalk.ai/bot)
Every request carries this exact string, whether the crawler fetches a page directly or renders it in a browser.
User agent
Every request we send identifies itself as Storewalk/1.0 (+https://storewalk.ai/bot).
The crawler uses no other name, on plain requests or on the pages it renders. We do not rotate user agents, and we do not disguise the crawler as an ordinary browser to get past a rule you set for it.
How to block it
Add this to your robots.txt. The crawler will then fetch only your homepage, once, to explain the block in the report.
When someone asks us for an llms.txt draft of a site, the crawler also fetches its robots.txt, its sitemaps and the pages the draft will list, about 100 at most, four requests a second at most. It obeys robots.txt there too, page by page: with the rule below it reads robots.txt and stops.
User-agent: Storewalk Disallow: /
You do not need to ask us or give a reason. Because robots.txt is read at the start of every scan, a rule you add today applies to the next scan of your store.
How to let it in
Some stores put bot protection in front of every page, and it answers our crawler with a challenge or a refusal. We do not try to get past it. The report then covers only what sits outside the wall: robots.txt, llms.txt and the discovery files under /.well-known. An AI shopping agent most likely meets the same wall.
To get a full report, add a rule in your firewall or bot-protection settings that lets the user agent above through, then scan again. Keep the rule to our crawler’s name only: a user agent is easy to copy, so a broad rule lets more in than you meant to. You can remove it once the scan is done.
This deployment lists no contact address. The robots.txt rule above works without one, and no one has to approve it.
In short
first
robots.txt
Read before any other page on every scan, and obeyed for the user agent above.
2
requests a second, per host
Each scan has a fixed request budget set in the crawler’s code, and the crawler refuses any request past it before it reaches the network. A rendered page counts like a plain fetch and loads its own scripts and styles, as in any browser.
never
pays or signs in
It never enters payment details, creates accounts, solves CAPTCHAs, uploads files or calls a tool on your MCP server.
none
orders started
The crawler reads cart and checkout pages without adding to the cart or filling in a form. Up to three product pages and the cart also load images and fonts, so the report shows them close to how a person sees them.
30
days, then deleted
A storage rule deletes screenshots and captured page data automatically.
one
user agent, never disguised
Plain requests and rendered pages carry the same name, and we do not rotate it.