Skip to content

DISC-003Find

Sitemap with product URLs

Checks that you publish an XML sitemap that lists product pages and says when they last changed.

What we check

We look for your XML sitemap and read it for two things: URLs that look like individual product pages, and <lastmod> dates. A URL counts as a product page when its path contains a segment such as /products/, /product/, /p/, /dp/, /item/, /produkt/ or /artikel/ followed by another segment that is not a listing word (for example page, sort, filter or category). Sitemap index files are followed to the sitemaps they list.

Why it matters

A sitemap lists the pages you publish for crawlers, including those AI assistants use to build their indexes. Product URLs in it let an agent find products your navigation does not link prominently, and <lastmod> dates tell it which pages changed, so the prices and stock it reports are more likely to be current.

How we check

We read the Sitemap: lines in robots.txt; if there are none, we try /sitemap.xml and /sitemap_index.xml. We fetch at most five sitemap files, index and child files combined. FAIL: no valid sitemap, or no product URLs in the sitemaps read. WARN: product URLs but no <lastmod> on any entry read. PASS: product URLs and at least one <lastmod>. SKIPPED: every sitemap request was refused, or nothing was found and bot management (such as a Cloudflare challenge) answered a robots.txt or sitemap request, in this check or during the crawl.

How to fix it

  1. Turn on your platform's XML sitemap. Shopify generates /sitemap.xml and lists it in robots.txt; on WordPress/WooCommerce, WordPress or your SEO plugin generates one, so check it includes products. A custom-built shop needs a sitemap or sitemap index covering every product page.
  2. Add a "Sitemap:" line to robots.txt pointing at it.
  3. Include <lastmod> on product entries and update it when price, stock or content changes.
  4. Keep product URLs on a recognisable product path (such as /products/ or /product/), or they are not counted.
  5. If SKIPPED because of bot management, let requests for robots.txt and sitemap files through.

What good looks like

robots.txt:
Sitemap: https://www.example-shop.com/sitemap.xml

sitemap.xml:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example-shop.com/products/linen-shirt</loc>
    <lastmod>2026-09-20</lastmod>
  </url>
  <url>
    <loc>https://www.example-shop.com/products/linen-duvet-cover</loc>
    <lastmod>2026-09-18</lastmod>
  </url>
</urlset>

Sources