Skip to content

SAFE-103Beside the path

Search/catalog rate limiting

Checks whether your search endpoint ever rate-limits or challenges a rapid burst of automated queries, or serves them all unthrottled.

What we check

An active safety probe, run only in the owner-requested full safety test after you verify the domain. We send 60 GET requests to your search URL over about a minute, each with a different randomised query (so one cached answer cannot cover the whole burst), using our own crawler user-agent. We measure whether the shop ever slows, challenges or refuses the burst.

Why it matters

Search is the cheapest endpoint to abuse: an unthrottled search lets anyone enumerate your catalogue, mine prices, or exhaust origin resources. Rate-limiting search is a basic defence that still lets a genuine shopping agent make the handful of queries it actually needs.

How we check

We stop the burst as soon as we see a throttle, and we never read our own budget or timeout as a merchant defence.

PASS: at least one request in the burst was challenged, rate-limited (for example HTTP 429) or slowed. FAIL (HIGH): the shop served the rapid requests with no rate limiting at all. SKIPPED: no search URL pattern was discovered to burst, or the endpoint answered none of the requests. Read-only GETs against the search URL only.

How to fix it

  1. Rate-limit your search endpoint per client (IP and/or session), for example a few requests per second with a small burst allowance.
  2. Return HTTP 429 with a Retry-After header when the limit is exceeded, rather than silently serving every request.
  3. Set the limit high enough that a shopper or a single agent is never affected, and low enough that a 60-requests-a-minute scrape is throttled.

What good looks like

GET /search?q=shoe-a1b2c3      (request 12 of a rapid burst)

HTTP/1.1 429 Too Many Requests
Retry-After: 30

The first handful of queries succeed; the burst is throttled before it can enumerate the catalogue.

Sources