Skip to content

SAFE-107Beside the path

Pay-per-crawl / content access policy

Informational only: notes whether your site advertises a price for crawling its content, via an HTTP 402 response or a crawl-pricing header. It never affects your score.

What we check

An informational probe (weight 0 — it colours the report but never moves the Safe score), run in the owner-requested safety test. We make one read-only GET to a content path and look for HTTP 402 Payment Required or a crawl-pricing header (for example crawler-price or x-pay-per-crawl). Pay-per-crawl is an emerging convention and the exact header names are not yet settled, so we treat 402 as the one unambiguous signal and the header names as best-effort.

Why it matters

Pay-per-crawl lets a site charge automated clients for access instead of blocking them outright — a way to admit agents on commercial terms. Whether you advertise one is useful context for how you want agents to reach your content; it is not a pass or fail of safety, which is why it carries no score.

How we check

INFO (recorded as a passing, informational row): the content path returned HTTP 402 or carried a crawl-pricing header. Otherwise SKIPPED (NO_PAY_PER_CRAWL_SIGNAL). Either way the probe is unscored. One read-only GET. Because the header names are unconfirmed against the external spec, a wrong name can under-report but can never mis-score.

How to fix it

  1. This is informational — no action is required.
  2. If you want to charge agents for access rather than block them, publish a pay-per-crawl policy (for example an HTTP 402 with pricing, or your CDN’s pay-per-crawl feature) instead of a hard block.

What good looks like

GET /products/anvil

HTTP/1.1 402 Payment Required
crawler-price: USD 0.002 per request

or an equivalent crawl-pricing header advertised by your CDN.

Sources