Pay-per-crawl / content access policy

What we check

This informational probe runs in the owner-requested safety test. Its weight is 0, so it appears in the report but never moves the Safe score. We make one read-only GET to a content path and look for HTTP 402 Payment Required or a crawl-pricing header (for example crawler-price or x-pay-per-crawl). Pay-per-crawl is an emerging convention and the exact header names are not yet settled, so we treat 402 as the one unambiguous signal and the header names as best-effort.

Why it matters

Pay-per-crawl lets a site charge automated clients for access instead of blocking them outright, which admits agents on commercial terms. Whether you advertise it says something about how you want agents to reach your content. It is not a safety pass or fail, so it carries no score.

How we check

INFO (recorded as a passing, informational row): the content path returned HTTP 402 or carried a crawl-pricing header. Otherwise SKIPPED (NO_PAY_PER_CRAWL_SIGNAL). Either way the probe is unscored. The probe sends one read-only GET. Because the header names are unconfirmed against the external spec, a wrong name can under-report but can never mis-score.

How to fix it

1. This check is informational; no action is required.
2. If you want to charge agents for access rather than block them, publish a pay-per-crawl policy (for example an HTTP 402 with pricing, or your CDN's pay-per-crawl feature) instead of a hard block.

What good looks like

GET /products/anvil

HTTP/1.1 402 Payment Required
crawler-price: USD 0.002 per request

or an equivalent crawl-pricing header advertised by your CDN.

Sources

• RFC 9110 §15.5.3: HTTP 402 Payment Required
• Cloudflare: Introducing pay per crawl