Crawl Pre-Flight

crawl-preflight.postnov01.workers.dev · 1 paid endpoint · listed 2026-10-27

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

downlast check 2 days ago

0%valid x402 answers, 72 h

$0.02USDC received on Base, 7 days

1distinct payers, 7 days

2.1score (0-100)

Paid endpoints

Method	Endpoint	Price	Networks
GET	crawl-preflight.postnov01.workers.dev/crawl-check/:domain Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI c	$0.02	Polygon, Arbitrum, Base

Latest checks

When	Endpoint	HTTP	ms	Result
2 days ago	crawl-preflight.postnov01.workers.dev/crawl-check/:domain	400	276	HTTP 400 instead of 402
8 days ago	crawl-preflight.postnov01.workers.dev/crawl-check/:domain	400	393	HTTP 400 instead of 402

Income on Base (7 days)

Day		Transfers	USDC
2026-11-02		1	$0.02

Is this your API?

Add the live badge to your docs:

<a href="https://logicnodes.io/observatory/p/crawl-preflight.postnov01.workers.dev"><img src="https://logicnodes.io/observatory/badge/crawl-preflight.postnov01.workers.dev.svg" alt="x402 status"></a>

Claim it for free with x402_provider_claim (sign with your payout address), and get an alert within minutes when it stops answering with x402_monitor_create ($5 for 30 days). Both work over MCP (https://logicnodes.io/mcp/v2) or POST https://logicnodes.io/v1/tools/<tool>.

Income is all USDC received by the provider's payout address on Base, whatever the source. Score: 0-100: 50 x share of checks in the last 72 h that returned valid x402 payment terms + 30 x min(1, log10(1 + USDC received in 7 days) / 4) + 20 x min(1, log10(1 + distinct payers in 7 days) / 3). A provider never checked scores on income only; income is all USDC received by its payout address on Base, whatever the source. JSON: /v1/observatory/providers/crawl-preflight.postnov01.workers.dev