Minifetch: Search, Scrape, Extract & SEO Audit web pages
Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as given — does not follow redirects. Supports GET & POST
Offering
- Provider
- minifetch.com
- Method
- GET
- Offering type
- Api Endpoint
- Transport type
- http
- Category
- Infrastructure
- Delivery
- Not provided
- Pricing model
- Not provided
- x402 version
- 2
Endpoint context
- URL
https://minifetch.com/api/v1/x402/preflight/url-check- MIME type
- application/json
- First seen
- 2026-10-16T01:50:06.715Z
- Last seen
- Tags
- seo, robots-txt, preflight, crawlability, indexability
Payment pricing
x402 payment terms are the authoritative price for calling this endpoint.
- Payment option 1
- 0.001 USDC
Atomic price: 1000 · Network: eip155:8453 · Asset: USDC · Scheme: exact - Payment option 2
- 0.001 USDC
Atomic price: 1000 · Network: solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp · Asset: USDC · Scheme: exact
Raw x402 terms
[
{
"scheme": "exact",
"network": "eip155:8453",
"asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
"amount": "1000",
"payTo": "0xffA9155A0e4B3d8d42351Da226cC1feEe449686f"
},
{
"scheme": "exact",
"network": "solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
"asset": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
"amount": "1000",
"payTo": "46vMcwuC4sK11sB3gkLhyA7J7GEwfkhn5rFyDtihBwqe"
}
]Observed health
- Status
- Healthy
- Latest latency
- 220 ms
- Average latency
- 558 ms
- HTTP status
- 402
- Last checked
- Reachability
- 100% over 14 checks
Latency is measured to the first response headers. A reachable result does not verify response accuracy or merchant identity.
Input schema
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "http"
},
"method": {
"type": "string",
"enum": [
"GET"
]
},
"queryParams": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The target URL to check against robots.txt"
},
"fresh": {
"type": "boolean",
"description": "Optional. If true, bypasses the 24h robots.txt cache and re-fetches robots.txt for this check. Use after editing your robots.txt or crawl-delay (or adding/removing a Minifetch block) to confirm the change took effect."
}
},
"required": [
"url"
]
}
},
"required": [
"type",
"method"
],
"additionalProperties": false
}Output schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"method": {
"enum": [
"GET"
],
"type": "string"
},
"queryParams": {
"properties": {
"fresh": {
"description": "Optional. If true, bypasses the 24h robots.txt cache and re-fetches robots.txt for this check. Use after editing your robots.txt or crawl-delay (or adding/removing a Minifetch block) to confirm the change took effect.",
"type": "boolean"
},
"url": {
"description": "The target URL to check against robots.txt",
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"results": {
"items": {
"properties": {
"data": {
"properties": {
"allowed": {
"description": "Whether the minifetch user agent may fetch this URL per robots.txt",
"type": "boolean"
},
"crawlDelay": {
"description": "Crawl-delay (seconds) for the minifetch UA; omitted when blocked or the domain is invalid",
"type": "number"
},
"message": {
"description": "Reason for allowed true or false verdict",
"type": "string"
},
"minifetchCache": {
"description": "robots.txt cache state (~24h TTL; pass fresh=true to bypass)",
"properties": {
"cachedAt": {
"type": "string"
},
"expiresAt": {
"type": "string"
},
"hit": {
"type": "boolean"
}
},
"type": "object"
},
"proxy": {
"description": "Fetch provenance — always minifetch UA here",
"properties": {
"proxyType": {
"enum": [
"minifetch"
],
"type": "string"
},
"userAgent": {
"type": "string"
}
},
"type": "object"
},
"url": {
"description": "The URL checked, exactly as given — redirects are not followed",
"type": "string"
}
},
"required": [
"url",
"allowed",
"message",
"proxy"
],
"type": "object"
}
},
"type": "object"
},
"type": "array"
},
"success": {
"type": "boolean"
},
"tip": {
"description": "Present only when allowed=false: how to retry via the external proxy routes, and how site owners can unblock the minifetch user agent",
"properties": {
"message": {
"type": "string"
},
"url": {
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}