robots.txt & AI crawler rules
Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.
Offering
- Provider
- api.vextorium.com
- Method
- POST
- Offering type
- Api Endpoint
- Transport type
- http
- Category
- Web Search Research
- Delivery
- Not provided
- Pricing model
- Not provided
- x402 version
- 2
Endpoint context
- URL
https://api.vextorium.com/v1/robots-txt- MIME type
- application/json
- First seen
- 2026-10-16T01:51:30.109Z
- Last seen
- Tags
- robots, crawler, ai-bots, scraping, compliance
Payment pricing
x402 payment terms are the authoritative price for calling this endpoint.
- Payment option 1
- 0.003 USDC
Atomic price: 3000 · Network: eip155:8453 · Asset: USDC · Scheme: exact - Payment option 2
- 0.003 USDC
Atomic price: 3000 · Network: solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp · Asset: USDC · Scheme: exact - Payment option 3
- 2310 atomic units
Atomic price: 2310 · Network: xrpl:0 · Asset: XRP · Scheme: exact - Payment option 4
- 0.003 USDC
Atomic price: 3000 · Network: eip155:137 · Asset: USDC · Scheme: exact - Payment option 5
- 0.003 USDC
Atomic price: 3000 · Network: eip155:10 · Asset: USDC · Scheme: exact - Payment option 6
- 3000 atomic units
Atomic price: 3000 · Network: eip155:5042 · Asset: 0x3600000000000000000000000000000000000000 · Scheme: exact
Raw x402 terms
[
{
"scheme": "exact",
"network": "eip155:8453",
"asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
"amount": "3000",
"payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf"
},
{
"scheme": "exact",
"network": "solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
"asset": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
"amount": "3000",
"payTo": "HcjfmZLgWRZzwj4deEiS7pewc4KBYSw39mW5nTyt3pz2"
},
{
"scheme": "exact",
"network": "xrpl:0",
"asset": "XRP",
"amount": "2310",
"payTo": "rEac5rTEr7y81iEyrMTrHCRbnktSE2pNCp"
},
{
"scheme": "exact",
"network": "eip155:137",
"asset": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359",
"amount": "3000",
"payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf"
},
{
"scheme": "exact",
"network": "eip155:10",
"asset": "0x0b2C639c533813f4Aa9D7837CAf62653d097Ff85",
"amount": "3000",
"payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf"
},
{
"scheme": "exact",
"network": "eip155:5042",
"asset": "0x3600000000000000000000000000000000000000",
"amount": "3000",
"payTo": "0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf"
}
]Observed health
- Status
- Healthy
- Latest latency
- 422 ms
- Average latency
- 616 ms
- HTTP status
- 200
- Last checked
- Reachability
- 100% over 11 checks
Latency is measured to the first response headers. A reachable result does not verify response accuracy or merchant identity.
Input schema
{
"additionalProperties": false,
"properties": {
"body": {
"properties": {
"agent": {
"description": "User-agent to test (optional)",
"type": "string"
},
"route": {
"description": "Path to test (default /)",
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"url"
]
},
"bodyType": {
"enum": [
"json",
"form-data",
"text"
],
"type": "string"
},
"method": {
"enum": [
"POST"
],
"type": "string"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method",
"bodyType",
"body"
],
"type": "object"
}Output schema
{
"properties": {
"example": {
"properties": {
"ai_bots": {
"properties": {
"Amazonbot": {
"type": [
"string",
"null"
]
},
"Applebot-Extended": {
"type": [
"string",
"null"
]
},
"Bytespider": {
"type": [
"string",
"null"
]
},
"CCBot": {
"type": [
"string",
"null"
]
},
"ChatGPT-User": {
"type": [
"string",
"null"
]
},
"Claude-SearchBot": {
"type": [
"string",
"null"
]
},
"Claude-User": {
"type": [
"string",
"null"
]
},
"ClaudeBot": {
"type": [
"string",
"null"
]
},
"Diffbot": {
"type": [
"string",
"null"
]
},
"DuckAssistBot": {
"type": [
"string",
"null"
]
},
"GPTBot": {
"type": [
"string",
"null"
]
},
"Google-Extended": {
"type": [
"string",
"null"
]
},
"MistralAI-User": {
"type": [
"string",
"null"
]
},
"OAI-SearchBot": {
"type": [
"string",
"null"
]
},
"Perplexity-User": {
"type": [
"string",
"null"
]
},
"PerplexityBot": {
"type": [
"string",
"null"
]
},
"YouBot": {
"type": [
"string",
"null"
]
},
"anthropic-ai": {
"type": [
"string",
"null"
]
},
"cohere-ai": {
"type": [
"string",
"null"
]
},
"meta-externalagent": {
"type": [
"string",
"null"
]
}
},
"type": [
"object",
"null"
]
},
"blocked_ai_bots": {
"type": [
"number",
"null"
]
},
"groups": {
"type": [
"number",
"null"
]
},
"query": {
"properties": {
"agent": {
"type": [
"string",
"null"
]
},
"allowed": {
"type": [
"boolean",
"null"
]
},
"applied_group": {
"items": {
"type": [
"null",
"string"
]
},
"type": [
"array",
"null"
]
},
"route": {
"type": [
"string",
"null"
]
}
},
"type": [
"object",
"null"
]
},
"robots_txt": {
"type": [
"boolean",
"null"
]
},
"site": {
"type": [
"string",
"null"
]
},
"sitemaps": {
"items": {
"type": [
"null",
"string"
]
},
"type": [
"array",
"null"
]
}
},
"type": [
"object",
"null"
]
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}