Index onlineMerchant portal
Api Endpoint

aeo-crawlers

robots.txt AI-crawler access matrix: for the 2026 cohort of AI and answer-engine crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and more) reports allowed, blocked or unlisted per bot, whether the default rule blocks everything, declared sitemaps and a score. One network operation.

Open interactive test console

Offering

Provider
payai.agentstools.dev
Method
GET
Offering type
Api Endpoint
Transport type
http
Category
Security
Delivery
Not provided
Pricing model
Not provided
x402 version
2

Endpoint context

URL
https://payai.agentstools.dev/aeo/crawlers
MIME type
application/json
First seen
2026-10-16T02:01:31.612Z
Last seen
Tags
aeo, robots-txt, ai-crawler, gptbot, claudebot, perplexitybot, ai-search, access-control, crawl

Website

Payment pricing

x402 payment terms are the authoritative price for calling this endpoint.

Payment option 1
0.005 USDC
Atomic price: 5000 · Network: eip155:8453 · Asset: USDC · Scheme: exact
Payment option 2
0.005 USDC
Atomic price: 5000 · Network: solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp · Asset: USDC · Scheme: exact
Raw x402 terms
[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "5000",
    "payTo": "0xF22e558a00D91Ee12A1F50C52186FecB8dDFf493"
  },
  {
    "scheme": "exact",
    "network": "solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
    "asset": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
    "amount": "5000",
    "payTo": "AytHnvc6bZx1ALTy7b5Px7hcavoSYszG9aeKvDcZ8R1u"
  }
]

Observed health

Status
Healthy
Latest latency
363 ms
Average latency
998 ms
HTTP status
402
Last checked
Reachability
100% over 11 checks

Latency is measured to the first response headers. A reachable result does not verify response accuracy or merchant identity.

Input schema

{
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "http"
    },
    "method": {
      "type": "string",
      "enum": [
        "GET",
        "HEAD",
        "DELETE"
      ]
    },
    "queryParams": {
      "type": "object",
      "properties": {
        "url": {
          "type": "string",
          "description": "Target URL to audit (e.g. https://example.com)"
        },
        "domain": {
          "type": "string",
          "description": "Alternative input: a bare domain (e.g. example.com)"
        },
        "host": {
          "type": "string",
          "description": "Alternative input: a hostname"
        }
      },
      "required": []
    }
  },
  "required": [
    "type",
    "method"
  ],
  "additionalProperties": false
}

Output schema

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "input"
  ],
  "properties": {
    "input": {
      "type": "object",
      "required": [
        "type",
        "method"
      ],
      "properties": {
        "type": {
          "type": "string",
          "const": "http"
        },
        "method": {
          "enum": [
            "GET",
            "HEAD",
            "DELETE"
          ],
          "type": "string"
        },
        "queryParams": {
          "type": "object",
          "required": [],
          "properties": {
            "url": {
              "type": "string",
              "description": "Target URL to audit (e.g. https://example.com)"
            },
            "host": {
              "type": "string",
              "description": "Alternative input: a hostname"
            },
            "domain": {
              "type": "string",
              "description": "Alternative input: a bare domain (e.g. example.com)"
            }
          }
        }
      },
      "additionalProperties": false
    },
    "output": {
      "type": "object",
      "required": [
        "type"
      ],
      "properties": {
        "type": {
          "type": "string"
        },
        "example": {
          "type": "object"
        }
      }
    }
  }
}