Index onlineMerchant portal
Api Endpoint

ah-ai-crawlers

Given a website domain, reports which AI crawlers may fetch its home page under its robots.txt (RFC 9309): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more, each allowed or blocked, plus Googlebot, listed sitemaps, whether /llms.txt is published and page-level noindex/nosnippet, with fixes. Static check, no browser. Not charged if robots.txt cannot be read (no response, or a bot block).

Open interactive test console

Offering

Provider
x402.agentharbor.agency
Method
GET
Offering type
Api Endpoint
Transport type
http
Category
Web Search Research
Delivery
Not provided
Pricing model
Not provided
x402 version
2

Endpoint context

URL
https://x402.agentharbor.agency/ai-crawlers
MIME type
Not provided
First seen
2026-10-16T01:50:24.663Z
Last seen
Tags
ai-crawlers, robots-txt, llms-txt, gptbot, seo

Website

Payment pricing

x402 payment terms are the authoritative price for calling this endpoint.

Payment option 1
0.01 USDC
Atomic price: 10000 · Network: eip155:8453 · Asset: USDC · Scheme: exact
Raw x402 terms
[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "10000",
    "payTo": "0x840C829530DF7B4CCF700Fa3A0060cF2C80b014e"
  }
]

Observed health

Status
Healthy
Latest latency
286 ms
Average latency
512 ms
HTTP status
402
Last checked
Reachability
100% over 14 checks

Latency is measured to the first response headers. A reachable result does not verify response accuracy or merchant identity.

Input schema

{
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "http"
    },
    "method": {
      "type": "string",
      "enum": [
        "GET",
        "HEAD",
        "DELETE"
      ]
    },
    "queryParams": {
      "type": "object",
      "properties": {
        "domain": {
          "type": "string",
          "description": "Bare public hostname, e.g. example.com",
          "pattern": "^[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+$",
          "maxLength": 253
        }
      },
      "required": [
        "domain"
      ],
      "additionalProperties": false
    }
  },
  "required": [
    "type",
    "method"
  ],
  "additionalProperties": false
}

Output schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "input": {
      "additionalProperties": false,
      "properties": {
        "method": {
          "enum": [
            "GET",
            "HEAD",
            "DELETE"
          ],
          "type": "string"
        },
        "queryParams": {
          "additionalProperties": false,
          "properties": {
            "domain": {
              "description": "Bare public hostname, e.g. example.com",
              "maxLength": 253,
              "pattern": "^[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+$",
              "type": "string"
            }
          },
          "required": [
            "domain"
          ],
          "type": "object"
        },
        "type": {
          "const": "http",
          "type": "string"
        }
      },
      "required": [
        "type",
        "method"
      ],
      "type": "object"
    },
    "output": {
      "properties": {
        "example": {
          "type": "object"
        },
        "type": {
          "type": "string"
        }
      },
      "required": [
        "type"
      ],
      "type": "object"
    }
  },
  "required": [
    "input"
  ],
  "type": "object"
}