Api Endpoint

robots.txt crawl permission check

Checks whether a given user agent may fetch a given path on a host, per the site's robots.txt (specific-agent groups, wildcard rules, Allow/Disallow longest-match). Also returns crawl-delay, sitemap URLs and whether the site publishes llms.txt or ai.txt. For scraping and research agents that want to crawl politely and prove compliance.

Open interactive test console

Offering

Provider: relay402.georgespring.workers.dev
Method: GET
Offering type: Api Endpoint
Transport type: http
Category: Web Search Research
Delivery: Not provided
Pricing model: Not provided
x402 version: 2

Endpoint context

URL: https://relay402.georgespring.workers.dev/api/robots-check
MIME type: application/json
First seen: 2026-10-16T02:01:07.550Z
Last seen: 2026-11-01T23:52:28.761Z
Tags: scraping, compliance, robots, crawling, llms-txt

Website

Payment pricing

x402 payment terms are the authoritative price for calling this endpoint.

Payment option 1: 0.001 USDC
Atomic price: 1000 · Network: eip155:8453 · Asset: USDC · Scheme: exact

Raw x402 terms

[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "1000",
    "payTo": "0x73fc43d426a89577c7b58f5fee50ec95d4396079"
  }
]

Observed health

Status: Healthy
Latest latency: 1293 ms
Average latency: 1195 ms
HTTP status: 402
Last checked: 2026-11-01T23:52:28.756Z
Reachability: 100% over 13 checks

Latency is measured to the first response headers. A reachable result does not verify response accuracy or merchant identity.

Input schema

{
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "http"
    },
    "method": {
      "type": "string",
      "enum": [
        "GET"
      ]
    },
    "queryParams": {
      "type": "object",
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "properties": {
        "host": {
          "type": "string",
          "pattern": "^[a-z0-9]([a-z0-9.-]{2,251})[a-z0-9]$"
        },
        "path": {
          "type": "string",
          "maxLength": 500
        },
        "user_agent": {
          "type": "string",
          "maxLength": 100
        }
      },
      "required": [
        "host"
      ],
      "additionalProperties": false
    }
  },
  "required": [
    "type",
    "method"
  ],
  "additionalProperties": false
}

Output schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "input": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "const": "http"
        },
        "method": {
          "type": "string",
          "enum": [
            "GET"
          ]
        },
        "queryParams": {
          "type": "object",
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "properties": {
            "host": {
              "type": "string",
              "pattern": "^[a-z0-9]([a-z0-9.-]{2,251})[a-z0-9]$"
            },
            "path": {
              "type": "string",
              "maxLength": 500
            },
            "user_agent": {
              "type": "string",
              "maxLength": 100
            }
          },
          "required": [
            "host"
          ],
          "additionalProperties": false
        }
      },
      "required": [
        "type",
        "method"
      ],
      "additionalProperties": false
    }
  },
  "required": [
    "input"
  ]
}