Index onlineMerchant portal
Api Endpoint

Robots.txt Analyzer

Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.

Open interactive test console

Offering

Provider
netintel.dev
Method
GET
Offering type
Api Endpoint
Transport type
http
Category
Web Search Research
Delivery
Not provided
Pricing model
Not provided
x402 version
2

Endpoint context

URL
https://netintel.dev/robots-txt/analyze
MIME type
application/json
First seen
2026-10-16T01:50:14.534Z
Last seen
Tags
robots.txt, crawl rules, seo, web intelligence, scraping

Website

Payment pricing

x402 payment terms are the authoritative price for calling this endpoint.

Payment option 1
0.01 USDC
Atomic price: 10000 · Network: eip155:8453 · Asset: USDC · Scheme: exact
Payment option 2
0.01 USDC
Atomic price: 10000 · Network: solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp · Asset: USDC · Scheme: exact
Raw x402 terms
[
  {
    "scheme": "exact",
    "network": "eip155:8453",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "10000",
    "payTo": "0xdaDc335482AD545296Fd7b28518A251fFCbEb9Df"
  },
  {
    "scheme": "exact",
    "network": "solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
    "asset": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
    "amount": "10000",
    "payTo": "8FVeybWnB4dYrxTuPK7WzFZNAiEcFVaDLFnKpRwVGr5s"
  }
]

Observed health

Status
Healthy
Latest latency
104 ms
Average latency
298 ms
HTTP status
402
Last checked
Reachability
100% over 13 checks

Latency is measured to the first response headers. A reachable result does not verify response accuracy or merchant identity.

Input schema

{
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "http"
    },
    "method": {
      "type": "string",
      "enum": [
        "GET"
      ]
    },
    "queryParams": {
      "type": "object",
      "properties": {
        "domain": {
          "type": "string",
          "description": "Domain name to fetch robots.txt from (e.g. example.com)"
        },
        "path": {
          "type": "string",
          "description": "URL path to check permission for (e.g. /api/data)"
        },
        "user_agent": {
          "type": "string",
          "description": "Bot name to check rules for (default: *)"
        }
      },
      "required": [
        "domain"
      ]
    }
  },
  "required": [
    "type",
    "method"
  ],
  "additionalProperties": false
}

Output schema

{
  "type": "object",
  "properties": {
    "type": {
      "type": "string"
    },
    "example": {
      "type": "object",
      "properties": {
        "domain": {
          "type": "string"
        },
        "robots_url": {
          "type": "string"
        },
        "found": {
          "type": "boolean"
        },
        "status_code": {
          "type": "number"
        },
        "raw_content_preview": {
          "type": "string"
        },
        "sitemaps": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "rules": {
          "type": "array",
          "items": {
            "type": "object"
          }
        },
        "path_check": {
          "type": "object"
        },
        "score": {
          "type": "number"
        },
        "grade": {
          "type": "string"
        },
        "findings": {
          "type": "array",
          "items": {
            "type": "object"
          }
        }
      }
    }
  },
  "required": [
    "type"
  ]
}