Browse indexable web search research x402 offerings, including provider, method, and service details.
3432 indexable offerings in this category.
Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged.
Daily or monthly page views (user agent, all access) for up to 5 articles in one Wikipedia edition over a date range of at most one year, with totals, peak day and the change between the first and second half. mode=top lists the most viewed articles of one day. Data lags by about a day; bots and unknown agents are excluded.
The redirect chain of a URL, hop by hop (at most 10 hops): each URL, its status, the Location it points to, and whether the hop goes from http to https or changes host. Returns the final URL and status, total hops, loop true when a URL repeats, and the Strict-Transport-Security header of the final response.
Searches the App Store (iTunes Search API) by keyword, or looks up one app by its numeric id: name, developer, bundle id, price, rating and rating count, genres, version, release and update dates, size and store link. Country sets the storefront (default us). Ratings are the store's averages for that country. Limit max 50.
A site's brand assets: its icons (favicon, apple-touch-icon and web app manifest icons, largest first), the og:image, theme colour and site name, read from the page and its manifest. Falls back to /favicon.ico when the page declares no icon.
Orders up to 50 documents by relevance to a query with the BGE reranker and returns rank, index and score for each. Use it after a keyword or vector search to pick the best few. Scores compare documents within one call only.
Every link on one page (anchor text, internal or external) and a status check of up to 40 of them: status, final URL after redirects, ok, and the error for network failures. Totals: found, checked, broken, redirected and skipped beyond the 40. scope=all, internal or external. robots.txt is respected.
SEO and sharing metadata of one web page, read from its HTML without JavaScript: final URL and status, title and description with their lengths, canonical, robots meta, lang, hreflang, Open Graph and Twitter card tags, JSON-LD types and objects, favicon, heading outline, word count, and an issues list of plain sentences. robots.txt is respected.
Items of an RSS, Atom or RDF feed, or of the feed a web page advertises: title, link, published date, author and a 300-character summary, newest first (max 50). since keeps items published on or after a date.
All URLs a website lists in its XML sitemaps (from robots.txt, /sitemap.xml or /sitemap_index.xml, following indexes), with lastmod, newest first. Filter by path prefix and changed-since date. For agents that need a site's page list before reading pages.
A summary of a web page (url) or your own text (up to 12,000 characters) as short, medium or bullets, in the language you name, with key points. Llama 3.3 70B. A page is read as in /api/read, robots.txt respected.
Read one web page as clean Markdown (or plain text) for an LLM: the main content without menus, cookie banners and footers, plus title, description, author, published date, language and canonical URL; links on request. robots.txt is respected (a disallowed page is refused and not charged). No JavaScript is run.
Whether a site publishes /llms.txt and /llms-full.txt: the parsed llms.txt (title, summary, sections with their links) and the size of llms-full.txt. Reads the site itself, through the same URL guard and size cap as the page tools.
Fetch and parse any XML sitemap or sitemap index file — returns URLs with their priority, change frequency, and last modified date, follows sitemap indexes one level deep (first 3 child sitemaps), and auto-discovers the sitemap from robots.txt or common paths when given a bare domain — so agents can enumerate site content for crawling, indexing, and SEO analysis.
Fetch and parse a sitemap.xml (or sitemap index): returns up to 500 URLs with lastmod, or the child sitemaps of an index.
List the URLs of a website without reading them: robots.txt, sitemaps and sitemap indexes plus homepage links, up to 1,000 same-site URLs with lastmod and source. A cheap first step to decide which pages to read with web-read-batch or site-crawl. Returns no page content. Params: url (site or domain, required), max_urls (1 to 1000, default 200).
Give this API a link to any website's sitemap.xml file and it will fetch and parse it, returning every URL found inside. If the sitemap links to other sitemaps (called sitemap indexes), it will follow and extract those too. Each URL is returned with any extra information available, such as when the page was last updated and how often it changes. Useful for SEO audits, content inventories, broken link checks, and competitive research.
List the URLs of a site's XML sitemap (found via robots.txt if needed, following sitemap indexes and .gz files) with last-modified dates, optionally filtered by text. Discover a site's pages for crawling or monitoring.
Sitemap reader: every URL a website publishes, from robots.txt and its sitemaps, with last-modified dates. Every URL a website publishes, from its own sitemaps: pass ?url= and we read robots.txt for Sitemap: lines (else /sitemap.xml and /sitemap_index.xml), follow nested sitemap indexes and gzipped files, and return each URL with its lastmod, up to 5,000. Unreadable files are listed with the reason. A site with no sitemap is answered before any charge.
Give this API a link to any website's sitemap.xml file and it will fetch and parse it, returning every URL found inside. If the sitemap links to other sitemaps (called sitemap indexes), it will follo…
Crawl a site from a start URL (same-domain, breadth-first) and return each page as clean Markdown
Extract structured metadata from a page: title, description, OpenGraph, JSON-LD, headings, links
Fetch a PDF by URL and return its text as Markdown-ish plain text, page by page
Enumerate a site's URLs from robots.txt + sitemap.xml (sitemap-index aware) — map a domain before crawling it