reader.scrapemole.workers.dev x402 offerings

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the URLs of the first child sitemaps up to the limit. Gzipped sitemaps supported.

Use when you need the outgoing links of one page (crawling, SEO audits, finding a contact/pricing/docs page, link building): fetches the URL live (respects robots.txt) and returns every http(s) link resolved to an absolute URL, deduplicated, with anchor text (or image alt), internal vs external, rel nofollow/sponsored/ugc, target=_blank and occurrence count, plus internal/external totals. Filter with type=internal|external.

Use when you need the latest posts or headlines from a site, blog, newsroom, podcast or subreddit: give a feed URL or just the site URL (the RSS/Atom feed is auto-discovered from the page) and get normalized items with title, link, publish date (ISO 8601), author, summary, image and categories, newest first as the feed lists them.

Use when you need a link preview or page metadata: fetches the URL live and returns title, description, site name, canonical URL, language, Open Graph and Twitter card tags, preview image, favicon, author and publish date, RSS/Atom feeds, robots meta and JSON-LD (schema.org) types, after following redirects.

Use when you need to read one public web page: fetches the URL live and returns its main readable content as clean Markdown (or plain text), with navigation, ads, cookie banners and boilerplate removed, plus title, byline, publish date, language and word count. Respects robots.txt. Server-rendered HTML only (no JavaScript execution).