WebberSites x402 Data API · SEO Robots Check

by WebberSites x402 Data API

GETBase

$0.001

per call · USD Coin on Base

robots.txt + llms.txt checker: crawler access verdicts for major search AND AI bots (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended and more), declared sitemaps, syntax warnings, and llms.txt / llms-full.txt presence with structure summary. ?url=any page on the site

Endpoint

GET https://api.webbersites.com/api/seo/robots-check

3

Calls / 30d

1

Unique payers / 30d

Aug 21

Last called

exact

Payment scheme

Call this service

TypeScript · @x402/fetch
import { wrapFetchWithPayment } from "@x402/fetch";
import { privateKeyToAccount } from "viem/accounts";

const account = privateKeyToAccount(process.env.PRIVATE_KEY);
const fetchWithPay = wrapFetchWithPayment(fetch, account);

const res = await fetchWithPay("https://api.webbersites.com/api/seo/robots-check", {
  method: "GET",
});
const data = await res.json();
cURL
curl -X GET \
  "https://api.webbersites.com/api/seo/robots-check?url=https%3A%2F%2Fexample.com" \
  -H "X-PAYMENT: <signed x402 payment>"

Example response

{
  "llms_txt": {
    "found": false
  },
  "origin": "https://example.com",
  "robots": {
    "bots": {
      "GPTBot": {
        "kind": "ai",
        "root_blocked": true
      },
      "Googlebot": {
        "kind": "search",
        "root_blocked": false
      }
    },
    "found": true,
    "sitemaps_declared": [
      "https://example.com/sitemap.xml"
    ]
  }
}

Payment details

Pay to0xdd5fcea81ca6f1eb39fed5b973de794293b81ba6
AssetUSD Coin · 0x833589fcd6edb6e08f4c7c32d4f71b54bda02913
NetworksBase
Schemesexact

Is this your API?

Pin it to the top of Developer Tools and the homepage with a featured placement.

Get featured →

More from WebberSites x402 Data API & similar services

WebberSites x402 Data API · Messages

api.webbersites.com

ANTHROPIC-COMPATIBLE /v1/messages — Claude Haiku 4.5 through the exact Anthropic API dialect: same request/response shape, no Anthropic account, the x402 payment IS the auth. Point an Anthropic SDK's base_url at https://api.webbersites.com and pay per call. Caps: 16,000 input chars, 1,024 output tokens (max_tokens clamped), text-only, no streaming.

POSTBaseAI & Inference
$0.01 / call55 calls / 30d

WebberSites x402 Data API · Scrape

api.webbersites.com

Fetch any public web page and return clean, readable Markdown with navigation, ads, and boilerplate stripped. For agents that need article text or documentation as Markdown.

GETBaseSearch & Web
$0.001 / call27 calls / 30d

WebberSites x402 Data API · SEO Metadata

api.webbersites.com

Raw metadata extractor: the complete, unopinionated head inventory of a page — title, charset, lang, canonical, all meta tags grouped by family (OpenGraph, Twitter, Dublin Core, named, http-equiv, itemprop), every link relation, and JSON-LD returned as parsed objects. For agents doing their own processing (head-check audits the same data; this just dumps it). ?url=

GETBaseDeveloper Tools
$0.001 / call14 calls / 30d