AI Bots Inspector

What it does

Fetches the robots.txt of the given URL, compares it against the list of the main AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) and flags the blocked ones. Also returns a suggested correct allowlist.

When to use it

• As the first technical check: without AI-bot accessibility, no other GEO work makes sense.
• To verify that recent robots.txt changes didn't accidentally block a new crawler.
• In an agency setting, during client onboarding, before promising GEO results.

Required inputs

• url (required in the body): the URL of the site to check.
• store_id (optional): for per-store metering.

Example output

{
  "url": "https://example.com",
  "blocked_bots": ["GPTBot", "PerplexityBot"],
  "allowed_bots": ["ClaudeBot", "Google-Extended"],
  "suggested_allowlist": "User-agent: GPTBot\nAllow: /\n..."
}