AI Bots Inspector

Checks the site's robots.txt, tells you which AI bots (GPTBot, ClaudeBot, PerplexityBot and others) are blocked and suggests how to fix it.

AI Bots Inspector

What it does

Fetches the robots.txt of the given URL, compares it against the list of the main AI bots and flags the blocked ones, telling apart two kinds. Answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) read pages to cite them in AI answers: blocking them costs visibility. Training bots (GPTBot, ClaudeBot, Google-Extended) only collect data for future models: blocking them is a legitimate choice, for example on pages with proprietary data. The suggested allowlist re-opens only the answer bots.

When to use it

  • As the first technical check: without AI-bot accessibility, no other GEO work makes sense.
  • To verify that recent robots.txt changes didn't accidentally block a new crawler.
  • In an agency setting, during client onboarding, before promising GEO results.

Required inputs

  • url (required in the body): the URL of the site to check.
  • store_id (optional): for per-store metering.

Example output

{
  "url": "https://example.com",
  "blocked_bots": ["GPTBot", "PerplexityBot"],
  "allowed_bots": ["ClaudeBot", "Google-Extended"],
  "suggested_allowlist": "User-agent: PerplexityBot\nAllow: /\n"
}