AI Bots Inspector
What it does
Fetches the robots.txt of the given URL, compares it against the list of the main AI bots and flags the blocked ones, telling apart two kinds. Answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) read pages to cite them in AI answers: blocking them costs visibility. Training bots (GPTBot, ClaudeBot, Google-Extended) only collect data for future models: blocking them is a legitimate choice, for example on pages with proprietary data. The suggested allowlist re-opens only the answer bots.
When to use it
- As the first technical check: without AI-bot accessibility, no other GEO work makes sense.
- To verify that recent
robots.txtchanges didn't accidentally block a new crawler. - In an agency setting, during client onboarding, before promising GEO results.
Required inputs
url(required in the body): the URL of the site to check.store_id(optional): for per-store metering.
Example output
{
"url": "https://example.com",
"blocked_bots": ["GPTBot", "PerplexityBot"],
"allowed_bots": ["ClaudeBot", "Google-Extended"],
"suggested_allowlist": "User-agent: PerplexityBot\nAllow: /\n"
}