Crawler Access

Make sure AI bots can actually reach you

Before you rewrite pages for AEO, confirm GPTBot, ClaudeBot, PerplexityBot, and other known crawlers are allowed — and that your homepage is not a thin JS shell to them.

robots.txt, per bot

We parse your robots rules for the crawlers that matter for AI search, so a single Disallow does not silently block the engines you are optimizing for.

Fetch as the bot

We request your homepage with each bot’s user-agent and flag responses that look blocked, empty, or rendering-risk for non-JS crawlers.

Honest scope

This is a crawlability check — not full log-based bot analytics or AI-referral attribution. It answers one question well: can known AI crawlers reach your site today?

Also in this module

  • Known AI crawler checklist
  • robots.txt parsing
  • Bot user-agent fetches
  • 0–100 access score

Become the brand AI recommends

See where you stand in AI answers — and what it takes to win.