Free tool
AI Crawler Access Checker
See whether GPTBot, ClaudeBot, PerplexityBot and 13 other AI crawlers can read your site. Crawler access is the technical floor of AI search visibility: if the bots cannot read you, the engines cannot recommend you.
What to do about a block
Not every block is a mistake. Training bots such as GPTBot and ClaudeBot only feed model training, and blocking them is a legitimate choice for some publishers. Search and assistant bots are different: OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered agents fetch pages to answer live questions. Block those and the engines stop citing you, which costs AI search visibility directly.
If you want the main AI bots in, this robots.txt addition opens the door explicitly:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /Access is only the floor. The free AI visibility check runs the full audit across 20+ bots plus page structure and metadata, and the Honeyb agent fixes what it finds.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI's web crawler. It collects public web content that may be used to train OpenAI's models. It is separate from OAI-SearchBot, which indexes pages for ChatGPT search results, and ChatGPT-User, which fetches a page when a user asks ChatGPT about it. Each responds to its own rules in robots.txt, which is why the checker reports them individually.
Should I block AI crawlers?
It depends on which ones. Blocking training bots like GPTBot or ClaudeBot keeps your content out of future model training and costs you little today. Blocking search and assistant bots is different: they fetch pages to answer live buyer questions, so blocking them removes you from AI answers and cuts your AI search visibility. Most brands that want to be recommended keep the search and assistant bots open.
Why does my robots.txt restrict every bot?
Usually because of boilerplate rules under the wildcard group: paths like /admin or /cart are disallowed for all crawlers equally, and every bot inherits them. That is normal hygiene, not an AI problem. The checker distinguishes these shared exclusions from rules that single out AI bots, and flags a bot as blocked only when it cannot read your actual content.
How do I allow a specific AI bot?
Add a rule group naming that bot's user agent in robots.txt. For example, User-agent: PerplexityBot on one line and Allow: / on the next opens your whole site to Perplexity, regardless of what the wildcard group says, because crawlers follow the most specific group that names them. Repeat the pattern for each bot you want to admit, and keep any private paths disallowed under the wildcard group.
Free to start
Get recommended by AI search models.
Run a free AI search visibility check, then let the Honeyb agent do the work that gets you into the answers.
ChatGPT
Claude
Gemini