Can AI actually read your site?
GPTBot, ClaudeBot, PerplexityBot and 13 other AI crawlers, tested three ways: the robots.txt rules, the connection itself, and the live engines fetching your pages through their own infrastructure. If the bots cannot read you, the engines cannot recommend you.
How the check works
Before AI can recommend you,
it has to read you.
01
Permission
robots.txt, evaluated for 16 AI bots: who is allowed, who is told to stay out.
02
Connection
Five crawler identities knock on the door for real: TLS, status codes, page parity. If a WAF answers, we name the vendor and the rule to change.
03
Ground truth
Gemini, Claude and Perplexity fetch your site through their own infrastructure and report back in their own words. Whatever they say overrides the simulation.
Most checkers stop at robots.txt, and browser extensions cannot see edge blocking at all: anything running inside a browser inherits the browser’s TLS fingerprint and gets a 200 from sites that reset real crawlers. This check runs where the crawlers run.
What to do about a block
Not every block is a mistake. Training bots such as GPTBot and ClaudeBot only feed model training, and blocking them is a legitimate choice for some publishers. Search and assistant bots are different: OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered agents fetch pages to answer live questions. Block those and the engines stop citing you, which costs AI search visibility directly.
If you want the main AI bots in, this robots.txt addition opens the door explicitly:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /robots.txt only matters if the crawler can connect. If a WAF or CDN sits in front, also allowlist the crawlers’ published IP ranges: OpenAI, Anthropic, Perplexity and Google each publish theirs, and most bot-protection vendors treat the big AI crawlers as verified bots you can allow with one rule.
Access is only the floor. The free AI visibility check runs the full audit across 20+ bots plus page structure and metadata, and the Honeyb agent fixes what it finds.
The state of AI crawler access
We ran this checker across 39 major sites (news, commerce, SaaS and platforms) on 29 July 2026. 51 percent block at least one AI crawler, and training bots are blocked far more often than search and assistant bots:
Blocked on n of 39 sites measured. The pattern matters: many publishers block training while leaving answer-time bots open, which keeps them citeable in live AI answers. Blocking everything severely limits direct citation and fresh retrieval: an engine may still surface the bare URL from third-party sources, but not your content or your framing.
Want to get recommended by AI?
Check your AI search visibility, then let the Honeyb agent write, fix, and earn what gets you recommended. Free to start.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI's web crawler. It collects public web content that may be used to train OpenAI's models. It is separate from OAI-SearchBot, which indexes pages for ChatGPT search results, and ChatGPT-User, which fetches a page when a user asks ChatGPT about it. Each responds to its own rules in robots.txt, which is why the checker reports them individually.
Should I block AI crawlers?
It depends on which ones. Blocking training bots like GPTBot or ClaudeBot keeps your content out of future model training and costs you little today. Blocking search and assistant bots is different: they fetch pages to answer live buyer questions, so blocking them removes you from AI answers and cuts your AI search visibility. Most brands that want to be recommended keep the search and assistant bots open.
Why does my robots.txt restrict every bot?
Usually because of boilerplate rules under the wildcard group: paths like /admin or /cart are disallowed for all crawlers equally, and every bot inherits them. That is normal hygiene, not an AI problem. The checker distinguishes these shared exclusions from rules that single out AI bots, and flags a bot as blocked only when it cannot read your actual content.
How do I allow a specific AI bot?
Add a rule group naming that bot's user agent in robots.txt. For example, User-agent: PerplexityBot on one line and Allow: / on the next opens your whole site to Perplexity, regardless of what the wildcard group says, because crawlers follow the most specific group that names them. Repeat the pattern for each bot you want to admit, and keep any private paths disallowed under the wildcard group.
Free to start
Get recommended by AI search models.
Run a free AI search visibility check, then let the Honeyb agent do the work that gets you into the answers.
ChatGPT
Claude
Gemini