Robots.txt Tester
Test any URL against Googlebot, Bingbot and 16 AI crawlers, from GPTBot to PerplexityBot. It reads your live robots.txt the way crawlers do, shows the exact line that decides, and re-tests as you edit.
Testing a file before it goes live?
Robots.txt generator
Choose who gets in and the file writes itself. It handles the rule most hand-written files get wrong: a group that names a crawler replaces the * group for that crawler, so your private paths are repeated inside every group you let in.
All other crawlers
Googlebot, Bingbot and anything not named below
AI search
OAI-SearchBot, Claude-SearchBot, PerplexityBot
AI assistant
ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User
AI training
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider
AI dataset
CCBot
Your robots.txt
User-agent: * Allow: /
Upload it as robots.txt at the root of your domain, served as plain text. Crawlers cache it for up to a day, so changes take effect on their next fetch.
How the tester decides
Three questions, in the order crawlers ask them.
01
Could the file be read?
A 4xx means no restrictions. A 5xx, a 429 or a timeout means the whole site is off limits. Up to five redirects are followed.
02
Which group applies?
Every group naming the crawler's token, merged. The * group only when none does. Case does not matter; version numbers are ignored.
03
Which rule wins?
The longest matching path. Allow wins a tie. * matches anything, $ ends the path, and the order of lines is irrelevant.
The rules follow RFC 9309, the IETF robots.txt standard, and Google’s documented parser where the standard leaves room, including its 500 KiB limit and the typos it tolerates.
Robots.txt matching, by example
Paths are case-sensitive and match from the start. These are Google’s documented examples, and the tester’s unit tests.
/fish
Matches /fish, /fish.html, /fishheads, /fish.php?id=1
Not /Fish.asp, /catfish, /desert/fish
/fish/
Matches /fish/, /fish/salmon.htm
Not /fish, /fish.html, /animals/fish/
/*.php
Matches /index.php, /folder/any.php.file.html
Not /, /windows.PHP
/*.php$
Matches /filename.php, /folder/filename.php
Not /filename.php?parameters, /filename.php5
/fish*.php
Matches /fish.php, /fishheads/catfish.php?x
Not /Fish.PHP
/$
Matches /
Not /page, /?x=1
When Allow and Disallow both match
/page
Allow: /p, Disallow: /
Allow: /p is longer
/folder/page
Allow: /folder, Disallow: /folder
Allow, same length, Allow wins ties
/page.htm
Allow: /page, Disallow: /*.htm
Disallow: /*.htm is longer
/page.php5
Allow: /page, Disallow: /*.ph
Allow, same length
/
Allow: /$, Disallow: /
Allow: /$ is longer
/page.htm
Allow: /$, Disallow: /
Disallow: /, the Allow only matches the root
Robots.txt for AI crawlers
AI companies run separate crawlers for separate jobs, and each obeys its own group. Training crawlers such as GPTBot and ClaudeBot collect pages for future models. Search crawlers such as OAI-SearchBot and PerplexityBot index pages so an engine can cite them. Assistant fetchers such as ChatGPT-User load a page when someone asks about it. Blocking the first is a content-licensing choice. Blocking the other two takes you out of AI answers.
A rule is only half of access. A site can allow every AI crawler in robots.txt and still refuse them at the CDN or firewall before the file is ever read. The AI crawler access checker tests that connection layer with real crawler requests. For the full list of tokens and what each one does, see the AI crawler user agent reference.
Access is the floor, not the goal. Being readable does not make an engine recommend you. The free AI visibility check shows whether it does, and the Honeyb agent does the work that gets you named.
Want to get recommended by AI?
Check your AI search visibility, then let the Honeyb agent write, fix, and earn what gets you recommended. Free to start.
Frequently asked questions
How do I check a robots.txt file?
Enter any URL above. The tester fetches the live robots.txt from that site's root the way a crawler does, follows up to five redirects, and shows the rule that decides access to that exact URL for Googlebot, Bingbot and 16 AI crawlers. Edit the file in place to test a change before you publish it; what you type stays in your browser.
How do I find a website's robots.txt file?
It always lives at the root of the host, for example https://example.com/robots.txt, in lowercase. Each protocol, host and port has its own file, so blog.example.com and www.example.com are governed separately, and a robots.txt placed in a subfolder is ignored.
Should I have a robots.txt file?
You do not need one: a missing file that returns a 404 tells every crawler it may fetch everything. Add one to keep crawlers out of low-value areas such as internal search or the cart, to list your sitemap, and to set a policy for AI crawlers. The file is public and advisory, so never use it to hide private pages; use authentication or a noindex tag for that.
Is robots.txt still a thing?
More than ever. The IETF published it as a formal standard, RFC 9309, in September 2022, and AI crawlers made it the main switch for who may use your content: OpenAI, Anthropic, Perplexity, Google and Apple all publish robots.txt tokens for their crawlers. One file now governs both search and AI visibility, so a single broad rule can cost you both.
Is robots.txt legally binding?
robots.txt is a technical convention, not a law or a contract, and RFC 9309 states that its rules are not a form of access authorization. Reputable crawlers follow it voluntarily. If you need enforceable limits, pair it with terms of use and server-side access control, and take legal advice for your jurisdiction.
Can crawlers ignore robots.txt?
Yes. Nothing technically stops a client from requesting a disallowed URL, which is why robots.txt steers well-behaved crawlers rather than securing anything. To actually stop a bot, refuse it at the server or CDN. To confirm what real AI crawlers can reach, the AI crawler access checker tests the connection itself as well as the rules.
Does Google still have a robots.txt tester?
No. Google retired the robots.txt Tester in Search Console in December 2023 and replaced it with the robots.txt report, which lists the files Google fetched and their parse errors but does not test a URL against your rules. This tester applies the matching rules Google documents: the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards.
Which rule wins when Allow and Disallow conflict?
The most specific one, meaning the longest matching path. For /page.htm, Disallow: /*.htm beats Allow: /page because it is longer. When two matching rules are the same length, Allow wins. The order of lines in the file does not matter, which is the most common misreading.
How do I block AI training but stay in AI answers?
Give the training crawlers their own group with Disallow: / and leave the answer-time crawlers open. Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot. Answer-time: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User. The generator on this page writes that file for you, and blocking Google-Extended does not affect Google Search.
Free to start
Get recommended by AI search models.
Run a free AI search visibility check, then let the Honeyb agent do the work that gets you into the answers.
ChatGPT
Claude
Gemini