GPTBot is OpenAI's web crawler for collecting content used to train its foundation models. It is one of three separate OpenAI agents, and it is not the one that powers ChatGPT's search results. Blocking GPTBot in robots.txt opts your content out of model training, but it does not remove you from ChatGPT answers that retrieve pages live, because that retrieval runs under OAI-SearchBot and ChatGPT-User.
That distinction is the whole reason this page exists. The most common configuration error we see is a site blocking every OpenAI user agent in one line, which quietly removes it from ChatGPT search answers while the operator believes they have only opted out of training.
The three OpenAI agents, and what each one does
All three strings below are taken from OpenAI's crawler documentation at developers.openai.com/api/docs/bots, checked 20 July 2026.
| Agent | Token for robots.txt | What it does | Blocking it means |
|---|---|---|---|
| GPTBot | `GPTBot` | Crawls pages to train and improve OpenAI's foundation models | Your content is not used for training |
| OAI-SearchBot | `OAI-SearchBot` | Indexes pages for ChatGPT's search feature | You are not shown in ChatGPT search answers |
| ChatGPT-User | `ChatGPT-User` | Fetches a specific page when a user or a custom GPT asks for it | ChatGPT cannot open your page on request |
OpenAI's documentation is explicit on the middle row: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. It is equally explicit that ChatGPT-User is not used for crawling the web in an automatic fashion, so it is a per-request fetch rather than a systematic crawl.
The full user agent strings, as published on the same page:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/botMatch on the product token (`GPTBot`, `OAI-SearchBot`, `ChatGPT-User`) rather than the full string. Version numbers move, and a rule keyed to `GPTBot/1.4` stops matching the day OpenAI ships 1.5. For the crawlers run by Google, Anthropic, Perplexity, Meta and the rest, see our AI crawler user agent reference.
Copy-paste robots.txt configurations
Three configurations cover almost every real decision. Robots.txt rules are matched per user agent group, so each block needs its own `User-agent` line rather than a comma-separated list.
Allow everything, the default if you write nothing at all:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /Block training only, keep ChatGPT search visibility. This is the configuration most publishers actually want:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /Block all three OpenAI agents. Understand that this removes you from ChatGPT search results entirely:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /Partial blocks work the same way as for any crawler. If you want GPTBot kept out of gated or member content but allowed on the public marketing site, list the paths:
User-agent: GPTBot
Disallow: /members/
Disallow: /pricing/internal/
Allow: /Robots.txt is a request, not an enforcement mechanism. It is honoured by the operators who publish user agents in the first place, which includes OpenAI, but it does nothing about scrapers that do not identify themselves. If you need enforcement, that is a firewall or WAF rule, not a text file.
How to verify a request really came from GPTBot
User agent strings are a header. Anyone can send one, and a meaningful share of traffic claiming to be GPTBot is not GPTBot. OpenAI's published verification method is IP range matching against JSON files it maintains per agent:
| Agent | Published IP range file |
|---|---|
| GPTBot | `https://openai.com/gptbot.json` |
| OAI-SearchBot | `https://openai.com/searchbot.json` |
| ChatGPT-User | `https://openai.com/chatgpt-user.json` |
Want to see this in action?
See how every major AI model talks about your brand. Free to start.
The GPTBot file listed 21 IPv4 CIDR ranges when we fetched it on 20 July 2026, with a documented update date of 30 October 2025. The practical implication is that the list changes, so hard-coding today's ranges into a firewall rule and forgetting about it will eventually block legitimate crawls. Fetch the file on a schedule and rebuild the allowlist from it.
Verification logic, in order:
1. Read the user agent and check for the product token. 2. Take the connecting IP, not any value from `X-Forwarded-For` that you do not control. 3. Test that IP for membership in a CIDR range from the relevant JSON file. 4. If the token says GPTBot and the IP is outside the published ranges, treat the request as spoofed.
Note the difference from Googlebot here. Google supports reverse DNS verification, where you resolve the IP to a hostname and forward-resolve it back. OpenAI does not document a reverse DNS verification path for GPTBot, so CIDR matching against the JSON files is the method to rely on. If you see a guide telling you to reverse-resolve GPTBot to an openai.com hostname, treat it as unverified.
Server-side, the cheapest version of this is a scheduled job that pulls the three JSON files, merges the prefixes into an IP set, and lets your reverse proxy answer anything outside the set with a 403 when the user agent claims to be an OpenAI agent.
The decision table
| Your situation | Allow | Block | What it costs you |
|---|---|---|---|
| Marketing site, wants AI referrals | All three | Nothing | Your public copy enters training data |
| Publisher with a licensing position | OAI-SearchBot, ChatGPT-User | GPTBot | Nothing in ChatGPT search, preserves the negotiating position |
| Paywalled archive | OAI-SearchBot on free pages | GPTBot everywhere, all agents on paid paths | Paid content is invisible to ChatGPT, which is usually the point |
| Docs or support site | All three | Nothing | Fewer support tickets is normally worth more than the training concern |
| Legal or compliance restriction on content reuse | Nothing | All three | Full removal from ChatGPT search |
| Ecommerce product catalogue | All three | Checkout, account, search-result URLs | Crawl budget wasted on parameter URLs if you do not block them |
The strategic argument, stated honestly
Here is what is known. Blocking GPTBot removes your pages from the corpus OpenAI uses for model training, on OpenAI's own statement. Blocking OAI-SearchBot removes you from ChatGPT search answers, also on OpenAI's own statement. These are two separate decisions with two separate consequences and they should be made separately.
Here is what is inferred rather than documented. Content already used in training before you added the block stays in models that have already been trained. Nobody publishes a mechanism for retroactive removal from a trained model, so a block is forward-looking only. Treat any claim that blocking GPTBot removes you from ChatGPT as false.
The commercial argument for keeping training access open is weaker than it looks, because training data is not where most brand mentions come from in practice. Ahrefs found AI visibility correlates most strongly with third-party mentions and video rather than on-page work, which points at what other sites say about you rather than what your own pages contain. Semrush found Reddit is 40.1% of all AI citations, the single most-cited source. Your own robots.txt has no influence over either.
The argument for keeping search access open is much stronger, and it is the one people get wrong. Live retrieval is how a brand that did not exist at training time appears in an answer at all. If you want the mechanics, see how AI search works and whether ChatGPT can search the internet.
One more piece of context on why crawler access is necessary but not sufficient. Semrush found 62% of AI citations never name the brand being cited. Being crawled gets your page into the retrieval pool. It does not guarantee the answer says your name, which is a different problem covered in why your brand isn't showing up in ChatGPT.
Related files people confuse with robots.txt
Robots.txt controls access. It does not describe your content. Two adjacent conventions come up in the same conversation:
| File | Status | What it does |
|---|---|---|
| `robots.txt` | Long-established, honoured by OpenAI | Grants or denies crawler access per user agent |
| `llms.txt` | Proposed convention, not honoured by any major AI vendor as an access control | A markdown map of your key pages |
| Meta `noindex` | Established for search engines | Does not substitute for a robots.txt crawler block |
We have written up what the first of those actually is in what is llms.txt, with real files in llms.txt examples. Neither replaces the robots.txt rules above.
Auditing what you have now
Check four things, in this order. First, fetch your own `/robots.txt` and confirm the three OpenAI tokens appear as you intend, with no catch-all `User-agent: *` `Disallow: /` above them that a CDN or staging config left behind. Second, grep your access logs for each product token and confirm you are seeing real traffic. Third, spot-check a sample of those IPs against the JSON files. Fourth, confirm your WAF or bot-management vendor is not blocking OpenAI agents independently of robots.txt, which is a common invisible failure because the rule lives outside your repository.
The full walkthrough sits in our GEO audit checklist, and the tactics for turning access into actual citations are in how to get cited by AI. If you want to know whether the engines are showing their working at all, do ChatGPT and other assistants cite sources covers which ones expose citations.
Getting crawler configuration right is the floor, not the ceiling. Once GPTBot and OAI-SearchBot are set the way you want them, the question that matters is whether AI assistants actually name you when buyers ask about your category. Run a free check at /tools/ai-visibility-checker to see what ChatGPT, Claude, Gemini and Perplexity say about your brand today.
