GPTBot and robots.txt: what blocking it actually does

One company, three crawlers

OpenAI publishes three user-agents, and the confusion around “blocking ChatGPT” comes from treating them as one. They are not.

• GPTBot collects pages that may be used to train future models. It crawls on its own schedule, like a search engine spider, and it is the only one of the three whose absence you would never notice: nothing a visitor sees today depends on it.
• OAI-SearchBot builds the index behind ChatGPT search. When ChatGPT shows a list of sources under an answer, those pages came from this crawler. Block it and the site drops out of that list.
• ChatGPT-User fetches a page live when a user asks ChatGPT about it, pastes a link, or a custom GPT’s action opens it. It is not a crawler in the scheduling sense — it acts on a person’s request — and blocking it means ChatGPT cannot read your page even when a user explicitly asks it to.

OpenAI’s own documentation lists all three as honouring robots.txt, and each has its own published IP range list so you can verify a request came from where it claims. The single most common mistake on the web is a “block AI” snippet that names all three, written by someone who only meant the first.

Run the AI visibility check on your page before you edit anything: it reads your live robots.txt — the one the CDN serves, not the one on disk — and reports each of the three by name.

The rule, three ways

Opt out of training, stay visible in ChatGPT

The most common intent, and the one most block lists get wrong. One group, one token.

User-agent: GPTBot
Disallow: /

# Nothing here for OAI-SearchBot or ChatGPT-User: they inherit the
# "*" group, which allows everything except your private paths.
User-agent: *
Allow: /
Disallow: /admin/

Why this works: a crawler uses the most specific group that names it and ignores the rest. GPTBot finds its own group and stops. OAI-SearchBot and ChatGPT-User find no group of their own and fall back to *, which lets them in.

Allow training, but keep some paths out

Useful for a site that is happy to be trained on in general but has sections it would rather not contribute — a members’ area, a support forum, drafts.

User-agent: GPTBot
Allow: /
Disallow: /members/
Disallow: /forum/
Disallow: /drafts/

Be explicit about everything

If you have inherited a robots.txt with a history of copy-pasted blocks, the safest file is one that names each OpenAI token and states its rule, so nobody has to reason about inheritance later:

# OpenAI — training: opt out
User-agent: GPTBot
Disallow: /

# OpenAI — ChatGPT search index: allow
User-agent: OAI-SearchBot
Allow: /

# OpenAI — ChatGPT fetching a page a user asked about: allow
User-agent: ChatGPT-User
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Allow: / in a group of its own is not redundant here. If a later edit adds a broad Disallow under *, the explicit groups keep the two answer-time fetchers working regardless.

Three mistakes that look like the rule above

1. Grouping the three tokens together

# Blocks ChatGPT search and ChatGPT-User too — probably not what you meant
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
Disallow: /

Consecutive User-agent lines share the rules that follow them. This block removes the site from ChatGPT search and stops ChatGPT opening it on request, to gain nothing beyond what the first line alone achieves.

2. A wildcard that catches everything

User-agent: *
Disallow: /

Left over from a staging site more often than written on purpose. Every crawler with no group of its own — Googlebot included — reads this as “go away”. The check reports it as the headline, because nothing else on the report matters until it is gone.

3. Relying on a rule the CDN rewrote

Cloudflare’s managed robots.txt prepends its own block of training-crawler rules to the file your server sends. Those rules are training-only, so they do no harm, but they mean the file you edit and the file the internet reads are different. Always check the live file from a browser, or with curl:

curl -s https://example.com/robots.txt | grep -i -A2 "GPTBot\|OAI-SearchBot\|ChatGPT-User"

Verifying it is really GPTBot

A robots.txt rule is a request. GPTBot honours it, but anything can put “GPTBot” in a user-agent string, and scrapers do. OpenAI publishes the IP ranges each crawler uses as a JSON file — openai.com/gptbot.json for GPTBot, with equivalents for the other two — and a request from outside those ranges is not OpenAI. If you see “GPTBot” traffic after adding a Disallow, check the source address before assuming the rule is being ignored. If it is genuinely outside the ranges, a firewall rule matching the IP, not the user-agent, is the tool.

In your CDN’s bot analytics, OpenAI’s crawlers appear as verified bots precisely because of these lists. That is also where to look if the check reports a 403 for one of them: the block is in the firewall, and that is a different fix.

What blocking GPTBot does not do

• It does not remove content OpenAI already collected. The rule applies to future crawls.
• It does not affect Bing, Copilot or any Microsoft product. Those are built on Bingbot’s index; OpenAI’s crawlers are separate.
• It does not stop ChatGPT from knowing the site exists. Models learn about sites from many sources — other pages that link to you, the search index — and a GPTBot block only removes your own pages from the training pile.
• It does not block the ChatGPT browsing feature. That is ChatGPT-User, and it has its own token.

Check it worked

Re-run the check. In the robots.txt table, GPTBot should read “Blocked (fine)” if you opted out of training, and OAI-SearchBot and ChatGPT-User should read “Allowed” whatever you decided. In the fetch grid above the table, every row should show 200 — if a crawler shows 403 while the browser gets the page, the problem was never robots.txt. The overview guide covers the other fourteen crawlers the same way; ClaudeBot and PerplexityBot have the same split with their own wrinkles.