If I block GPTBot, does ChatGPT stop showing my site?

No. GPTBot only collects training data. ChatGPT search results come from OAI-SearchBot, and ChatGPT opening a page a user asked about is ChatGPT-User. You can block GPTBot and keep both of those, and many sites do exactly that.

Does robots.txt actually stop AI crawlers?

The named ones from OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Meta, Amazon and Common Crawl all publish that they honour it, and independent checks of server logs bear that out. Some smaller scrapers ignore it. robots.txt is a request, not a wall — if you need a wall, that is a firewall rule, which is the subject of the companion guide.

Is blocking training crawlers bad for SEO?

No. Google-Extended, GPTBot, ClaudeBot and the rest have no connection to search ranking. Googlebot is the only crawler that affects Google Search, and AI Overviews use its index — blocking Google-Extended changes nothing there. Blocking Googlebot itself removes you from Google, so never put it in the same group.

My robots.txt has rules I never wrote. Where did they come from?

Almost certainly your CDN. Cloudflare's 'managed robots.txt' feature prepends a block of training-crawler Disallow rules and 'Content-Signal' lines to whatever your server serves. It is on by default for many zones. Fetch your robots.txt from a browser and look for a 'BEGIN Cloudflare Managed content' comment; the toggle is under AI Crawl Control.

Should I add an llms.txt too?

It costs ten minutes and hurts nothing: a Markdown file at /llms.txt that says what the site is and links the pages worth reading. But no major AI company has confirmed using it yet, so do it after the robots.txt and firewall are right, not instead of them.

Which AI crawlers to allow in robots.txt

Sixteen AI crawlers, split by what blocking them actually costs — the fetchers that cite your page in answers versus the bots that only train models. With a copy-paste robots.txt.

This diagram is free to reuse with credit — all eighteen, as SVG or PNG.

Two kinds of crawler, one file

Every AI company runs at least two crawlers, and they do different jobs. One collects pages to train future models. The other fetches a page on demand so the product can quote or cite it in an answer — when someone asks ChatGPT about your product, ChatGPT-User goes and reads the page right then. Block the first kind and nothing visible changes. Block the second and you disappear from AI answers. Most block lists on the internet lump them together, which is how sites end up invisible by accident.

Run the AI visibility check on your page first: it reads your robots.txt and reports each crawler by group, so you know which of the rules below you actually need to change.

The crawlers, by what blocking them costs

Fetchers that cite pages in answers — keep these allowed

OAI-SearchBot — ChatGPT search results
ChatGPT-User — ChatGPT opening a page a user asked about
Claude-SearchBot — Claude’s search results
Claude-User — Claude opening a page a user asked about
PerplexityBot — Perplexity’s index and citations

The search indexes AI answers are built on — never block these

Googlebot — Google Search, and therefore AI Overviews and AI Mode
Bingbot — Bing, and therefore Microsoft Copilot
Applebot — Siri and Spotlight suggestions
Amazonbot — Alexa answers

Training-only crawlers — your choice, costs nothing today

GPTBot (OpenAI), ClaudeBot (Anthropic)
Google-Extended (Gemini training — Google Search is unaffected)
Applebot-Extended (Apple models — Siri is unaffected)
CCBot (Common Crawl, the open dataset many models train on)
meta-externalagent (Meta), Bytespider (ByteDance)

A robots.txt that gets this right

This allows everything that can send you a visitor and blocks only training. Delete the training block if you are happy to be trained on; keep the rest either way.

# Everyone: allowed, except the private bits
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# Training-only crawlers: opt out (optional — costs no visibility)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

Sitemap: https://example.com/sitemap.xml

Two details matter. A crawler uses the most specific group that names it and ignores * entirely, so listing GPTBot in its own group with Disallow: / blocks it even though * allows everything. And consecutive User-agent lines share the rules that follow them, which is why the training block needs only one Disallow.

The three mistakes that make sites invisible

1. Blocking the on-demand fetchers along with the trainers

Copy-pasted “block all AI” lists usually include ChatGPT-User, OAI-SearchBot and PerplexityBot. Those are the ones that cite you. Take them out of the block.

2. `Disallow: /` under `User-agent: *`

Left over from a staging site, this blocks every crawler that has no group of its own — including Googlebot. The check flags it as the headline because nothing else matters until it is gone.

3. Rules you didn’t write

Cloudflare’s managed robots.txt prepends its own training-crawler block. It is training-only, so it does no harm — but if you are deliberately opting in to training, you need to switch it off in Cloudflare, not just in your file.

Check it worked

Fetch your robots.txt in a browser (not from your codebase — the CDN may have changed it), then re-run the check. Every crawler in the first two groups should read “Allowed”. Blocked training crawlers read “Blocked (fine)”. If a crawler shows allowed here but the fetch test above it shows a 403, the problem is not robots.txt at all — it is the firewall, and that is the other guide.

Common questions

If I block GPTBot, does ChatGPT stop showing my site?: No. GPTBot only collects training data. ChatGPT search results come from OAI-SearchBot, and ChatGPT opening a page a user asked about is ChatGPT-User. You can block GPTBot and keep both of those, and many sites do exactly that.
Does robots.txt actually stop AI crawlers?: The named ones from OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Meta, Amazon and Common Crawl all publish that they honour it, and independent checks of server logs bear that out. Some smaller scrapers ignore it. robots.txt is a request, not a wall — if you need a wall, that is a firewall rule, which is the subject of the companion guide.
Is blocking training crawlers bad for SEO?: No. Google-Extended, GPTBot, ClaudeBot and the rest have no connection to search ranking. Googlebot is the only crawler that affects Google Search, and AI Overviews use its index — blocking Google-Extended changes nothing there. Blocking Googlebot itself removes you from Google, so never put it in the same group.
My robots.txt has rules I never wrote. Where did they come from?: Almost certainly your CDN. Cloudflare's 'managed robots.txt' feature prepends a block of training-crawler Disallow rules and 'Content-Signal' lines to whatever your server serves. It is on by default for many zones. Fetch your robots.txt from a browser and look for a 'BEGIN Cloudflare Managed content' comment; the toggle is under AI Crawl Control.
Should I add an llms.txt too?: It costs ten minutes and hurts nothing: a Markdown file at /llms.txt that says what the site is and links the pages worth reading. But no major AI company has confirmed using it yet, so do it after the robots.txt and firewall are right, not instead of them.

Check a page against this

Enter any public URL. You’ll get the one thing stopping AI crawlers from reading the page (if anything is), then a crawler-by-crawler table and what the HTML actually tells a machine.

Other guides

How to fix a slow LCP — Your main image or headline takes too long to appear.
How to fix layout shift — Content jumps around while the page loads.
How to fix slow INP — Taps and clicks take a moment to do anything.
How to fix a slow server response — Your server is slow to start replying at all.
How to eliminate render-blocking CSS and JS — CSS and JS files stop anything appearing on screen.
Why you're seeing "no field data" — Google has no real-visitor data for your site yet.
How to fix oversized and outdated images — Your images weigh far more than they need to.
How to cut unused JavaScript — You ship JavaScript this page never runs.
How to fix your cache headers — Returning visitors re-download files they already have.
Why your score is different on every run — The score is different every time you test.
Fast on desktop, slow on mobile — Desktop looks fine; the phone test says otherwise.
Why Search Console still shows the old numbers — You fixed it weeks ago and Search Console hasn't noticed.
How to stop plugins loading on every page — A form plugin's CSS on pages with no form — times every plugin.
Your firewall is blocking AI crawlers — robots.txt allows everyone, yet AI crawlers get a 403 before they read a byte.
Your content only exists after JavaScript runs — The server sends an empty shell and a script fills it in — AI crawlers see nothing.
Structured data that AI crawlers actually use — The page is readable but nothing tells a machine what it is.
llms.txt — what it is, what it isn't, and a template — A cheap bet, not a fix — here's the honest evidence and a template.