ClaudeBot, Claude-User and Claude-SearchBot in robots.txt

Three tokens, two consequences

Anthropic’s crawlers follow the same split as OpenAI’s: one collects for training, the others fetch pages so the product can show them to a person. The names are different and the split is the same.

• ClaudeBot — the training crawler. It gathers pages that may be used to improve future Claude models. Blocking it is a choice about training and changes nothing a Claude user sees today.
• Claude-User — fetches a page live when someone asks Claude about it or pastes a link. Every request is on behalf of a person’s question. Block it and Claude cannot read your page even when explicitly asked.
• Claude-SearchBot — builds the index Claude’s search feature draws on. Block it and your pages stop appearing among the sources Claude cites.

There are also two names you will find in old block lists: anthropic-ai and Claude-Web. Anthropic retired both in favour of ClaudeBot. Rules written for them are said to be honoured for backwards compatibility, but a rule for a retired name is not a plan; write the live token.

Run the AI visibility check first — it reads the live robots.txt and reports all three Anthropic tokens, plus the retired one, so you can see what your current file actually does before editing it.

The rule

Opt out of training, stay citable

User-agent: ClaudeBot
Disallow: /

# Claude-User and Claude-SearchBot inherit "*" and stay allowed
User-agent: *
Allow: /
Disallow: /admin/

Explicit about all three

# Anthropic — training: opt out
User-agent: ClaudeBot
Disallow: /

# Anthropic — Claude fetching a page a user asked about: allow
User-agent: Claude-User
Allow: /

# Anthropic — Claude search index: allow
User-agent: Claude-SearchBot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

Training allowed, some paths excluded

User-agent: ClaudeBot
Allow: /
Disallow: /members/
Disallow: /internal/

The matching rules are Google’s, which every major crawler follows: the most specific group wins, the longest matching path wins within it, and an Allow beats a Disallow of the same length. So a Disallow: /members/ and an Allow: /members/public/ do what they look like they do.

Cleaning up the old tokens

If your file still has this, it is doing less than it appears to:

User-agent: anthropic-ai
User-agent: Claude-Web
Disallow: /

Replace it with the ClaudeBot group above. Keep the old lines for a week if you like a belt with your braces; then delete them, because the next person to edit the file will assume they are the live rule.

When the crawl itself is the problem

ClaudeBot has a reputation for fetching a lot of pages quickly on some sites. Two things help before a block. First, Anthropic says the crawler backs off when the server returns 429 Too Many Requests, so a rate limit at the CDN or web server that returns 429 rather than 403 slows it without excluding it. Second, a Crawl-delay line is honoured by some crawlers and ignored by others, so it is worth a try and not worth relying on. If neither is enough and you do not want to be trained on anyway, the Disallow is the right answer — it costs no visibility.

What you should not do is reach for a firewall rule matching the string “Claude”. That catches Claude-User and Claude-SearchBot too, and the check will report a 403 for the crawlers that cite pages. The firewall guide covers how to allow those while still limiting the trainer.

Returning 429 instead of 403

If the goal is “slow down” rather than “go away”, the status code matters. A 403 tells a crawler the page is forbidden and it will not retry for a long time; a 429 tells it to back off and try later, which is what Anthropic says ClaudeBot does. In nginx, a rate limit that returns 429 for a named user-agent looks like this:

map $http_user_agent $is_ai_trainer {
    default 0;
    ~*ClaudeBot 1;
    ~*GPTBot   1;
}
limit_req_zone $is_ai_trainer zone=ai_trainers:1m rate=30r/m;

server {
    location / {
        limit_req zone=ai_trainers burst=10 nodelay;
        limit_req_status 429;
        ...
    }
}

Because the map only matches the training tokens, Claude-User and Claude-SearchBot are untouched. The same shape works in Caddy’s rate-limit module and in Cloudflare’s rate-limiting rules, where the expression is http.user_agent contains "ClaudeBot" and the action is Block with a 429 response.

Verifying the requests

Anthropic publishes the IP ranges its crawlers use. A request carrying “ClaudeBot” from outside those ranges is somebody else, and no robots.txt rule will affect it. Check the address before concluding that a rule is being ignored, and block by IP if it really is an impostor. CDNs that maintain a verified-bots list do this comparison for you and label the genuine requests.

What a ClaudeBot block does not do

• It does not remove pages already collected. It applies to future crawls.
• It does not stop Claude knowing your site exists via other pages that link to it.
• It does not touch Claude-User or Claude-SearchBot, so it has no effect on whether Claude can cite you — unless you wrote all three into one group by mistake.
• It does not affect the llms.txt file Anthropic says its crawlers read; that file is for fetchers, and a training block is a separate decision.

Check it worked

Re-run the check: ClaudeBot reads “Blocked (fine)” if you opted out, Claude-User and Claude-SearchBot read “Allowed”, and the fetch row “As Claude-User” shows 200. The same split applies to OpenAI and, with one important twist, to Perplexity.