Every Cloudflare setting that blocks AI crawlers, and how to allow-list

Why the fix is not in robots.txt

Every Cloudflare setting on this page acts at the edge, before a request reaches your server. A crawler that is refused there never fetches your robots.txt, never sees your page, and never reads your carefully written Allow rules. The symptom is always the same and the check reports it as the headline: a phone browser gets 200, and a request identifying as GPTBot, ClaudeBot or PerplexityBot gets 403, 503, or a “Just a moment” challenge page. A cf-mitigated: challenge header in the response names Cloudflare specifically.

Five settings can produce that. Work through them in order; most sites have two or three of them on.

1. Block AI bots, and its successor AI Crawl Control

The one-click Block AI Scrapers and Crawlers toggle arrived in July 2024 for every plan, under Security → Bots. It became the default for new zones on 1 July 2025. In September 2025 it was folded into AI Crawl Control (previously AI Audit), which is where it lives now: your zone → AI Crawl Control → Crawlers.

The screen lists each known AI crawler with its operator, its purpose, and an Allow/Block switch. Two mistakes to avoid. Do not flip the global toggle off to fix visibility — that also allows the training crawlers you may want blocked. And do not assume the defaults distinguish training from citing; they largely do not. Set each row deliberately:

• Allow: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, DuckAssistBot, plus Googlebot, Bingbot and Applebot if they appear. These send you visitors.
• Your choice: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, cohere-ai. Training only; blocking costs no visibility.

AI Crawl Control also shows a request log per crawler — how many fetches, how many blocked, over the last day. That log is the ground truth the check approximates: if it shows OAI-SearchBot requests being blocked, no robots.txt edit will help until this screen is fixed.

2. Bot Fight Mode and Super Bot Fight Mode

Security → Bots (older dashboards: Security → Settings). Bot Fight Mode, on the free plan, challenges requests that Cloudflare’s heuristics score as automated. It exempts verified bots in principle, but it offers no allow-list, and any crawler not on Cloudflare’s verified list is fair game. PerplexityBot lost verified status in August 2025, so on a free plan with Bot Fight Mode on, Perplexity is typically challenged whatever AI Crawl Control says.

Your options on the free plan are binary: leave it on and accept that some fetchers are challenged, or turn it off and rely on the WAF and rate limiting for bot defence. Super Bot Fight Mode (Pro and above) adds three separate controls — definitely automated, likely automated, verified bots — and the fix is to set verified bots to Allow and leave the other two alone.

3. WAF custom rules and managed rulesets

Security → WAF → Custom rules. Look for anything matching http.user_agent against “bot”, “crawler”, “GPT” or “Claude”, and for rules using the cf.client.bot or cf.verified_bot_category fields. Copy-pasted “block bad bots” expressions from 2023 catch every AI fetcher.

Rather than editing each blocking rule, add one Skip rule above them that exempts the crawlers you want, with “All remaining custom rules” selected in what to skip:

(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "Claude-User") or
(http.user_agent contains "Claude-SearchBot") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "DuckAssistBot")

On plans with verified-bot categories you can write the same thing more robustly, so a spoofed user-agent does not get the exemption:

(cf.verified_bot_category in {"AI Search" "AI Assistant" "Search Engine Crawler"})

Also check Security → WAF → Managed rules for overrides that block by user-agent, and Security → Settings for a “Definitely automated” action set to Block.

4. AI Labyrinth

AI Crawl Control (or Security → Bots on older layouts) → AI Labyrinth. Launched in March 2025 and off by default, it serves crawlers Cloudflare has decided to block a set of hidden links to AI-generated decoy pages, so a crawler that ignores robots.txt wastes its time on nonsense. It only applies to requests already classified as unwanted, which is exactly why it matters here: if your AI Crawl Control rows are wrong, a fetcher you meant to allow is not just blocked but fed decoys, and a model that reads them learns garbage about your site.

Fix the allow rules first. Then AI Labyrinth is safe to leave on for the crawlers you really do not want.

5. Managed robots.txt and Content Signals

AI Crawl Control → Manage AI crawlers with robots.txt (introduced with the Content Signals policy in September 2025). When on, Cloudflare prepends a block to the robots.txt your origin serves: a Content-Signal: line stating whether the site consents to search, ai-input and ai-train, and Disallow groups for the training crawlers. This is the one setting on the page that is not a block — it is a file edit, honoured by crawlers that read robots.txt — and it does not name the answer-time fetchers.

Two things follow. If you want to opt in to training, switch this off; your own file cannot override a group Cloudflare prepends. And always read the live file, because yours is not the one being served:

curl -s https://example.com/robots.txt | head -30
# look for "BEGIN Cloudflare Managed content" and a Content-Signal line

Pay per crawl

Alongside the July 2025 default block, Cloudflare launched a private beta that lets a site charge crawlers per request rather than block them outright, using the HTTP 402 status. It applies to crawlers that opt into paying, which as of now is a short list. It is a commercial choice rather than a visibility fix; if you enable it, an unpaid fetcher still gets refused, and the check will report that as a block.

Confirming it from the other side

Cloudflare’s own analytics are the authoritative view. AI Crawl Control’s request log shows each crawler’s allowed and blocked counts; Security → Events shows individual blocked requests with the rule that fired. A day after your changes, the fetchers you allowed should appear with allowed counts and no events. The check sends requests from one server with the published user-agent strings; the analytics show the genuine crawlers from their verified ranges. When they disagree, the analytics win.

Check it worked

Re-run the check. All seven rows in the fetch grid should read 200, and the “in front of the server” line will still say Cloudflare — that is fine; the point was never to remove Cloudflare. If a crawler still shows a block, the honesty note applies: Cloudflare may be blocking our server’s request while letting the verified crawler through, so confirm in the request log before changing anything else. Other vendors’ equivalents are in the general firewall guide, and which hosts block out of the box is in the defaults guide.