Your firewall is blocking AI crawlers

How to tell it’s the firewall

Your robots.txt allows everyone, your page is public, and yet AI products never seem to see it. Run the AI visibility check and look at the first grid: the page fetched as a phone browser and as three AI crawlers. If the browser gets 200 and a crawler gets 403, 429, 503 or a bot challenge, something in front of your server is refusing requests by user-agent. The crawler never sees your page or your robots.txt — the diagram above is the whole story.

You can reproduce it from any terminal:

# as a browser
curl -sI -A "Mozilla/5.0 (Linux; Android 13) Chrome/126" https://example.com/ | head -1
# as ClaudeBot
curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/ | head -1
# as GPTBot
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/ | head -1

Different status lines for the same URL means a user-agent rule. A cf-mitigated: challenge header means Cloudflare specifically.

Cloudflare

Four separate settings can produce this. Check them in order.

1. AI Crawl Control (previously “Block AI bots”)

Dashboard → your zone → AI Crawl Control (older dashboards: Security → Bots → Block AI bots). Each crawler is listed with an Allow/Block switch. Allow the fetchers that cite pages — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot — and decide about the training bots separately. Do not flip the global toggle off if you want to keep blocking training.

2. Bot Fight Mode / Super Bot Fight Mode

Security → Settings → Bot fight mode. On free plans it challenges anything scored as automated, with no exceptions possible; the only options are off, or upgrading to a plan where verified bots can be allowed. Super Bot Fight Mode (Pro and up) has a “verified bots” allow setting — turn it on.

3. WAF custom rules and managed rulesets

Security → WAF. Look for custom rules matching http.user_agent with “bot”, and for managed ruleset overrides. Add an exception rule above them:

# Expression for a "Skip" rule, placed first
(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "Claude-User") or
(http.user_agent contains "Claude-SearchBot") or
(http.user_agent contains "PerplexityBot")

4. Managed robots.txt

Not a block, but related: AI Crawl Control can also prepend a training-crawler Disallow block to your robots.txt. It does not touch the answer fetchers, so it costs nothing — but it is worth knowing it is there, because your file on disk and the file the internet sees are different.

Other providers

• Vercel — Firewall → Bot Protection. The “AI bots” managed rule blocks trainers and fetchers together; switch it to log-only or add a custom rule allowing the five fetchers.
• AWS CloudFront / WAF — the Bot Control managed rule group’s CategoryAI label covers all of them. Add a rule that allows requests carrying that label when the user-agent matches a fetcher, placed before the block.
• Akamai — Bot Manager → known bots → the “AI” category. Set the fetchers to monitor rather than deny.
• Sucuri, Wordfence, and similar plugins — search their blocklists for “GPT”, “Claude” and “Perplexity” and remove the fetchers. Wordfence’s rate limiting can also 429 a crawler that fetches several pages quickly; raise the crawler threshold.
• nginx / Apache on your own server — look for if ($http_user_agent ~* ...) blocks or a BrowserMatchNoCase deny list. Copy-pasted “bad bot” lists from 2023 nearly all include the AI fetchers.

Check it worked

Re-run the check. All four fetches should return 200. Then look at your firewall’s own bot analytics a day later — the genuine crawlers, with verified IP ranges, should appear as allowed. If robots.txt still shows a crawler as blocked once the firewall is fixed, that is the other guide.