Which hosts and firewalls block AI crawlers by default

The short answer

AI crawlers are blocked by default on Cloudflare zones created since July 2025, on many Squarespace sites, and behind any WAF whose “bad bot” list predates the distinction between training crawlers and answer-time fetchers. They are not blocked by default on Netlify, on plain VPS hosting, on most shared hosting, or by WordPress itself. WordPress security plugins sit in between: they rarely block by name out of the box, but their rate limits and fake-crawler checks catch AI fetchers routinely.

None of that is in your robots.txt, which is why the file looks fine. The only way to know your own site’s answer is to fetch a page as each crawler and compare the status codes, which is what the AI visibility check does and what the curl lines at the end of this page reproduce.

Host by host

Defaults change; the table is what these providers did as of September 2026. Your own site’s answer comes from the fetch test, not from this table.

Cloudflare

The big one, because a large share of the web is behind it. Three defaults matter. Zones added since 1 July 2025 have Block AI bots on from the start; the setting now lives in AI Crawl Control with a per-crawler switch. Bot Fight Mode, common on free plans, challenges anything its heuristics score as automated and cannot exempt individual crawlers. And the managed robots.txt, when on, adds training-crawler disallows to the file you serve. Every one of these is covered setting by setting in the Cloudflare guide.

Vercel

Nothing is blocked on a new project. The Firewall page offers Bot Protection (challenge traffic that does not look like a browser) and a managed rule for AI bots that can be set to log, challenge or deny. Turn either on and the answer-time fetchers are caught with the trainers, because the managed list does not split them. The fix is a custom rule above it that allows the fetcher user-agents, or setting the managed rule to log only and handling training crawlers in robots.txt.

Netlify

Nothing is blocked. There is no AI-crawler setting; the paid Traffic Rules feature does rate limiting and geo-blocking, and an Edge Function can inspect the user-agent if you write one. A Netlify site that blocks AI crawlers is doing so because someone built it. The one Netlify-specific gotcha is a site whose content is client-rendered — that is a different problem and the check reports it separately.

WordPress hosts and plugins

WordPress itself blocks nothing; its default virtual robots.txt disallows /wp-admin/ and allows everything else. The blocking comes from what is installed around it.

• WP Engine and Kinsta both sit behind Cloudflare with their own rule sets. Verified crawlers get through; unverified or very fast ones may not. Neither exposes an AI-crawler toggle, so a 403 in the check means a support conversation, with the crawler name and timestamp.
• Sucuri’s WAF is the most aggressive of the common ones. Its default posture challenges or blocks traffic it classifies as bots, and its blocklists have historically included AI user-agents. The settings are under Firewall → Settings → Access Control, where you can whitelist user-agents, and Security, where the bot-blocking level is set.
• Wordfence does not block AI crawlers by name unless you add a pattern. What catches them is Firewall → Rate Limiting: a crawler fetching more pages per minute than the threshold is throttled or blocked, and “block fake Google crawlers” refuses anything claiming a name from an unexpected address. Raise the crawler threshold, or add the fetcher user-agents to the allow list, rather than turning rate limiting off.
• Other security plugins (iThemes/Solid Security, All In One WP Security, MalCare, Cloudways’ bot protection) ship user-agent blocklists of varying vintage. Search each plugin’s settings for “user agent” and look for GPT, Claude, Perplexity or a generic “bot” match.

Hosted platforms

Shopify serves a permissive robots.txt and lets you edit it through robots.txt.liquid; its edge does its own bot mitigation, which can challenge unverified automated requests, and there is no merchant control over that. Squarespace has an explicit switch, Settings → Crawlers → Block known artificial intelligence crawlers, which is on for many sites; it writes Disallow rules for a list that includes both trainers and fetchers, so if you want to be citable, switch it off and manage the trainers by hand. Wix and Webflow block nothing by default and expose a robots.txt editor; Webflow also offers a block toggle in its SEO settings.

The two-minute test

From any terminal, fetch one page four ways and compare the first line of each response:

U="https://example.com/"
curl -sI -A "Mozilla/5.0 (Linux; Android 13) Chrome/126" "$U" | head -1
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" "$U" | head -1
curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" "$U" | head -1
curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "$U" | head -1
curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" "$U" | head -1

Same status on every line: nothing in front of you distinguishes crawlers, and your robots.txt is the whole story. Different statuses: a user-agent rule is firing, and the headers of the refused response (server, cf-ray, x-sucuri-id, and so on) tell you whose. The check runs this with six crawlers, names the vendor from the headers, and tells you which setting to open.

Check it worked

After changing a setting, the fetch rows for the crawlers you allowed should read 200 within a minute; CDN rules apply immediately. Then look at the vendor’s own bot analytics a day later to see the genuine crawlers, from their verified ranges, being admitted. For pages that exist but return 404 to crawlers — a different symptom with different causes — see the 404 guide.