Why AI crawlers get a 404 on pages that exist

First, confirm which kind of 404 it is

The AI visibility check fetches your page as a phone browser and as six crawlers, and shows the status each one got. A 404 in the crawler rows with 200 in the browser row is this problem. Reproduce it from a terminal to see the whole response, not just the status:

U="https://example.com/some/page"
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -A "Mozilla/5.0 (Linux; Android 13) Chrome/126" "$U"
curl -s -D - -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" "$U" | head -40

Three things in the crawler response tell you which section below applies: the server and CDN headers (who sent the 404), a location header on any intermediate hop (a redirect went wrong), and whether the body is your own not-found template or something generic (a rule fired at the edge).

Cause 1: client-side routing

A single-page app serves one HTML file for every path and lets JavaScript work out which screen to show. Hosts that support this configure a fallback: any path that is not a real file returns index.html. There are two failure shapes.

The fallback returns 404 status. Some hosts serve the fallback page with a 404 code, so the browser gets the app (which renders the page) and a crawler that trusts status codes gets “not found”. On Netlify this is the difference between a 200 and a 404 in the redirects rule; on Vercel it is the rewrites configuration; on S3/CloudFront it is the error document setting.

# Netlify _redirects — the fallback must be a 200 rewrite, not a 404
/*    /index.html   200

The fallback returns 200 but an empty shell. Now the status is right and the content is missing: the crawler receives <div id="root"></div> and a script tag, which it classifies as a soft 404. That is the JavaScript-only content problem, and the fix is server-side rendering or pre-rendering for the routes that matter.

Cause 2: a rule that answers bots with 404

Security tooling sometimes hides refused resources as 404 rather than 403, so an attacker cannot tell a blocked page from a missing one. Applied to an AI fetcher, it is indistinguishable from the page not existing. The signs: the 404 body is a generic vendor page rather than your theme’s not-found template, and the headers name a CDN or WAF. Places to look:

• Cloudflare WAF custom rules with a “Block” action and a custom response set to 404; also Transform Rules that rewrite bot requests to a dead path. Every Cloudflare setting that touches crawlers is listed separately.
• nginx if ($http_user_agent ~* (bot|crawler)) { return 404; } — a common pattern in copied configs.
• Apache RewriteCond %{HTTP_USER_AGENT} rules ending in [R=404], and BrowserMatchNoCase deny lists that a custom ErrorDocument turns into 404s.
• WordPress security plugins with “hide” options: Wordfence and Solid Security can return 404 for blocked requests; Sucuri’s WAF has a “stealth” block mode.

The fix is not to remove the rule but to exempt the fetchers that cite pages, by user-agent or, better, by the vendor’s verified-bot classification.

Cause 3: redirects that end nowhere

A crawler follows redirects; a chain that assumes a browser can strand it. Three recurring shapes:

• Locale detection. The site reads Accept-Language, finds none (crawlers often send nothing), and redirects to a default locale path that does not exist for this page, or to a country selector that is itself 404. Serve the canonical page when there is no language header instead of guessing.
• Trailing slash. /page redirects to /page/ on one layer and back on another, until the crawler gives up; or the slash form is served for browsers and the bare form 404s. Pick one form, redirect the other once, and make sure the CDN and the origin agree.
• Cookie-gated redirects. A consent or age gate redirects everything without its cookie to an interstitial that serves the crawler a 404 or a soft 404. Overlay the gate on the page instead of redirecting to it.

curl -sIL shows every hop. The check follows up to five and reports the final status; a chain longer than that is reported as unreachable.

Cause 4: the crawler asks for a slightly different URL

• Case sensitivity. Linux servers distinguish /About from /about. A link elsewhere on the web with the wrong case works for a browser only because a redirect or a case-insensitive host was in front; move the site and the crawler’s stored URL 404s. Add a lowercase redirect at the edge.
• Percent-encoding. URLs containing spaces, accented characters or + arrive encoded differently from different clients. A route that only matches one encoding 404s the other. Normalise on the server.
• Query strings that change the route. A tracking parameter that a router treats as part of the path. Strip or ignore unknown parameters.

Cause 5: HEAD is not GET

Some crawlers and checkers send HEAD first. A server or function that handles GET and returns 404 (or 405) for HEAD looks broken to them. Frameworks usually get this right; hand-written serverless functions and API routes often do not. Make HEAD return the GET headers with an empty body.

Cause 6: hosting that treats non-browsers as “coming soon”

A few page-builder hosts and maintenance-mode plugins serve the real site to browsers with a session cookie and a placeholder to everything else, sometimes with a 404 status. Maintenance mode left on for a deploy is the usual story. Check the plugin list and the host’s site status page.

Fixing it, in order

1. Reproduce with two curl commands, as a browser and as a crawler, and diff the full responses.
2. If the crawler response has a CDN or WAF header and a generic body: exempt the fetchers in that product’s rules.
3. If the crawler followed a redirect the browser did not: fix the locale, slash or cookie logic to serve the page without a browser’s headers.
4. If the status is 200 but the body is a shell: server-render or pre-render the route.
5. If the status is 404 from your own template: check case, encoding and the fallback configuration of your host.
6. Re-run the check; every crawler row should match the browser row.

Check it worked

The check should show 200 in all seven fetch rows for the same URL, and a “Present” verdict for content without JavaScript. If the fetch rows are fine and the content row says missing, the remaining problem is rendering, covered in the JavaScript-only guide. If some rows are 403 rather than 404, it is a straightforward block: the firewall guide.