AI visibility check
Can the crawlers behind ChatGPT, Claude, Perplexity, Gemini and Copilot reach and read this page? One blocker first, then the detail.
Enter any public URL. You’ll get the one thing stopping AI crawlers from reading the page (if anything is), then a crawler-by-crawler table and what the HTML actually tells a machine.
What it checks
Four things, in the order they can stop a crawler. Each one makes everything after it irrelevant, which is why the result leads with the first that fails.
1. Is the page hidden on purpose?
A noindex robots meta tag or X-Robots-Tag header tells every well-behaved crawler to drop the page. It is the most common leftover from a staging site, and it overrides everything else.
2. Does the firewall let crawlers in?
We fetch the page as a phone browser and again identifying as GPTBot, ClaudeBot and PerplexityBot. If the browser gets the page and the crawlers get a 403 or a challenge page, something in front of your server — typically Cloudflare’s Block AI bots toggle, Bot Fight Mode, or a WAF rule — is turning them away. robots.txt never enters into it.
3. Does robots.txt allow the crawlers that matter?
Sixteen crawlers, grouped by what blocking them costs. The fetchers that open a page to cite it in an answer (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) are the ones that affect visibility. The training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) are a policy choice — blocking them costs nothing today, and the report says so rather than scoring you down for it. Googlebot and Bingbot are listed because AI Overviews and Copilot are built on the classic indexes.
4. Is the content in the HTML?
Googlebot renders JavaScript. The fetchers behind AI answers largely do not — they read the HTML the server sends. A single-page app that ships an empty <div id="root"> and fills it in later is a blank page to them. We count the visible words in the raw HTML and flag empty app shells.
After those four: does the page label itself (title, H1, meta description), carry structured data, declare a language and a canonical, and offer an llms.txt? None of these is a blocker; all of them help a machine decide what the page is.
Guides
Each one is the complete fix for a headline this check can give you — no product to buy, real settings and real robots.txt.
- Which AI crawlers to allow in robots.txtYou block training bots and accidentally block the ones that cite you.
- Your firewall is blocking AI crawlersrobots.txt allows everyone, yet AI crawlers get a 403 before they read a byte.
- Your content only exists after JavaScript runsThe server sends an empty shell and a script fills it in — AI crawlers see nothing.
- Structured data that AI crawlers actually useThe page is readable but nothing tells a machine what it is.
- llms.txt — what it is, what it isn't, and a templateA cheap bet, not a fix — here's the honest evidence and a template.
All eighteen guidesThe diagrams, free to reuse
What it deliberately does not do
It does not tell you whether AI products cite you. That depends on what people ask and on ranking inside each product, and no outside tool can measure it — any tool that claims to is estimating. It also does not run JavaScript, by design: seeing the page the way a non-rendering crawler sees it is the point.
For the speed side of the same page, run the Core Web Vitals check — a page AI crawlers can read but that takes six seconds to load has a different problem first.
Common questions
Why doesn't ChatGPT know about my website?
Usually one of three reasons, in this order of likelihood: a firewall or CDN setting is refusing AI crawlers before they read a byte (Cloudflare's 'Block AI bots' toggle is on by default for new zones); robots.txt blocks the fetchers that cite pages, not just the training bots; or the page's content only exists after JavaScript runs and the crawler never runs it. This check tests all three. The fourth reason — the page is readable but nobody asks questions it answers — is not something a tool can fix.
Should I block GPTBot and ClaudeBot?
That is a genuine choice, and blocking them costs you nothing visible today: GPTBot and ClaudeBot collect data for training future models. What you should not block by accident are the on-demand fetchers — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot — which open a page to cite it in an answer. Many block lists lump them together. The check separates the two so you can see which you have actually blocked.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended only controls whether Gemini is trained on your content. AI Overviews and AI Mode use the ordinary search index, which Googlebot builds — the only way out of them is the only way out of Google Search, which is blocking Googlebot or using noindex.
What is llms.txt, and do I need one?
A proposed convention: a Markdown file at /llms.txt describing the site and linking its most useful pages, for language models rather than browsers. It costs ten minutes and hurts nothing, but as of now no major AI provider has confirmed reading it. Treat it as a cheap bet, and fix the actual blockers first.
The check says a crawler is blocked, but I never blocked anything.
Check your CDN. Cloudflare added a one-click 'Block AI bots' setting in 2024 and enables bot-fight features on many plans by default; Vercel, Akamai and Fastly have equivalents. Also check your WAF's managed rules — several vendors' 'bad bot' lists include the AI fetchers. robots.txt is irrelevant in that case, because the crawler never receives the page.