PerplexityBot and Perplexity-User in robots.txt

Two crawlers, and only one of them listens

Perplexity runs two named user-agents, and they differ in a way OpenAI’s and Anthropic’s do not.

• PerplexityBot builds the index Perplexity answers from. Perplexity documents it as honouring robots.txt and says it does not train models on what it collects. Block it and your pages leave the pool Perplexity draws sources from.
• Perplexity-User fetches a page live when a user asks about it or opens a link inside an answer. Perplexity’s documentation says this agent generally ignores robots.txt, on the grounds that a person requested the page. So a Disallow for it is a statement of preference, not a control.

That second point is the twist. For OpenAI and Anthropic, the user-triggered fetcher honours robots.txt, so one file governs both behaviours. For Perplexity, the file governs the index and not the on-demand fetch. If you want to stop Perplexity-User, the tool is a firewall rule, and you should decide whether you actually want that: it means a person who asks Perplexity about your page gets told it could not be read.

Run the AI visibility check to see what your live robots.txt says about both tokens and whether your firewall answers PerplexityBot with a 200.

The rule for PerplexityBot

Stay in the index (the default worth keeping)

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

The explicit group is worth having even though * already allows it: a future wildcard Disallow will not catch it by accident.

Keep some sections out of the index

User-agent: PerplexityBot
Allow: /
Disallow: /members/
Disallow: /search/

Leave the index entirely

User-agent: PerplexityBot
Disallow: /

Only do this knowing what it costs: the site stops being discoverable through Perplexity questions. Unlike GPTBot or ClaudeBot, this is not a training decision.

Perplexity-User: what you can and cannot do

You can write a rule. It documents your intent and some tooling reads it:

User-agent: Perplexity-User
Disallow: /private/

You cannot rely on it. If a path must not be fetched on a user’s behalf, the options are authentication (a login wall stops every crawler), or a firewall rule on the user-agent string — which, as the August 2025 dispute with Cloudflare showed, only stops requests that carry the name. Before adding that rule, weigh what it does: a user asks Perplexity “what does this page say?” and receives “I could not access it”. For most sites that is a worse outcome than the fetch.

If you do want to stop Perplexity-User

Some sites have a real reason: paywalled articles, licensed data, a contract that forbids redistribution. Since robots.txt will not do it, the rule goes in the layer that sees the user-agent before the page is served. In Cloudflare, a WAF custom rule:

(http.user_agent contains "Perplexity-User") and not (http.request.uri.path contains "/public/")
# action: Block

In nginx:

if ($http_user_agent ~* "Perplexity-User") { return 403; }

Scope it to the paths that need it rather than the whole site, and leave PerplexityBot out of the expression — the index crawler honours robots.txt, and a firewall block on it also removes you from the answers you may still want to appear in. And remember what the rule does to a person: they asked Perplexity about your page and were told it could not be read. For a paywall that is the intended outcome; for a marketing site it is a lost visitor.

Verifying the requests

Perplexity publishes IP ranges for both agents as JSON files. A request carrying either name from an address outside them is not Perplexity, whatever the header says. The reverse is harder: traffic from addresses that are not on the list, with a browser-like user-agent, cannot be attributed to anyone. Cloudflare’s August 2025 report described exactly that pattern — fetches continuing after a block, from undeclared ranges, with generic Chrome user-agents — and Perplexity disputed the attribution. Nobody outside the two companies can settle it from logs alone.

The practical stance: treat the named user-agents as the polite path and configure them deliberately; treat unattributed browser-like traffic as you would any other scraper, with rate limits rather than name-based blocks. That is also the setting that keeps the genuine fetchers from OpenAI and Anthropic working.

The firewall is the more common problem

Because of the dispute, several CDNs removed PerplexityBot from their verified-bot lists, and some bot-management products now challenge it by default. That produces the pattern the check flags as an edge block: your robots.txt allows PerplexityBot, but a request carrying its user-agent gets a 403 or a challenge page. If you want Perplexity to index you, the fix is in the firewall — every Cloudflare setting involved and the other vendors are covered separately.

How this checker classifies Perplexity

PerplexityBot sits in the “cites pages in AI answers” group, because that is what Perplexity says it is for and what blocking it costs. Perplexity-User sits in the same group, with a note that its robots.txt verdict is advisory. Neither is counted as a training crawler. If Perplexity’s stated behaviour changes, the classification will change with it; the report says what it does today.

Check it worked

Re-run the check. PerplexityBot should read “Allowed” in the robots.txt table and 200 in the fetch grid. If the table says allowed and the grid says 403, the firewall is the problem, not the file. For the equivalent decisions on the other two big providers, see GPTBot and ClaudeBot; for the sixteen-crawler overview, the robots.txt guide.