PerplexityBot user agent and robots.txt rules

Perplexity runs two bots and each one sends its own user agent. The PerplexityBot user agent is the one that crawls. Perplexity-User is the one that fetches a page while someone waits for an answer. Both strings below come from Perplexity's crawler documentation, checked on 28 September 2026.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Match on the product token, not the whole string. A rule that looks for PerplexityBot or Perplexity-User keeps working when the version number or the browser prefix changes. In robots.txt the bot name matches without regard to case. A firewall rule may not, so lowercase the user agent before you compare it.

The difference decides what a robots.txt rule can do for you. PerplexityBot builds the index behind Perplexity search. Perplexity-User acts for one person at one moment.

So a robots.txt block stops PerplexityBot from crawling your website. It won't reliably stop Perplexity-User from opening a page a person asked about. For that you need a rule on your server or firewall.

Add the group to the robots.txt at the root of your website. Perplexity says changes can take up to 24 hours to show up in its systems.

User-agent: PerplexityBot
Disallow: /

User-agent: PerplexityBot
Disallow: /members/
Allow: /

User-agent: PerplexityBot
Allow: /

Watch out for one trap. A crawler follows the most specific group that names it and ignores the User-agent: * group. If your * group keeps bots out of /admin/ and you add a PerplexityBot group, copy that line into the new group too.

A Perplexity-User group is still worth writing. It documents your intent. It won't enforce it, because Perplexity says this fetcher generally ignores robots.txt.

To refuse both bots, answer their requests with a 403. Two common setups are below. Test on a staging copy first. A typo in a server rule can block real visitors.

if ($http_user_agent ~* "(PerplexityBot|Perplexity-User)") {
    return 403;
}

(lower(http.user_agent) contains "perplexitybot") or (lower(http.user_agent) contains "perplexity-user")

Think about what the block costs before you ship it. Perplexity recommends allowing PerplexityBot to make sure your website appears in its search results. Blocking it trades that exposure for control over your content. You can also decide per bot. Block the crawlers that collect training data and keep search and answer bots open.

Anyone can put PerplexityBot in a user agent. The IP address is the part a scraper can't fake as easily. Perplexity publishes one list per bot, as JSON with CIDR prefixes. On 28 September 2026 the PerplexityBot list held 8 prefixes and the Perplexity-User list held 4.

This short Python script checks one IP address from your log against both lists.

import ipaddress, json, sys, urllib.request

LISTS = {
    "PerplexityBot": "https://www.perplexity.com/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.com/perplexity-user.json",
}

ip = ipaddress.ip_address(sys.argv[1])
for bot, url in LISTS.items():
    with urllib.request.urlopen(url) as response:
        prefixes = json.load(response)["prefixes"]
    for entry in prefixes:
        cidr = entry.get("ipv4Prefix") or entry.get("ipv6Prefix")
        if cidr and ip in ipaddress.ip_network(cidr):
            print(f"{ip} is on the {bot} list ({cidr})")
            sys.exit(0)
print(f"{ip} is not on a Perplexity list")
sys.exit(1)

Run it as python3 check_perplexity_ip.py 18.97.1.229. A hit that claims to be PerplexityBot from an address on neither list is not from Perplexity's declared crawler. The lists change, so fetch them fresh. Perplexity's own advice for firewalls is to combine the user agent and the IP address in one rule and update the IP ranges on a schedule.

Nginx and Apache write the user agent at the end of every line in the default combined log format. A few shell commands answer the first questions. Point them at your own log path.

grep -oE "PerplexityBot|Perplexity-User" /var/log/nginx/access.log | sort | uniq -c

grep "PerplexityBot" /var/log/nginx/access.log | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20

grep -E "PerplexityBot|Perplexity-User" /var/log/nginx/access.log | awk '{print $1}' | sort -u

Look at the status codes first. A 404 or 500 on a page PerplexityBot tries to read is a page Perplexity can't cite. Pages served from a CDN cache never reach your server, so your origin logs will show fewer hits than the bots really made.

On 4 August 2025 Cloudflare reported that Perplexity reached websites that had blocked its declared bots by using a generic browser user agent and IP addresses outside its published lists. Cloudflare said it removed Perplexity from its verified bots and added the pattern to its managed rules that block AI crawling.

Perplexity disputed the report. As TechCrunch reported, a spokesperson said the bot Cloudflare named was not Perplexity's, and the company argued that fetching a page for a user is different from crawling. For a website owner the practical lesson holds either way. A user agent is a claim. Check the IP address before you trust it, and put a firewall rule behind any block that matters to you.

Grepping a log answers today's question. It won't tell you next month whether PerplexityBot still reads your pricing page. The AI Crawlers tab in VerifiedDR keeps that record for you.

You connect your website once with the WordPress plugin, a middleware for Next.js, Cloudflare, Node.js or Python, or a cron script over your Nginx or Apache logs. It runs on your server, so visitors get no extra JavaScript. From then on VerifiedDR logs PerplexityBot as an AI search index bot and Perplexity-User as a live answer fetch. You see the pages each bot reads, the broken pages it hits and when it last came by, with about ninety days of history. VerifiedDR also reads your robots.txt and marks a bot as blocked when the file keeps it out of the whole website.

Two honest limits. VerifiedDR recognises bots by their user agent and does not check IP addresses, so run the script above when a hit looks off. And a Perplexity-User visit can still show up next to a blocked mark, because that fetcher generally ignores robots.txt. Crawler logs are part of the paid plans. The AI scrapers page shows how the log works. To see whether the pages bots read turn into mentions, pair it with AI Visibility.