GPTBot robots.txt for all three OpenAI crawlers

Blocking GPTBot in robots.txt takes two lines. What those two lines leave open matters just as much.

GPTBot is only OpenAI's training crawler. ChatGPT search uses a different bot, and pages read inside a conversation come from a third one. If you want to keep your content out of training but still show up when someone asks ChatGPT for a recommendation, you need to treat the three separately.

OpenAI runs three bots that matter for a website owner, and each one takes its own line in robots.txt. According to OpenAI's crawler documentation (read on 28 September 2026), GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it tells OpenAI not to use your content for that training. OAI-SearchBot is for search. It decides which websites can appear in ChatGPT search answers, and a website that blocks it won't be shown there, apart from plain navigational links. ChatGPT-User is not a crawler in the usual sense. It fetches a page when a person asks ChatGPT or a Custom GPT a question, and OpenAI says robots.txt rules may not apply to it because a person started the request.

OpenAI states that each setting is independent. Its own example is to disallow GPTBot and allow OAI-SearchBot. Your pages stay out of model training and stay eligible for ChatGPT search. OpenAI says robots.txt changes take about 24 hours to reach its search systems.

OpenAI also lists a fourth bot, OAI-AdsBot. It checks pages submitted as ads on ChatGPT. OpenAI says its data isn't used for training.

These are the full user-agent strings OpenAI publishes. The bot is named by the token in front of the version number. That's GPTBot, OAI-SearchBot and ChatGPT-User. Match on that token in your logs and rules, not on the whole string. The version number changes over time.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Anyone can send these strings. A scraper that calls itself GPTBot is still a scraper. The IP lists in the next sections are how you tell a real OpenAI request from a copy.

Pick the setup that matches what you want. Each one goes in the robots.txt file at the root of your website.

Block training, stay in ChatGPT search

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This matches OpenAI's own example. The explicit Allow for OAI-SearchBot isn't strictly needed when nothing else blocks it, but it keeps the intent readable for the next person who edits the file.

Block all of OpenAI's crawlers

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /

This keeps you out of training and out of ChatGPT search answers. ChatGPT-User may still fetch pages when a person asks for them, because OpenAI says robots.txt may not apply there. To stop those requests you have to block them on the server, by user agent or by the addresses in openai.com/chatgpt-user.json.

Allow OpenAI but keep private paths closed

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /
Disallow: /admin
Disallow: /api/

The mistake that undoes your other rules

A crawler follows only the group that names it. It ignores the User-agent: * group once it finds its own. That's the robots.txt standard, RFC 9309.

So if your * group blocks /admin and you add a separate User-agent: GPTBot group with only Allow: /, GPTBot may now read /admin. Copy every private-path rule into each named group, or list the bots in the same group as *.

Both bots are about ChatGPT answers, which is why they get mixed up. The difference is who starts the visit.

OAI-SearchBot visits on its own schedule. It builds the index, and that index decides whether your website can appear in ChatGPT search results. Block it and you drop out of those answers.

ChatGPT-User visits because a person is in a conversation right now. Someone pasted your URL, or ChatGPT decided to open your page while answering them. OpenAI says ChatGPT-User doesn't crawl the web automatically and isn't used to decide whether content may appear in search. A ChatGPT-User hit in your logs means a real person's question led to your page.

Allowing OAI-SearchBot is what keeps you eligible for ChatGPT search. Blocking ChatGPT-User doesn't remove you from search, and it doesn't reliably stop the fetches either.

Your access log shows which of the three bots already visit. On a standard Nginx or Apache setup, count the hits per bot with one command.

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User" /var/log/nginx/access.log \
  | sort | uniq -c

To see which pages each bot requested, print the path field for one bot at a time. In the combined log format the path is the seventh field.

grep "OAI-SearchBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Then check that a hit really came from OpenAI. OpenAI publishes the IP ranges of each bot as JSON at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. This Python snippet checks one address against the GPTBot list. Swap the file name for the other bots.

import ipaddress, json, urllib.request

ip = ipaddress.ip_address("20.171.207.10")  # the address from your log
data = json.load(urllib.request.urlopen("https://openai.com/gptbot.json"))
nets = [
    ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
    for p in data["prefixes"]
]
print(any(ip in net for net in nets))

True means the address sits in OpenAI's published range. False means the request only borrowed the name. Blocking those in your firewall does nothing to your ChatGPT presence.

We let all three in. The robots.txt on verifieddr.com (read on 28 September 2026) names GPTBot and ChatGPT-User in the same group as User-agent: *. That group allows the public pages and disallows private product paths such as /admin, /settings and the JSON under /api/. OAI-SearchBot isn't named, so it falls under the same * rules.

Listing the bots in the shared group avoids the mistake above. A separate GPTBot group would drop the private-path rules for GPTBot unless they were copied into it.

Grepping a log once tells you what happened last week. It doesn't tell you when a bot stops coming, or that a robots.txt edit locked out the search bot you wanted.

The AI crawler log in VerifiedDR records visits from known AI bots, OpenAI's three included. It runs on your server. Pick the WordPress plugin, a middleware for Next.js, Cloudflare, Node.js or Python, or a cron script that reads your Nginx or Apache log. Nothing runs in your visitors' browsers.

Each bot gets a type. GPTBot shows as Training, OAI-SearchBot as AI search index and ChatGPT-User as Live answer fetch. The Used in answers count adds up the live-answer fetches, so you see how often a person's question led an assistant to your pages. The log also reads your robots.txt and marks a bot as blocked when its rules disallow the whole website. If a blocked bot keeps visiting, the log shows those visits.

Two limits to know. The log recognises bots by the user-agent token and doesn't check IP addresses, so use the snippet above to verify a suspicious hit. And responses served from a CDN cache never reach your server, so they can't be counted. Crawler logs are part of the paid plans and keep about 90 days of visits per website.

Being read is the first step. Whether ChatGPT then names you is a separate question, and AI Visibility measures it by asking the questions your buyers ask. Links from trusted websites are the other half, and the free domain rating checker shows where your website stands.