GPTBot, OAI-SearchBot and ChatGPT-User are three different bots. Block the wrong one and you vanish from ChatGPT

In 2023 a lot of founders pasted User-agent: GPTBot / Disallow: / into their robots.txt and moved on. It felt like a free decision: no training on my content, no downside. Two years later those same sites are missing from ChatGPT search results, and the founders cannot work out why.

The reason is that "the OpenAI bot" is now three bots with three jobs, and the rules for one do not have to apply to the others. The same split exists at Anthropic and Perplexity. If you only remember one thing from this article: training, search and user fetches are separate decisions, and robots.txt lets you make them separately.

The three kinds of AI crawler

Training crawlers collect pages to build the next model. OpenAI's is GPTBot, Anthropic's is ClaudeBot, Common Crawl's CCBot feeds many open models, ByteDance runs Bytespider, Apple has Applebot-Extended, Meta has meta-externalagent. Google uses a robots-only token, Google-Extended, to let you opt out of Gemini training without touching Googlebot. Blocking these means your text is less likely to end up inside a model's weights. It does not affect whether an assistant can find you today.

Search-index crawlers build the index an assistant queries when a person asks a question. OAI-SearchBot powers ChatGPT search. Claude-SearchBot does the same for Claude. PerplexityBot is Perplexity's index. Googlebot and Bingbot feed AI Overviews and Copilot as well as classic search. Block these and you are not in the candidate set; the assistant cannot recommend what it cannot retrieve.

User-requested fetchers open one page because a person asked. ChatGPT-User, Claude-User and Perplexity-User are the user agents. They are not crawling; they are the assistant acting as a browser for one human. Block them and a person who pastes your URL into ChatGPT gets "I couldn't access that page" — which is a worse first impression than any summary.

What the 2023 robots.txt actually did

A file like this:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

only blocks GPTBot. That is fine. The problem is the more thorough versions that listed every AI user agent anybody had heard of, or the CDN "block AI bots" toggles that match on the string "GPT" or the OpenAI IP ranges. Those catch OAI-SearchBot and ChatGPT-User too. The site is now invisible to ChatGPT search and unreadable when a user shares its link — for the sake of an opt-out that only ever covered training.

Perplexity is the other common casualty: people blocked PerplexityBot because of a 2024 story about undisclosed crawling, and forgot that it is also the only way to be cited in Perplexity answers.

A robots.txt that says what you mean

Decide the three questions separately, then write them down explicitly. This is a reasonable default for a product that wants to be found:

# Search & answer engines — allow
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
Allow: /

# User-requested fetches — allow
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

# Training — your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

User-agent: *
Allow: /
Sitemap: https://yourproduct.com/sitemap.xml

Two notes on the training block. First, it is advisory. Well-run crawlers honour it; the ones you are actually worried about do not read robots.txt at all. Second, for a marketing site the case for blocking training is weaker than it looks: a model that has seen your product's name next to your category is a model that can mention it. Many founders now allow everything on the marketing site and block training only on docs or user-generated content.

Things robots.txt cannot see

Three other layers decide whether an assistant can read you, and none of them are in robots.txt:

• WAF and CDN rules. Cloudflare, Vercel and AWS all ship "block AI bots" switches. They match on user agent and on published IP ranges, and they return 403 — the crawler never gets as far as robots.txt. Check the setting, not just the file.
• X-Robots-Tag and <meta name="robots">. noindex here overrides an Allow in robots.txt. It is easy to leave noindex on from a staging config.
• JavaScript-only content. Fetchers read the server HTML. If the page is an empty <div id="root"> until React runs, the assistant sees a title and nothing else. This is the most common reason a model gives a vague answer about a product that is allowed everywhere.

How to check in one minute

Our free AI crawler access checker evaluates your robots.txt for fifteen AI user agents and then fetches your homepage with each one, from our servers, so you can see the status each bot gets. It also reads the meta robots tag, X-Robots-Tag and whether /llms.txt exists. The probe cannot prove how a WAF treats the official crawler's IP range, but it catches the user-agent mistakes, which is nearly all of them.

Then run What does AI think my product is? to see what a model reads on the page it is allowed to fetch. If the one-liner is wrong, the fix is on your homepage, not in robots.txt.

Summary

• Training crawlers, search crawlers and user fetchers are separate bots with separate user agents.
• Blocking search and user fetchers makes you invisible in ChatGPT, Claude and Perplexity answers; blocking training does not.
• Name each user agent explicitly, check your CDN's bot switch, and make sure the page is readable without JavaScript.
• Re-check after every infrastructure change; the defaults move.