Does blocking GPTBot hurt my store?

Three kinds of AI crawler, three separate decisions

The big AI companies now split their bots by job. A training crawler collects pages that may be used to train future models. A search crawler builds the index an AI assistant searches when it answers a question and cites sources. A user-triggered fetcher opens one page because a person asked the assistant to, the way a browser would.

Each one has its own robots.txt token, so each is its own decision. OpenAI states it plainly: each setting is independent, and you can allow OAI-SearchBot while you disallow GPTBot.

The same split at Anthropic, Google, Perplexity, Apple and Meta

If your goal is to opt out of training while staying visible to AI assistants, you need the training token for each company, not just GPTBot. Two of them, Google-Extended and Applebot-Extended, are control tokens: no crawler visits under that name, the token only tells Google or Apple how pages their main crawler fetched may be used.

How to block training and keep search

1. 1

   Step 1: Read your current robots.txt first

   Open https://yourstore.com/robots.txt and note every User-agent group. If one already names an AI crawler, edit that group instead of adding a second one: under RFC 9309, groups for the same crawler are merged, which makes the result hard to read. Our guide on checking which AI crawlers your store blocks walks through this.

2. 2

   Step 2: Add one group for the training tokens

   List each training token on its own User-agent line, then one Disallow: /. One group can name several crawlers.

   # Opt out of AI model training
   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: Google-Extended
   User-agent: Applebot-Extended
   User-agent: meta-externalagent
   Disallow: /

3. 3

   Step 3: Leave the search crawlers in the catch-all group

   You don't need an Allow group for OAI-SearchBot, Claude-SearchBot or PerplexityBot. A crawler with no group of its own follows User-agent: *. If you give it its own group, it ignores the * group completely, including rules that keep bots out of cart, checkout and account pages.

4. 4

   Step 4: If you do add a named group, repeat the private paths

   Sometimes you need a named group, for example when * disallows a section you want AI search to read. Then copy the private-path rules into it.

   User-agent: OAI-SearchBot
   User-agent: Claude-SearchBot
   User-agent: PerplexityBot
   Disallow: /cart
   Disallow: /checkout
   Disallow: /account
   Allow: /

5. 5

   Step 5: Give it a day, then check again

   OpenAI and Perplexity say changes take up to about 24 hours, and RFC 9309 tells crawlers not to cache robots.txt for longer than 24 hours. Recheck the live file after you publish, because some platforms and plugins rewrite it.

The blocks that do hurt a store

• User-agent: * with Disallow: /. It blocks every crawler that has no group of its own, including Googlebot and every AI search crawler. It is often left over from a staging site.
• A robots.txt that returns a 5xx error. RFC 9309 says crawlers must then assume everything is disallowed. A 404 has the opposite effect: crawlers may fetch everything.
• A firewall or CDN bot rule. robots.txt is a request; a firewall is a wall. A bot setting at your CDN can turn away search crawlers and shoppers' AI agents that robots.txt allows. See does Cloudflare block AI agents.
• Blocking Googlebot or Storebot-Google to stop AI. Google uses Googlebot for Search, including its AI features, and Storebot-Google for all Google Shopping surfaces. Blocking them removes you from Search or Shopping, not just from AI.

When blocking GPTBot makes sense

Opting out of training is a content decision, not a visibility one. Stores with original writing, photography or research often opt out; stores whose pages are mostly product facts often don't bother. Either way, nothing in OpenAI's documentation connects GPTBot to whether ChatGPT search shows or links your store. That is OAI-SearchBot's job.

If you allow both, OpenAI says it may use the results of one crawl for both purposes, so allowing GPTBot does not double the load on your store.