# Does blocking GPTBot hurt my store?

> Blocking GPTBot only opts out of OpenAI model training. ChatGPT search uses OAI-SearchBot. How training, search and user-triggered AI crawlers differ.

- Canonical URL: https://aiscovery.io/resources/does-blocking-gptbot-hurt-your-store
- Markdown version: https://aiscovery.io/resources/does-blocking-gptbot-hurt-your-store.md
- Last updated: 2026-10-21

Blocking `GPTBot` only tells OpenAI not to use your pages to train its models; it does not remove your store from ChatGPT search, which uses a separate crawler, `OAI-SearchBot`. What hurts a store is blocking the search crawlers and the user-triggered agents by accident, usually with a catch-all rule or a firewall setting.

- Topic: AI crawlers
- Author: Aiscovery team
- Published: 2026-10-21
- Updated: 2026-10-21
- Reading time: 7 min read

## Key facts

- OpenAI runs separate bots: GPTBot for model training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages a person asks ChatGPT to open.
- OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
- Google-Extended controls Gemini training and grounding only. Google says it does not affect inclusion in Google Search.
- User-triggered fetchers from OpenAI, Perplexity, Google and Meta may not follow robots.txt, because a person asked for the page.
- OpenAI and Perplexity say a robots.txt change can take about 24 hours to take effect.

## Three kinds of AI crawler, three separate decisions

The big AI companies now split their bots by job. A training crawler collects pages that may be used to train future models. A search crawler builds the index an AI assistant searches when it answers a question and cites sources. A user-triggered fetcher opens one page because a person asked the assistant to, the way a browser would.

Each one has its own robots.txt token, so each is its own decision. OpenAI states it plainly: each setting is independent, and you can allow `OAI-SearchBot` while you disallow `GPTBot`.

OpenAI's bots and what a block does:

| Token | Job | What blocking it does |
| --- | --- | --- |
| `GPTBot` | Training generative AI foundation models | Signals that your content should not be used in training. OpenAI does not tie it to search. |
| `OAI-SearchBot` | Search results in ChatGPT | Your pages are not shown in ChatGPT search answers, though they can still appear as navigational links. |
| `ChatGPT-User` | Opens a page when a person asks ChatGPT, a custom GPT or a GPT Action to | OpenAI says robots.txt rules may not apply, because a person started the action. It is not used to decide what appears in search. |
| `OAI-AdsBot` | Checks landing pages submitted as ChatGPT ads | Only visits pages submitted as ads. OpenAI says its data is not used for training. |

## The same split at Anthropic, Google, Perplexity, Apple and Meta

If your goal is to opt out of training while staying visible to AI assistants, you need the training token for each company, not just GPTBot. Two of them, `Google-Extended` and `Applebot-Extended`, are control tokens: no crawler visits under that name, the token only tells Google or Apple how pages their main crawler fetched may be used.

Training, search and user-triggered tokens by company:

| Company | Training | Search | User-triggered |
| --- | --- | --- | --- |
| OpenAI | `GPTBot` | `OAI-SearchBot` | `ChatGPT-User` |
| Anthropic | `ClaudeBot` | `Claude-SearchBot` | `Claude-User` |
| Google | `Google-Extended` (control token) | `Googlebot`, `Storebot-Google` for Shopping | `Google-Agent` (no robots.txt token) |
| Perplexity | None: Perplexity says PerplexityBot is "not used to crawl content for AI foundation models" | `PerplexityBot` | `Perplexity-User` |
| Apple | `Applebot-Extended` (control token) | `Applebot` | Not documented |
| Meta | `meta-externalagent` | `meta-webindexer` | `meta-externalfetcher` |

> **Anthropic doesn't make that exception.** OpenAI, Perplexity, Google and Meta all say their user-triggered fetchers may not follow robots.txt. Anthropic says its bots honor robots.txt, and `Claude-User` can be blocked there: Anthropic says that prevents its system from retrieving your content in response to a user query.

## How to block training and keep search

### 1. Read your current robots.txt first

Open `https://yourstore.com/robots.txt` and note every `User-agent` group. If one already names an AI crawler, edit that group instead of adding a second one: under RFC 9309, groups for the same crawler are merged, which makes the result hard to read. Our guide on [checking which AI crawlers your store blocks](https://aiscovery.io/resources/check-which-ai-crawlers-your-store-blocks) walks through this.

### 2. Add one group for the training tokens

List each training token on its own `User-agent` line, then one `Disallow: /`. One group can name several crawlers.

**robots.txt: opt out of AI training**

```text
# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /
```

### 3. Leave the search crawlers in the catch-all group

You don't need an `Allow` group for `OAI-SearchBot`, `Claude-SearchBot` or `PerplexityBot`. A crawler with no group of its own follows `User-agent: *`. If you give it its own group, it ignores the `*` group completely, including rules that keep bots out of cart, checkout and account pages.

### 4. If you do add a named group, repeat the private paths

Sometimes you need a named group, for example when `*` disallows a section you want AI search to read. Then copy the private-path rules into it.

**robots.txt: a named group that keeps private paths closed**

```text
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart
Disallow: /checkout
Disallow: /account
Allow: /
```

### 5. Give it a day, then check again

OpenAI and Perplexity say changes take up to about 24 hours, and RFC 9309 tells crawlers not to cache robots.txt for longer than 24 hours. Recheck the live file after you publish, because some platforms and plugins rewrite it.

## The blocks that do hurt a store

- `User-agent: *` with `Disallow: /`. It blocks every crawler that has no group of its own, including Googlebot and every AI search crawler. It is often left over from a staging site.
- A robots.txt that returns a 5xx error. RFC 9309 says crawlers must then assume everything is disallowed. A 404 has the opposite effect: crawlers may fetch everything.
- A firewall or CDN bot rule. robots.txt is a request; a firewall is a wall. A bot setting at your CDN can turn away search crawlers and shoppers' AI agents that robots.txt allows. See [does Cloudflare block AI agents](https://aiscovery.io/resources/does-cloudflare-block-ai-agents).
- Blocking `Googlebot` or `Storebot-Google` to stop AI. Google uses Googlebot for Search, including its AI features, and Storebot-Google for all Google Shopping surfaces. Blocking them removes you from Search or Shopping, not just from AI.

> **Where agents get stuck.** A shopper's AI agent that is turned away at the door never reaches your product page. The store sees nothing in analytics, and the shopper sees an assistant that says it could not open the site.

## When blocking GPTBot makes sense

Opting out of training is a content decision, not a visibility one. Stores with original writing, photography or research often opt out; stores whose pages are mostly product facts often don't bother. Either way, nothing in OpenAI's documentation connects GPTBot to whether ChatGPT search shows or links your store. That is OAI-SearchBot's job.

If you allow both, OpenAI says it may use the results of one crawl for both purposes, so allowing GPTBot does not double the load on your store.

## Keep going

- [AI crawler checker](https://aiscovery.io/tools/ai-crawler-checker): See what your robots.txt tells each AI company's crawler.
- [Aiscovery on Shopify](https://aiscovery.io/platforms/shopify): What commonly breaks for AI agents on Shopify stores.
- [Product tour](https://aiscovery.io/product): The site check, AI agent test runs and the ranked fix list.
- [Request a demo](https://aiscovery.io/demo): A short call about what AI agents can do in your own store.

## Questions store owners ask

**Q: Will blocking GPTBot remove my products from ChatGPT?**

A: No. OpenAI uses OAI-SearchBot for ChatGPT search and says each setting is independent. Blocking GPTBot signals that your pages should not be used for model training.

**Q: Does blocking Google-Extended remove my store from AI Overviews?**

A: Google says Google-Extended does not affect inclusion in Google Search, and treats AI features in Search as part of Search, controlled by Googlebot and snippet rules such as nosnippet. Google-Extended covers Gemini training and grounding.

**Q: Can I block ChatGPT-User?**

A: You can write the rule, but OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the request. The only hard block is a firewall rule, and that also blocks shoppers who ask ChatGPT about your products.

**Q: How long does a robots.txt change take?**

A: OpenAI and Perplexity say up to about 24 hours. RFC 9309 tells crawlers not to use a cached robots.txt for more than 24 hours.

**Q: Does Aiscovery check which AI crawlers my store blocks?**

A: Yes. The automatic site check reads your robots.txt and shows what it declares for each AI company's crawler. It reports declared policy only: it never pretends to be an AI crawler, so a firewall block needs a separate check.

## Sources

Every technical claim on this page comes from the vendor's own documentation, checked on the date shown as Updated. Crawler names and settings change, so check the source before you edit a live store.

1. [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI)
2. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic)
3. [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (Perplexity)
4. [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (Google for Developers)
5. [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) (Google for Developers)
6. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) (Google Search Central)
7. [About Applebot](https://support.apple.com/en-us/119829) (Apple Support)
8. [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (Meta for Developers)
9. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (IETF)

## More guides

- [Does Cloudflare block AI agents on my store?](https://aiscovery.io/resources/does-cloudflare-block-ai-agents) (Cloudflare)
- [How do I add llms.txt to Shopify or WooCommerce, and do I need it?](https://aiscovery.io/resources/add-llms-txt-shopify-woocommerce) (llms.txt)
- [Why can't AI agents find my delivery price?](https://aiscovery.io/resources/why-ai-agents-cant-find-delivery-price) (Product facts)

## See what AI agents do in your store.

A short call: we run the site check on your store, show what robots.txt tells each AI crawler, and walk through a test run where named AI agents try key shopping tasks.

[Request a demo](https://aiscovery.io/demo)

Have a real workflow to test now? [Apply as a launch partner](https://aiscovery.io/partners)

---

This is the Markdown version of the page for AI agents. The HTML page at the canonical URL is authoritative. Site index: https://aiscovery.io/llms.txt