# How do I check which AI crawlers my store blocks?

> Read your robots.txt the way AI crawlers do: find each crawler's group, apply the most specific rule, then check your CDN or firewall. With a crawler table.

- Canonical URL: https://aiscovery.io/resources/check-which-ai-crawlers-your-store-blocks
- Markdown version: https://aiscovery.io/resources/check-which-ai-crawlers-your-store-blocks.md
- Last updated: 2026-10-21

Open `yourstore.com/robots.txt`, find the `User-agent` group for each AI crawler (or the `*` group if it has none) and read that group's `Disallow` and `Allow` rules. Then check your CDN or firewall, because a bot block there turns crawlers away even when robots.txt lets them in.

- Topic: AI crawlers
- Author: Aiscovery team
- Published: 2026-10-21
- Updated: 2026-10-21
- Reading time: 8 min read

## Key facts

- A crawler that has its own group in robots.txt ignores the `User-agent: *` group completely.
- Crawler names are matched case-insensitively; paths are matched case-sensitively, and the longest matching rule wins.
- If robots.txt returns a server error (5xx), RFC 9309 says crawlers must treat the whole site as disallowed.
- Some AI fetchers that act for a person, such as ChatGPT-User and Perplexity-User, may not follow robots.txt at all.
- robots.txt only states a policy. A CDN or firewall bot rule is what actually blocks a request.

## Read your robots.txt the way a crawler does

### 1. Open the live file

Go to `https://yourstore.com/robots.txt` in a browser, or fetch it from a terminal so you see the status code too. A `200` means crawlers read the rules. A `404` means they may crawl everything. A `5xx` means they must assume everything is disallowed.

**Terminal: status code and file**

```bash
curl -s -w "\nHTTP %{http_code}\n" https://yourstore.com/robots.txt
```

### 2. Repeat it on every host you sell from

robots.txt only covers the host it is served from. If you use `shop.example.com`, a `www` host or a separate checkout or help-center domain, each has its own file. Anthropic, for example, asks site owners to add rules for every subdomain they want to opt out.

### 3. Find the group that applies to each crawler

A group starts with one or more `User-agent` lines. A crawler looks for a group naming its token. Names are case-insensitive, so `gptbot` matches `GPTBot`. If several groups name the same crawler, RFC 9309 says to merge them. Only a crawler with no group of its own uses `User-agent: *`.

### 4. Apply the most specific rule

Inside the group, the rule with the longest matching path must be used. If an `Allow` and a `Disallow` rule match equally, RFC 9309 says the `Allow` rule should be used. Paths are case-sensitive, so `Disallow: /Products` does not block `/products`.

**Example: what GPTBot and OAI-SearchBot may fetch**

```text
User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
Disallow: /

# GPTBot: blocked everywhere (its own group).
# OAI-SearchBot: no group of its own, so it follows *.
#   /products/blue-tee  allowed
#   /cart               disallowed
```

## The AI crawlers to look for

These are the tokens the companies document today. Training crawlers collect pages for model training. Search crawlers build the index an AI assistant answers from. User-triggered fetchers open a page because a person asked.

AI crawler tokens, purpose and robots.txt behavior:

| Token | Company | Purpose | Follows robots.txt |
| --- | --- | --- | --- |
| `GPTBot` | OpenAI | Model training | Yes |
| `OAI-SearchBot` | OpenAI | ChatGPT search | Yes |
| `ChatGPT-User` | OpenAI | Pages a person asks ChatGPT to open | OpenAI says rules may not apply |
| `ClaudeBot` | Anthropic | Model training | Yes |
| `Claude-SearchBot` | Anthropic | Search quality | Yes |
| `Claude-User` | Anthropic | Pages a person asks Claude to open | Anthropic says its bots honor robots.txt |
| `PerplexityBot` | Perplexity | Perplexity search | Yes |
| `Perplexity-User` | Perplexity | Pages a person asks Perplexity to open | Generally ignores robots.txt |
| `Googlebot` | Google | Google Search, including its AI features | Yes |
| `Storebot-Google` | Google | All Google Shopping surfaces | Yes |
| `Google-Extended` | Google | Control token for Gemini training and grounding; no crawler of its own | Read from Googlebot's crawl |
| `Applebot` | Apple | Siri, Spotlight and Safari search; may also be used for training | Yes; follows Googlebot rules if Applebot has none |
| `Applebot-Extended` | Apple | Control token for Apple model training | Read from Applebot's crawl |
| `meta-externalagent` | Meta | Training and indexing | Yes |
| `meta-webindexer` | Meta | Meta AI search | Yes |
| `meta-externalfetcher` | Meta | Links a person shares or asks Meta AI to open | Meta says it may bypass robots.txt |
| `Amazonbot` | Amazon | Amazon products and services; may train Amazon AI models | Yes |
| `bingbot` | Microsoft | Bing search and Copilot | Yes |

> **Copilot has no crawler of its own.** Microsoft says Bing and Copilot search rely on the same crawling and indexing. Blocking `bingbot` removes you from both. Copilot's use of a page is controlled with meta tags such as `noarchive` and `nocache`, not robots.txt.

## Check what your firewall or CDN does

robots.txt is a request that well-behaved crawlers honor. Your CDN, firewall or hosting platform can still answer an AI crawler with a challenge page or a `403`, whatever robots.txt says. This is the block store owners most often don't know they have.

### 1. Look at the bot settings in your CDN

On Cloudflare, check the AI crawler and bot settings, then the security events for AI crawlers. Our guide on [whether Cloudflare blocks AI agents](https://aiscovery.io/resources/does-cloudflare-block-ai-agents) lists where each setting lives.

### 2. Search your access logs for AI crawlers

If you run your own server or can export CDN logs, count the status codes each AI crawler received. Many `403` or `429` responses mean a block or a rate limit.

**Terminal: status codes per AI crawler (combined log format)**

```bash
grep -Eio 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user' access.log \
  | sort | uniq -c

grep -Ei 'oai-searchbot' access.log | awk '{print $9}' | sort | uniq -c
```

### 3. Don't trust a spoofed user agent test

Sending a request from your laptop with a GPTBot user agent only tests rules that match on the user agent. Bot protection checks the real crawler's IP ranges, and the companies publish them for that reason. A fake GPTBot may be blocked while the real one gets through, or the other way round.

## Check it automatically

The [AI crawler checker](https://aiscovery.io/tools/ai-crawler-checker) and Aiscovery's site check read your live robots.txt and show what it declares for each AI company's crawler, including the groups that override `*`. They report declared policy only: Aiscovery never pretends to be an AI crawler, so a firewall block needs the log check above.

On a platform where you can't see the server, start with the platform guide. On Shopify, robots.txt comes from a theme template; see [how to allow ChatGPT and other AI agents on a Shopify store](https://aiscovery.io/resources/allow-ai-agents-shopify-robots-txt).

## Keep going

- [AI crawler checker](https://aiscovery.io/tools/ai-crawler-checker): Paste your store URL and see what robots.txt declares for each AI crawler.
- [Aiscovery on Shopify](https://aiscovery.io/platforms/shopify): Where robots.txt and bot rules come from on a Shopify store.
- [Our checker](https://aiscovery.io/checker): How AiscoveryAudit identifies itself and how to allow or block it.
- [Request a demo](https://aiscovery.io/demo): See the site check and an AI agent test run on your own store.

## Questions store owners ask

**Q: My robots.txt doesn't mention GPTBot. Is it allowed?**

A: It follows your User-agent: \* group. If that group doesn't disallow a path, GPTBot may crawl it.

**Q: Why does OAI-SearchBot ignore my \* rules?**

A: Because your robots.txt has a group that names it. A crawler with its own group uses only that group, so repeat any rules from \* that should still apply, such as cart and checkout paths.

**Q: Can robots.txt block ChatGPT-User or Perplexity-User?**

A: Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Anthropic says its bots honor robots.txt, and Claude-User can be blocked there.

**Q: How long until crawlers see my change?**

A: OpenAI and Perplexity say up to about 24 hours. RFC 9309 tells crawlers not to use a cached robots.txt for more than 24 hours.

**Q: Does Bytespider respect robots.txt?**

A: We couldn't find documentation from ByteDance itself, so we don't list it. Treat claims about it on third-party sites with care.

## Sources

Every technical claim on this page comes from the vendor's own documentation, checked on the date shown as Updated. Crawler names and settings change, so check the source before you edit a live store.

1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (IETF)
2. [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI)
3. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic)
4. [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (Perplexity)
5. [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (Google for Developers)
6. [About Applebot](https://support.apple.com/en-us/119829) (Apple Support)
7. [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (Meta for Developers)
8. [About Amazonbot](https://developer.amazon.com/amazonbot) (Amazon Developer)
9. [Which crawlers does Bing use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) (Microsoft Bing Webmaster Tools)
10. [Bing Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a) (Microsoft Bing Webmaster Tools)

## More guides

- [Does blocking GPTBot hurt my store?](https://aiscovery.io/resources/does-blocking-gptbot-hurt-your-store) (AI crawlers)
- [Does Cloudflare block AI agents on my store?](https://aiscovery.io/resources/does-cloudflare-block-ai-agents) (Cloudflare)
- [How do I add llms.txt to Shopify or WooCommerce, and do I need it?](https://aiscovery.io/resources/add-llms-txt-shopify-woocommerce) (llms.txt)

## See what AI agents do in your store.

A short call: we run the site check on your store, show what robots.txt tells each AI crawler, and walk through a test run where named AI agents try key shopping tasks.

[Request a demo](https://aiscovery.io/demo)

Have a real workflow to test now? [Apply as a launch partner](https://aiscovery.io/partners)

---

This is the Markdown version of the page for AI agents. The HTML page at the canonical URL is authoritative. Site index: https://aiscovery.io/llms.txt