How do I check which AI crawlers my store blocks?

Read your robots.txt the way a crawler does

1. 1

   Step 1: Open the live file

   Go to https://yourstore.com/robots.txt in a browser, or fetch it from a terminal so you see the status code too. A 200 means crawlers read the rules. A 404 means they may crawl everything. A 5xx means they must assume everything is disallowed.

   curl -s -w "\nHTTP %{http_code}\n" https://yourstore.com/robots.txt

2. 2

   Step 2: Repeat it on every host you sell from

   robots.txt only covers the host it is served from. If you use shop.example.com, a www host or a separate checkout or help-center domain, each has its own file. Anthropic, for example, asks site owners to add rules for every subdomain they want to opt out.

3. 3

   Step 3: Find the group that applies to each crawler

   A group starts with one or more User-agent lines. A crawler looks for a group naming its token. Names are case-insensitive, so gptbot matches GPTBot. If several groups name the same crawler, RFC 9309 says to merge them. Only a crawler with no group of its own uses User-agent: *.

4. 4

   Step 4: Apply the most specific rule

   Inside the group, the rule with the longest matching path must be used. If an Allow and a Disallow rule match equally, RFC 9309 says the Allow rule should be used. Paths are case-sensitive, so Disallow: /Products does not block /products.

   User-agent: *
   Disallow: /cart
   Disallow: /checkout
   
   User-agent: GPTBot
   Disallow: /
   
   # GPTBot: blocked everywhere (its own group).
   # OAI-SearchBot: no group of its own, so it follows *.
   #   /products/blue-tee  allowed
   #   /cart               disallowed

The AI crawlers to look for

These are the tokens the companies document today. Training crawlers collect pages for model training. Search crawlers build the index an AI assistant answers from. User-triggered fetchers open a page because a person asked.

Check what your firewall or CDN does

robots.txt is a request that well-behaved crawlers honor. Your CDN, firewall or hosting platform can still answer an AI crawler with a challenge page or a 403, whatever robots.txt says. This is the block store owners most often don't know they have.

1. 1

   Step 1: Look at the bot settings in your CDN

   On Cloudflare, check the AI crawler and bot settings, then the security events for AI crawlers. Our guide on whether Cloudflare blocks AI agents lists where each setting lives.

2. 2

   Step 2: Search your access logs for AI crawlers

   If you run your own server or can export CDN logs, count the status codes each AI crawler received. Many 403 or 429 responses mean a block or a rate limit.

   grep -Eio 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user' access.log \
     | sort | uniq -c
   
   grep -Ei 'oai-searchbot' access.log | awk '{print $9}' | sort | uniq -c

3. 3

   Step 3: Don't trust a spoofed user agent test

   Sending a request from your laptop with a GPTBot user agent only tests rules that match on the user agent. Bot protection checks the real crawler's IP ranges, and the companies publish them for that reason. A fake GPTBot may be blocked while the real one gets through, or the other way round.

Check it automatically

The AI crawler checker and Aiscovery's site check read your live robots.txt and show what it declares for each AI company's crawler, including the groups that override *. They report declared policy only: Aiscovery never pretends to be an AI crawler, so a firewall block needs the log check above.

On a platform where you can't see the server, start with the platform guide. On Shopify, robots.txt comes from a theme template; see how to allow ChatGPT and other AI agents on a Shopify store.