# Does your store let AI agents in?

> Free tool: enter a store's domain and see, crawler by crawler, what its robots.txt lets OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Meta and Common Crawl do, with the lines that decide it and a fix.

- Canonical URL: https://aiscovery.io/tools/ai-crawler-checker
- Markdown version: https://aiscovery.io/tools/ai-crawler-checker.md
- Last updated: 2026-10-21

Free tool

Enter any store's domain. We read its robots.txt and show, crawler by crawler, what each AI company may do there: train, search or shop for a person.

AI companies checked: OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Common Crawl, Meta.

## AI crawler checker

> Interactive demo on the HTML page: a "Store domain or URL" field (such as yourstore.com) and a "Check robots.txt" button. The check shows its steps (Resolving the host, Fetching robots.txt, Reading the rules), then a headline for the store, a verdict for each crawler below (Allowed, Blocked, Partly blocked) with the lines that decide it, a "Test a page path" field, the robots.txt file itself, a "Suggested fix" and "Copy link to this result".

We request your /robots.txt and nothing else, following at most two redirects within the same site. We never pretend to be an AI crawler, and a rule only says what a crawler is asked to do.

> Example domains are public stores, picked as illustrations. Try: amazon.com, allbirds.com, ikea.com, bestbuy.com.

### Headlines the check can show

- **Open: every AI crawler we check may read this store.** AI search, AI agents acting for shoppers and AI training can all read the pages robots.txt covers.
- **Open, with a few paths closed.** Every AI crawler we check may read the store. Some paths are closed, which is normal for cart, account and admin pages.
- **Search and shopping agents are in. Training is out.** AI search and agents acting for shoppers may read the store, while some AI training crawlers are asked to stay out.
- **Some AI agents are turned away at the door.** AI search crawlers or agents acting for shoppers are asked to stay out of the whole store, so those AI tools can't read or cite its pages.
- **Closed: this store asks every AI crawler to stay out.** No AI search crawler, shopping agent or training crawler we check may read it.
- **No robots.txt, so nothing is blocked.** The site has no robots.txt (it answered with a 4xx status). By the robots.txt standard, every crawler may read everything.
- **robots.txt answered with an error.** It answered our request for robots.txt with a 5xx or 429 status, which tells crawlers to stay out of the whole site until it works again. It may answer AI crawlers differently.
- **robots.txt is a web page, not rules.** The site answers /robots.txt with an HTML page, so crawlers find no rules in it and may read everything. Most likely the file is missing.
- **robots.txt is not a text file.** The site answers /robots.txt with a file that is not text, so crawlers find no rules in it.
- **The site turned our request away.** It answered /robots.txt with 401 or 403. The robots.txt standard reads that as no rules, so the grid shows every crawler allowed. In practice it usually means a firewall or bot protection that can turn AI agents away too, and robots.txt can't show that.

### Test a page path

Each crawler's answer for this one path, with the rule that decides it.

### Suggested fix

To let the AI search crawlers and shopping agents that are blocked read the store:

- A crawler that has its own group follows only that group, so this opens the store to them while your \* group still applies to everyone else.
- Without a group of their own, these crawlers follow your \* group, like every other visitor.

**Optional: stay visible, opt out of training.** This store lets AI training crawlers in. If you would rather not, this group asks them to stay out while AI search and shopping agents keep their access.

**Nothing to fix here.** AI search crawlers and shopping agents may read this store. robots.txt is only the first door, though: an agent can still get stuck on the pages themselves.

## How the checker reads robots.txt

1. **Only robots.txt.** We request only /robots.txt, over https, or over http if https doesn't connect, as AiscoveryAudit, our identified checker. We follow at most two redirects within the same site, never from https to http, and stop at 5 seconds or 500 KiB.
2. **The standard's rules.** A crawler follows the groups that name it, or the \* group if none do. Within them the longest matching path wins, and Allow wins a tie. \* and $ work as wildcards.
3. **Declared, not tested.** A rule says what a crawler is asked to do. It can't show a firewall that blocks agents, or a page an agent can't use. That is what an Aiscovery test run shows.

## The crawlers we check

Thirteen user agents from eight companies, sorted by what they do with your pages.

### Acts for a person

Opens a page because a person asked, for example to compare products.

| User agent | Company | What it does | If blocked |
| --- | --- | --- | --- |
| ChatGPT-User | OpenAI | Opens a page when a person asks ChatGPT to, for example to look up a product. OpenAI says robots.txt rules may not apply to these visits, because a person started them. | ChatGPT is asked not to open your pages when a shopper asks. |
| Claude-User | Anthropic | Opens a page when a Claude user asks a question that needs it. | Claude won't open your pages when a shopper asks about them. |
| Perplexity-User | Perplexity | Opens a page when a person asks Perplexity a question about it. Perplexity says this fetcher generally ignores robots.txt, because a person asked for the page. | Perplexity is asked not to open your pages when a shopper asks. |

### AI search

Finds pages to show and cite in AI answers.

| User agent | Company | What it does | If blocked |
| --- | --- | --- | --- |
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search answers. | Your store is left out of ChatGPT search answers. |
| Claude-SearchBot | Anthropic | Reads pages to improve the search results Claude gives. | Your pages are not indexed for Claude's search. |
| PerplexityBot | Perplexity | Finds pages to show and link in Perplexity's answers. Perplexity says it is not used for training. | Your store is not linked in Perplexity's search results. |
| Bingbot | Microsoft | Crawls pages for Bing search, the index Microsoft Copilot also searches. | Your store can't be crawled for Bing search results. |

### AI training

Collects pages to train AI models.

| User agent | Company | What it does | If blocked |
| --- | --- | --- | --- |
| GPTBot | OpenAI | Collects pages that may be used to train OpenAI's models. | Your pages stay out of OpenAI's model training. |
| ClaudeBot | Anthropic | Collects pages that may be used to train Anthropic's models. | Your pages stay out of Anthropic's model training. |
| Google-Extended | Google | Not a crawler but a switch: whether Google may use your pages to train Gemini and to ground its answers. | Gemini won't train on or ground answers in your pages. Google Search is not affected. |
| Applebot-Extended | Apple | Not a crawler but a switch: whether Apple may train its models on pages Applebot collected. | Apple won't train its models on your pages. Siri and Spotlight search are not affected. |
| CCBot | Common Crawl | Builds Common Crawl, a free web archive that many AI models are trained on. | Your pages stay out of new Common Crawl archives. |
| meta-externalagent | Meta | Crawls pages to train Meta's AI models and improve its products. | Meta is asked not to use your pages for AI training. |

Company documentation:

- [OpenAI](https://developers.openai.com/api/docs/bots)
- [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity](https://docs.perplexity.ai/guides/bots)
- [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Apple](https://support.apple.com/en-us/119829)
- [Microsoft](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0)
- [Common Crawl](https://commoncrawl.org/ccbot)
- [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

## Questions about AI crawlers and robots.txt

**Q: What is the difference between training, search and user-triggered crawlers?**

A: Training crawlers, such as GPTBot or ClaudeBot, collect pages to train AI models. Search crawlers, such as OAI-SearchBot or PerplexityBot, find pages to show and cite in AI answers. User-triggered agents, such as ChatGPT-User or Claude-User, open a page because a person asked, for example to compare two products. You can allow one kind and block another.

**Q: If my robots.txt doesn't name a crawler, is it allowed?**

A: A crawler that is not named follows the \* group. If there is no \* group either, nothing is blocked. That is why many stores that never thought about AI are fully open to it.

**Q: Does blocking GPTBot keep my store out of ChatGPT?**

A: No. GPTBot is for training only. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when a person asks. OpenAI says robots.txt may not apply to those user-started visits.

**Q: Does this tool crawl my store?**

A: No. It requests /robots.txt and nothing else, as AiscoveryAudit: over https, or http if https doesn't connect, following at most two redirects within the same site. It never pretends to be an AI crawler. It keeps the answer for at most a minute, so repeat checks in that minute don't reach your store again; after that, a shared link runs the check again.

**Q: Why would an AI agent still fail on a store that allows it?**

A: robots.txt is only the first door. Bot protection can still turn the agent away, and pages can hide prices, variants or delivery costs where an agent can't read them. Aiscovery lets named AI agents try key shopping tasks to find those places.

## robots.txt says who may come in. We test what happens next.

Aiscovery lets named AI agents try key shopping tasks in your store, such as finding a product, its price and delivery, and shows where they get stuck and what to fix.

- [Request a demo](https://aiscovery.io/demo)
- [See the product](https://aiscovery.io/product)

---

This is the Markdown version of the page for AI agents. The HTML page at the canonical URL is authoritative. Site index: https://aiscovery.io/llms.txt