Free tool
Does your store let AI agents in?
Enter any store's domain. We read its robots.txt and show, crawler by crawler, what each AI company may do there: train, search or shop for a person.
- Common Crawl
We request your /robots.txt and nothing else, following at most two redirects within the same site. We never pretend to be an AI crawler, and a rule only says what a crawler is asked to do.
How the checker reads robots.txt
- Only robots.txtWe request only /robots.txt, over https, or over http if https doesn't connect, as AiscoveryAudit, our identified checker. We follow at most two redirects within the same site, never from https to http, and stop at 5 seconds or 500 KiB.
- The standard's rulesA crawler follows the groups that name it, or the * group if none do. Within them the longest matching path wins, and Allow wins a tie. * and $ work as wildcards.
- Declared, not testedA rule says what a crawler is asked to do. It can't show a firewall that blocks agents, or a page an agent can't use. That is what an Aiscovery test run shows.
The crawlers we check
Thirteen user agents from eight companies, sorted by what they do with your pages.
Acts for a person
Opens a page because a person asked, for example to compare products.
- ChatGPT-UserOpenAI
- Claude-UserAnthropic
- Perplexity-UserPerplexity
AI search
Finds pages to show and cite in AI answers.
- OAI-SearchBotOpenAI
- Claude-SearchBotAnthropic
- PerplexityBotPerplexity
- BingbotMicrosoft
AI training
Collects pages to train AI models.
- GPTBotOpenAI
- ClaudeBotAnthropic
- Google-ExtendedGoogle
- Applebot-ExtendedApple
- CCBotCommon Crawl
- meta-externalagentMeta
Questions about AI crawlers and robots.txt
What is the difference between training, search and user-triggered crawlers?
Training crawlers, such as GPTBot or ClaudeBot, collect pages to train AI models. Search crawlers, such as OAI-SearchBot or PerplexityBot, find pages to show and cite in AI answers. User-triggered agents, such as ChatGPT-User or Claude-User, open a page because a person asked, for example to compare two products. You can allow one kind and block another.
If my robots.txt doesn't name a crawler, is it allowed?
A crawler that is not named follows the * group. If there is no * group either, nothing is blocked. That is why many stores that never thought about AI are fully open to it.
Does blocking GPTBot keep my store out of ChatGPT?
No. GPTBot is for training only. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when a person asks. OpenAI says robots.txt may not apply to those user-started visits.
Does this tool crawl my store?
No. It requests /robots.txt and nothing else, as AiscoveryAudit: over https, or http if https doesn't connect, following at most two redirects within the same site. It never pretends to be an AI crawler. It keeps the answer for at most a minute, so repeat checks in that minute don't reach your store again; after that, a shared link runs the check again.
Why would an AI agent still fail on a store that allows it?
robots.txt is only the first door. Bot protection can still turn the agent away, and pages can hide prices, variants or delivery costs where an agent can't read them. Aiscovery lets named AI agents try key shopping tasks to find those places.
robots.txt says who may come in. We test what happens next.
Aiscovery lets named AI agents try key shopping tasks in your store, such as finding a product, its price and delivery, and shows where they get stuck and what to fix.