# Can AI agents shop a custom-built store?

> Can AI agents shop a custom-built online store? The firewall, robots.txt, rendering and policy mistakes that stop them, and the config and code to fix each.

- Canonical URL: https://aiscovery.io/platforms/custom-built
- Markdown version: https://aiscovery.io/platforms/custom-built.md
- Last updated: 2026-10-21

They can when your public pages put product facts in the first response and your firewall lets identified bots through, and on a custom stack both are your own code, which is where it commonly breaks.

No platform sets defaults for you. Your robots.txt, CDN and firewall rules, how prices and stock reach the page and where policies live are decisions your team made, often before AI agents existed.

- [Request a demo](https://aiscovery.io/demo)
- [Check AI crawler rules free](https://aiscovery.io/tools/ai-crawler-checker)

Reviewed 21 October 2026 against official documentation. [20 sources](https://aiscovery.io/platforms/custom-built#sources)

> Example: what an agent runs into on custom-built stores

> Interactive demo on the HTML page: an illustrative storefront scene with numbered spots that match the common problems below.

Common problems:

1. Prices or stock come from an API call after the page loads.
2. Firewall rules written for scrapers also block AI agents.
3. Policies sit in PDFs that agents don't open.

> Example: An illustrative store, not a real result. Your own result shows only what is really on your store.

## Custom-built stores at a glance

- **robots.txt:** Whatever your server returns at /robots.txt. A server error there tells compliant crawlers to stay out.
- **Bot protection:** Your CDN or WAF. Cloudflare now sets Search, Agent and Training policies per domain.
- **Structured data:** Only what you add. Google reads store-wide shipping and returns from Organization markup.
- **Agent checkout:** Open protocols you can implement: ACP from OpenAI and Stripe, UCP from Google.

## What usually breaks on custom-built stores

The usual suspects on a custom stack, roughly in the order an agent meets them.

### 1. Firewall rules written for scrapers stop agents too

Rate limits, challenges and user agent blocks catch AI crawlers and shoppers' agents along with scrapers. On Cloudflare, Bot Fight Mode can't be skipped with WAF rules, and new domains that show ads, added since 15 September 2026, block user-directed agents on those pages by default. A 403 tells Google the page doesn't exist; a 429 says slow down. [\[3\]](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes) [\[4\]](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) [\[5\]](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/)

### 2. A robots.txt that errors, or blocks more than you meant

If /robots.txt returns a server error, compliant crawlers must assume the whole site is disallowed. And a crawler that finds a group with its own name ignores the \* group, so a GPTBot group with only Allow: / also lifts your cart and account rules for GPTBot. [\[1\]](https://www.rfc-editor.org/rfc/rfc9309)

### 3. Prices and stock arrive after the page loads

Product HTML that says 'loading' while a script calls a price or stock API gives an agent nothing until the call returns, and agents that read the first response never see it.

### 4. Policies in PDFs and chat widgets

Returns terms in a downloadable PDF, or delivery answers that only a chat widget gives, are facts an agent commonly can't reach from a product page.

### 5. Markup that disagrees with the page

Google's merchant listings want an Offer per buyable product, not a price range, and settings in Merchant Center override on-page markup. A hand-built JSON-LD block that drifts from the page leaves agents with two different answers. [\[11\]](https://developers.google.com/search/docs/appearance/structured-data/merchant-listing)

## How to fix it on custom-built stores

Each fix is config or a small change in the templates that render your product pages.

[Check which AI crawlers your robots.txt allows, free](https://aiscovery.io/tools/ai-crawler-checker)

### 1. Let verified bots past your custom rules

Cloudflare's own pattern is a first rule that skips the remaining custom rules for verified bots. Shoppers' agents, such as ChatGPT agent, increasingly sign requests with Web Bot Auth, which Cloudflare treats as signed agents. When you rate limit, answer 429 with Retry-After rather than 403. [\[2\]](https://www.rfc-editor.org/rfc/rfc6585) [\[6\]](https://developers.cloudflare.com/use-cases/solutions/stop-malicious-bots/) [\[7\]](https://blog.cloudflare.com/signed-agents/)

Cloudflare WAF custom rule:

```
Rule name:  Allow verified bots
Expression: (cf.client.bot)
Action:     Skip > All remaining custom rules
Placement:  First
```

### 2. Serve robots.txt as a static file with explicit groups

Make /robots.txt a static file that can't fail with your app. Name the crawlers you care about in groups, repeat private paths in each group, and decide training separately from search. User-requested fetchers like ChatGPT-User and Perplexity-User say robots.txt may not apply to them; Anthropic says its bots honor robots.txt, and Claude-User can be blocked there. [\[1\]](https://www.rfc-editor.org/rfc/rfc9309) [\[8\]](https://developers.openai.com/api/docs/bots) [\[9\]](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) [\[10\]](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

robots.txt:

```text
# AI search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

# Model training: your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
```

### 3. Put price, stock and delivery in the first HTML

Render the current price, stock status and delivery summary on the server, and let scripts update them afterwards if they must. Cache the page briefly instead of moving the facts into a browser call.

### 4. Mark up offers, shipping and returns

Give each buyable product an Offer with price, currency and availability. Put shipping and returns either on the offer or once, store-wide, on your Organization, which Google has supported for returns since 2024 and for shipping since November 2025. [\[11\]](https://developers.google.com/search/docs/appearance/structured-data/merchant-listing) [\[12\]](https://developers.google.com/search/docs/appearance/structured-data/shipping-policy) [\[13\]](https://developers.google.com/search/blog/2025/11/more-ways-to-share-shipping) [\[14\]](https://developers.google.com/search/docs/appearance/structured-data/return-policy)

JSON-LD in the product page (example values, use your own):

```json
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Rain jacket",
  "sku": "RJ-BLUE-M",
  "image": "https://example.com/images/rain-jacket.jpg",
  "offers": {
    "@type": "Offer",
    "url": "https://example.com/products/rain-jacket?size=m",
    "price": 129.0,
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock",
    "shippingDetails": {
      "@type": "OfferShippingDetails",
      "shippingRate": { "@type": "MonetaryAmount", "value": 4.95, "currency": "USD" },
      "shippingDestination": { "@type": "DefinedRegion", "addressCountry": "US" },
      "deliveryTime": {
        "@type": "ShippingDeliveryTime",
        "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" },
        "transitTime": { "@type": "QuantitativeValue", "minValue": 2, "maxValue": 5, "unitCode": "DAY" }
      }
    },
    "hasMerchantReturnPolicy": {
      "@type": "MerchantReturnPolicy",
      "applicableCountry": "US",
      "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
      "merchantReturnDays": 30,
      "returnMethod": "https://schema.org/ReturnByMail",
      "returnFees": "https://schema.org/FreeReturn"
    }
  }
}
```

### 5. Turn PDFs and chat answers into pages

Publish delivery and returns as HTML pages, link them from every product page and the footer, and keep the wording in step with what your checkout and support team actually do.

## Agent protocols a custom store can adopt

None of these is needed for agents to read your pages, but they decide how agents identify themselves and check out. Newest last.

- **Aug 2025: Signed agents on Cloudflare.** Cloudflare started recognising agents that sign their requests with Web Bot Auth, with ChatGPT agent among the first. [\[7\]](https://blog.cloudflare.com/signed-agents/)
- **Sep 2025: Agentic Commerce Protocol.** OpenAI and Stripe released ACP, an open checkout standard, with Instant Checkout in ChatGPT. Product feeds in ChatGPT are open to approved partners. [\[15\]](https://stripe.com/newsroom/news/stripe-openai-instant-checkout) [\[16\]](https://www.agenticcommerce.dev/) [\[17\]](https://developers.openai.com/commerce/guides/get-started)
- **Sep 2025: Agent Payments Protocol.** Google announced AP2 for agent-led payments, built on signed mandates that record what the shopper authorised. [\[18\]](https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol)
- **Oct 2025: Visa Trusted Agent Protocol.** Visa, with Cloudflare, published a protocol that helps merchants tell trusted shopping agents from malicious bots, based on HTTP message signatures. [\[19\]](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21716.html)
- **Jan 2026: Universal Commerce Protocol.** Google announced UCP, an open standard for the whole shopping journey, co-developed with Shopify, Etsy, Wayfair, Target and Walmart. [\[20\]](https://blog.google/products/ads-commerce/agentic-commerce-ai-tools-protocol-retailers-platforms/)
- **Jul 2026: Separate Search, Agent and Training policies on Cloudflare.** Cloudflare replaced its single AI bot switch with three behaviours, each allowed, blocked, or blocked on pages with ads, with new defaults from 15 September 2026: new domains that show ads block Agent on pages with ads and disallow Training, while Search stays allowed; sites without ads allow all three. [\[4\]](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)

## What Aiscovery checks on custom-built stores

From the outside, the way an agent sees your store. Nothing to install, and agents stop before login, cart, checkout and payment.

[Request a demo](https://aiscovery.io/demo) · [Free AI crawler checker](https://aiscovery.io/tools/ai-crawler-checker)

- Whether /robots.txt answers, and what it declares for each AI crawler, group by group.
- Reachability and redirects, and pages that fail for our identified checker. We never pretend to be an AI crawler, so rules for their user agents aren't tested.
- Page basics and JSON-LD syntax on the HTML your server sends, read without running JavaScript.
- AI-facing files: sitemaps, llms.txt and llms-full.txt.
- GPT-6 Luna and Gemini try to find a product, its price, stock, delivery and returns in a real browser, stopping before cart and checkout.

## Questions about custom-built stores and AI agents

**Q: Should I allow GPTBot on my store?**

A: It's your call: GPTBot collects data for training OpenAI's models. Whether you appear in ChatGPT search depends on OAI-SearchBot, and ChatGPT-User fetches pages when a user asks. You can block the first and allow the other two.

**Q: Why do AI agents get 403 errors on my site?**

A: Commonly a WAF, CDN or rate limit rule built for scrapers. Let verified bots skip your custom rules, check your CDN's AI bot settings, and answer real overload with 429 and Retry-After, which crawlers read as 'slow down' rather than 'gone'.

**Q: Do AI agents obey robots.txt?**

A: Their crawlers say they do: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot among them. Fetchers that act on a user's request, such as ChatGPT-User, Perplexity-User and Google-Agent, say robots.txt may not apply. Anthropic says its bots honor robots.txt, and Claude-User can be blocked there.

**Q: Do I need ACP or UCP for agents to shop my store?**

A: Not to be read. Agents can browse your pages today. The protocols matter for checkout inside an AI assistant: OpenAI takes product feeds from approved partners, and Google's UCP is an open standard you can implement.

**Q: What structured data does a store need for AI agents?**

A: Product markup with an Offer per buyable item: price, currency and availability. Add shipping and returns on each offer, or once on your Organization, and keep both in step with what the page says.

**Q: Do you need access to our code or servers?**

A: No. Aiscovery tests your public storefront from the outside with an identified checker, so nothing is installed and nothing changes on your side.

## Sources

Official documentation and announcements, checked on 21 October 2026. Platforms change often, so check the source before you change settings.

1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), IETF, 2022-09
2. [RFC 6585: 429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585), IETF, 2012-04
3. [How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes), Google Search Central
4. [New AI traffic options: Search, Agent and Training](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/), Cloudflare changelog, 1 July 2026
5. [Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/), Cloudflare docs
6. [Stop malicious bots: skip verified bots](https://developers.cloudflare.com/use-cases/solutions/stop-malicious-bots/), Cloudflare docs
7. [Signed agents](https://blog.cloudflare.com/signed-agents/), Cloudflare, 28 August 2025
8. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), OpenAI
9. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic
10. [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), Perplexity
11. [Merchant listing structured data](https://developers.google.com/search/docs/appearance/structured-data/merchant-listing), Google Search Central
12. [Shipping policy structured data](https://developers.google.com/search/docs/appearance/structured-data/shipping-policy), Google Search Central
13. [More ways to share your shipping and returns policies with Google](https://developers.google.com/search/blog/2025/11/more-ways-to-share-shipping), Google Search Central blog, 12 November 2025
14. [Merchant return policy structured data](https://developers.google.com/search/docs/appearance/structured-data/return-policy), Google Search Central
15. [Stripe powers Instant Checkout in ChatGPT](https://stripe.com/newsroom/news/stripe-openai-instant-checkout), Stripe, 29 September 2025
16. [Agentic Commerce Protocol](https://www.agenticcommerce.dev/), OpenAI and Stripe
17. [Commerce in ChatGPT: get started](https://developers.openai.com/commerce/guides/get-started), OpenAI
18. [Announcing the Agent Payments Protocol (AP2)](https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol), Google Cloud, 16 September 2025
19. [Visa introduces Trusted Agent Protocol](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21716.html), Visa, 14 October 2025
20. [New agentic commerce tools and the Universal Commerce Protocol](https://blog.google/products/ads-commerce/agentic-commerce-ai-tools-protocol-retailers-platforms/), Google, 11 January 2026

## Related guides

- [Can AI agents shop a headless store?](https://aiscovery.io/platforms/headless)
- [Can AI agents shop a WooCommerce store?](https://aiscovery.io/platforms/woocommerce)
- [Can AI agents shop a Magento store?](https://aiscovery.io/platforms/magento)

### Go deeper

- [Every store engine](https://aiscovery.io/platforms)
- [Testing e-commerce stores with AI agents](https://aiscovery.io/resources/test-ecommerce-ai-agents)
- [The llms.txt guide](https://aiscovery.io/resources/llms-txt)
- [Our checker and how to allow it](https://aiscovery.io/checker)

Platform names and logos belong to their owners. Aiscovery is not affiliated with them; we test any store from the outside.

## See what agents see on custom-built stores.

Tell us your store and we'll show you, on a short call, what AI agents could and couldn't find there, and the fix for each.

[Request a demo](https://aiscovery.io/demo)

Have a real workflow to test now? [Apply as a launch partner](https://aiscovery.io/partners)

---

This is the Markdown version of the page for AI agents. The HTML page at the canonical URL is authoritative. Site index: https://aiscovery.io/llms.txt